Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62167

[Windows][Vector Search] : Service 'fts' exited with status 1 on a loop

    XMLWordPrintable

Details

    Description

      Steps to reproduce

      1. Created a 3 node kv cluster in Windows 2022
      2. Created a magma bucket named vector_bucket
      3. Created 3 collections under _default scope. The collection name as as follows
        1. steady_state
        2. mutation
        3. mutation_expiry
      4. Loaded 10M docs onto each collection. 30M docs in total in the bucket
      5. The state of the doc is as follows

       

      {
      		 
        "content" : a text content,
      		 
        "vec" : A vector of dims 4096,
      		 
        "doc_type" : "vector",
      		 
        "key" : key
      		 
      }

      6. Added 5 fts nodes onto the cluster

       

      7. Created 2 indexes. The definitions are attached

      8. Service fts crashed immediately after

       

      Service 'fts' exited with status 1. Restarting. Messages:
      		 
      2024-06-04T10:07:16.613+00:00 [INFO] pindex_bleve: started runBatchWorker: 2 for pindex: vector_bucket._default.index_mutation_688ab0af2eead4f1_df981270
      		 
      2024-06-04T10:07:16.613+00:00 [INFO] pindex_bleve: started runBatchWorker: 1 for pindex: vector_bucket._default.index_mutation_688ab0af2eead4f1_df981270
      		 
      2024-06-04T10:07:16.613+00:00 [INFO] pindex_bleve: started runBatchWorker: 0 for pindex: vector_bucket._default.index_mutation_688ab0af2eead4f1_df981270
      		 
      2024-06-04T10:07:16.620+00:00 [FATA] scorch AsyncError, path: c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@fts\vector_bucket._default.index_mutation_688ab0af2eead4f1_df981270.pindex\store, treating this as fatal, err: merging err: merging failed: Error in struct faiss::InvertedListsIOHook *__cdecl faiss::InvertedListsIOHook::lookup(int) at C:\Jenkins\workspace\cbdeps-platform-build\faiss\faiss\invlists\InvertedListsIOHook.cpp:63: read_InvertedLists: could not load ArrayInvertedLists as 646f6c69 ("ilod"), stack dump: c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@fts\dumps\1717495636.fts.stack.dump.txt -- main.initBleveOptions.func2() at init_bleve.go:113

       

      Service 'fts' exited with status 1. Restarting. Messages:
      		 
      2024-06-04T10:08:17.368+00:00 [INFO] vector_bucket._default.index_mutation_688ab0af2eead4f1_43f87f2c
      		 
      2024-06-04T10:08:17.368+00:00 [INFO] vector_bucket._default.index_mutation_688ab0af2eead4f1_259380a5
      		 
      2024-06-04T10:08:17.368+00:00 [INFO] vector_bucket._default.index_mutation_688ab0af2eead4f1_137764b5
      		 
      2024-06-04T10:08:17.368+00:00 [INFO] vector_bucket._default.index_mutation_688ab0af2eead4f1_8a53903c
      		 
      2024-06-04T10:08:17.368+00:00 [INFO] vector_bucket._default.index_mutation_688ab0af2eead4f1_3b831f22
      		 
      2024-06-04T10:08:17.368+00:00 [INFO] vector_bucket._default.index_mutation_688ab0af2eead4f1_85b5aacd
      		 
      2024-06-04T10:08:17.377+00:00 [FATA] scorch AsyncError, path: c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@fts\vector_bucket._default.index_mutation_688ab0af2eead4f1_8a53903c.pindex\store, treating this as fatal, err: merging err: merging failed: rm�, stack dump: c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@fts\dumps\1717495697.fts.stack.dump.txt -- main.initBleveOptions.func2() at init_bleve.go:113 

      Service 'fts' exited with status 1. Restarting. Messages:
      		 
      2024-06-04T10:08:49.810+00:00 [INFO] cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/nodeDefs-wanted/e6c3b5a7bbd5c4759d84c4a32cac52f6, key: nodeDefs-wanted/e6c3b5a7bbd5c4759d84c4a32cac52f6, deletion: false
      		 
      2024-06-04T10:08:49.810+00:00 [INFO] cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/planPIndexesLean/planPIndexesLean-7df40867525b46fa41e2e8e32e3d1629-1717494566128/0-$#$, key: planPIndexesLean/planPIndexesLean-7df40867525b46fa41e2e8e32e3d1629-1717494566128/0-$#$, deletion: false
      		 
      2024-06-04T10:08:49.810+00:00 [INFO] cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/planPIndexesLean/planPIndexesLean-b0fec9356345f77af0e1cd8ae80cf028-1717494622272/0-$#$, key: planPIndexesLean/planPIndexesLean-b0fec9356345f77af0e1cd8ae80cf028-1717494622272/0-$#$, deletion: false
      		 
      2024-06-04T10:08:49.817+00:00 [FATA] scorch AsyncError, path: c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@fts\vector_bucket._default.index_steady_state_1a073c12c05399bc_4c1c5584.pindex\store, treating this as fatal, err: merging err: merging failed: 0���, stack dump: c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@fts\dumps\1717495729.fts.stack.dump.txt -- main.initBleveOptions.func2() at init_bleve.go:113

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            raghav.sk Raghav S K
            raghav.sk Raghav S K
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty