Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62590

[Windows][Vector Search] : Service 'fts' exited with status 1

    XMLWordPrintable

Details

    Description

      The steps to reproduce are an extension of what was followed for MB-62589

      Steps to reproduce

      1. Created a 3 node kv cluster
      2. Created a magma bucket with 3 collections
      3. Loaded around 4.3M docs onto each collection - 4096 dim vectors
      4. Added 5 fts nodes and created 3 indexes - 1 per collection - Index definitions attached
      5. During indexing, failed over one of the nodes and added back - Rebalance was successful
      6. Post indexing, failed over one of the nodes and added back - Rebalance was successful
      7. Created 3 clones of the existing indexes and during indexing, failed over 2 nodes. Added back one node and removed one. - Rebalance fails due to an OOM - tracked in MB-62589
      8. Started querying the indexes randomly (manually)
      9. Service fts crashed

      2024-07-03T18:51:55.590Z, ns_log:0:info:message(ns_1@ec2-174-129-77-219.compute-1.amazonaws.com) - Service 'fts' exited with status 1. Restarting. Messages:2024/07/03 18:47:52 got err removing file: 00000000041a.zap, err: remove c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@fts\vector_bucket._default.index_mutation-copy_628c1fcafc4cd04a_f53da44b.pindex\store\00000000041a.zap: The process cannot access the file because it is being used by another process.2024/07/03 18:47:52 got err removing file: 00000000041b.zap, err: remove c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@fts\vector_bucket._default.index_mutation-copy_628c1fcafc4cd04a_f53da44b.pindex\store\00000000041b.zap: The process cannot access the file because it is being used by another process.2024-07-03T18:51:47.475+00:00 [FATA] scorch AsyncError, path: c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@fts\vector_bucket._default.index_steady_state-copy_61d6b7b006fb4cff_4c1c5584.pindex\store, treating this as fatal, err: merging err: merging failed: , stack dump: c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@fts\dumps\1720032702.fts.stack.dump.txt -- main.initBleveOptions.func2() at init_bleve.go:113
       

      Stack trace seen in dump

      scorch AsyncError, path: c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@fts\vector_bucket._default.index_steady_state-copy_61d6b7b006fb4cff_4c1c5584.pindex\store, err: merging err: merging failed: 
      goroutine 536 [running]:runtime/pprof.writeGoroutineStacks({0x27000c0, 0xc09e2704d8})	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/pprof/pprof.go:743 +0x6aruntime/pprof.writeGoroutine({0x27000c0?, 0xc09e2704d8?}, 0x120?)	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/pprof/pprof.go:732 +0x25runtime/pprof.(*Profile).WriteTo(0x1def643?, {0x27000c0?, 0xc09e2704d8?}, 0xd2?)	C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/pprof/pprof.go:369 +0x14bmain.dumpStack({0xc0001be009?, 0x24?}, {0xc000c8a4e0, 0xd0})	cbft/cmd/cbft/stack_dump.go:59 +0x477main.initBleveOptions.func2({0x27005c0, 0xc0479869b0}, {0xc000c92960, 0x93})	cbft/cmd/cbft/init_bleve.go:110 +0xc6github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).fireAsyncError(...)	C:/Users/Administrator/cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.4.1-0.20240613145246-64ab0086a752/index/scorch/scorch.go:199github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).mergerLoop(0xc0008ab688)	C:/Users/Administrator/cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.4.1-0.20240613145246-64ab0086a752/index/scorch/merge.go:108 +0x3b4created by github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).Open in goroutine 296	C:/Users/Administrator/cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.4.1-0.20240613145246-64ab0086a752/index/scorch/scorch.go:217 +0x145
      goroutine 1 [chan receive, 13 minutes]:main.main()	cbft/cmd/cbft/main.go:284 +0x1d3a
      goroutine 19 [chan receive, 13 minutes]:github.com/couchbase/regulator/utils/logging.duplicateFlushLoop(...)	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/regulator/utils/logging/logger_golog.go:253created by github.com/couchbase/regulator/utils/logging.init.0 in goroutine 1	C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/regulator/utils/logging/logger_golog.go:49 +0x7b
      goroutine 67 [chan receive, 13 minutes]:github.com/couchbase/cbft.(*serverGroupTracker).handleServerGroupUpdates.func1()	C:/Jenkins/workspace/couchbase-server-windows/cbft/server_groups.go:111 +0x68created by github.com/couchbase/cbft.(*serverGroupTracker).handleServerGroupUpdates in goroutine 98	C:/Jenkins/workspace/couchbase-server-windows/cbft/server_groups.go:107 +0x1c5 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              rahul.rampure Rahul Rampure
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty