Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.6.4
-
Operating System : Microsoft Windows Server 2022
Couchbase Enterprise Edition build 7.6.2-3720
-
Untriaged
-
Windows 64-bit
-
-
0
-
Unknown
Description
The steps to reproduce are an extension of what was followed for MB-62589
Steps to reproduce
- Created a 3 node kv cluster
- Created a magma bucket with 3 collections
- Loaded around 4.3M docs onto each collection - 4096 dim vectors
- Added 5 fts nodes and created 3 indexes - 1 per collection - Index definitions attached
- During indexing, failed over one of the nodes and added back - Rebalance was successful
- Post indexing, failed over one of the nodes and added back - Rebalance was successful
- Created 3 clones of the existing indexes and during indexing, failed over 2 nodes. Added back one node and removed one. - Rebalance fails due to an OOM - tracked in
MB-62589 - Started querying the indexes randomly (manually)
- Service fts crashed
2024-07-03T18:51:55.590Z, ns_log:0:info:message(ns_1@ec2-174-129-77-219.compute-1.amazonaws.com) - Service 'fts' exited with status 1. Restarting. Messages:2024/07/03 18:47:52 got err removing file: 00000000041a.zap, err: remove c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@fts\vector_bucket._default.index_mutation-copy_628c1fcafc4cd04a_f53da44b.pindex\store\00000000041a.zap: The process cannot access the file because it is being used by another process.2024/07/03 18:47:52 got err removing file: 00000000041b.zap, err: remove c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@fts\vector_bucket._default.index_mutation-copy_628c1fcafc4cd04a_f53da44b.pindex\store\00000000041b.zap: The process cannot access the file because it is being used by another process.2024-07-03T18:51:47.475+00:00 [FATA] scorch AsyncError, path: c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@fts\vector_bucket._default.index_steady_state-copy_61d6b7b006fb4cff_4c1c5584.pindex\store, treating this as fatal, err: merging err: merging failed: , stack dump: c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@fts\dumps\1720032702.fts.stack.dump.txt -- main.initBleveOptions.func2() at init_bleve.go:113 |
|
Stack trace seen in dump
scorch AsyncError, path: c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@fts\vector_bucket._default.index_steady_state-copy_61d6b7b006fb4cff_4c1c5584.pindex\store, err: merging err: merging failed: |
goroutine 536 [running]:runtime/pprof.writeGoroutineStacks({0x27000c0, 0xc09e2704d8}) C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/pprof/pprof.go:743 +0x6aruntime/pprof.writeGoroutine({0x27000c0?, 0xc09e2704d8?}, 0x120?) C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/pprof/pprof.go:732 +0x25runtime/pprof.(*Profile).WriteTo(0x1def643?, {0x27000c0?, 0xc09e2704d8?}, 0xd2?) C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.22.2/go/src/runtime/pprof/pprof.go:369 +0x14bmain.dumpStack({0xc0001be009?, 0x24?}, {0xc000c8a4e0, 0xd0}) cbft/cmd/cbft/stack_dump.go:59 +0x477main.initBleveOptions.func2({0x27005c0, 0xc0479869b0}, {0xc000c92960, 0x93}) cbft/cmd/cbft/init_bleve.go:110 +0xc6github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).fireAsyncError(...) C:/Users/Administrator/cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.4.1-0.20240613145246-64ab0086a752/index/scorch/scorch.go:199github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).mergerLoop(0xc0008ab688) C:/Users/Administrator/cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.4.1-0.20240613145246-64ab0086a752/index/scorch/merge.go:108 +0x3b4created by github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).Open in goroutine 296 C:/Users/Administrator/cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.4.1-0.20240613145246-64ab0086a752/index/scorch/scorch.go:217 +0x145 |
goroutine 1 [chan receive, 13 minutes]:main.main() cbft/cmd/cbft/main.go:284 +0x1d3a |
goroutine 19 [chan receive, 13 minutes]:github.com/couchbase/regulator/utils/logging.duplicateFlushLoop(...) C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/regulator/utils/logging/logger_golog.go:253created by github.com/couchbase/regulator/utils/logging.init.0 in goroutine 1 C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/regulator/utils/logging/logger_golog.go:49 +0x7b |
goroutine 67 [chan receive, 13 minutes]:github.com/couchbase/cbft.(*serverGroupTracker).handleServerGroupUpdates.func1() C:/Jenkins/workspace/couchbase-server-windows/cbft/server_groups.go:111 +0x68created by github.com/couchbase/cbft.(*serverGroupTracker).handleServerGroupUpdates in goroutine 98 C:/Jenkins/workspace/couchbase-server-windows/cbft/server_groups.go:107 +0x1c5 |
Attachments
Issue Links
- relates to
-
MB-62589 [Windows][Vector Search] : Service 'fts' exited with status 2
- Resolved