Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.0
-
Couchbase Enterprise Edition 7.6.0 build 2140
-
Untriaged
-
Linux x86_64
-
-
0
-
Unknown
Description
- Created a cluster with 3 kv nodes and 3 fts nodes
- Created a magma bucket named "test_bucket"
- Created a vector index - definition attached
- Started loading data with different dimension vector in field "vec" - A snapshot of dataset attached
- Ran a query - Query attached
FTS service crashed
2024-02-15T06:52:53.245-08:00 [FATA] scorch AsyncError, path: /opt/couchbase/var/lib/couchbase/data/@fts/test_bucket._default.test_index_22a14bd6eebde5b2_aa574717.pindex/store, treating this as fatal, err: merger panic when processing /opt/couchbase/var/lib/couchbase/data/@fts/test_bucket._default.test_index_22a14bd6eebde5b2_aa574717.pindex/store, stack dump: /opt/couchbase/var/lib/couchbase/data/@fts/dumps/1708008773.fts.stack.dump.txt -- main.initBleveOptions.func2() at init_bleve.go:113 |
Observe this stacktrace from /opt/couchbase/var/lib/couchbase/data/@fts/dumps/1708008773.fts.stack.dump.txt
goroutine 147 [running]: |
runtime/pprof.writeGoroutineStacks({0x1dd45a0, 0xc00071c2c0}) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.6/go/src/runtime/pprof/pprof.go:703 +0x6a |
runtime/pprof.writeGoroutine({0x1dd45a0?, 0xc00071c2c0?}, 0xc000800b00?) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.6/go/src/runtime/pprof/pprof.go:692 +0x25 |
runtime/pprof.(*Profile).WriteTo(0x15d2d01?, {0x1dd45a0?, 0xc00071c2c0?}, 0x121?) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.6/go/src/runtime/pprof/pprof.go:329 +0x146 |
main.dumpStack({0x7ffe418c3364?, 0x24?}, {0xc0001a2360, 0x11f}) |
cbft/cmd/cbft/stack_dump.go:59 +0x47e |
main.initBleveOptions.func2({0x1dd4b60?, 0xc000796380}, {0xc0001a6680, 0x71}) |
cbft/cmd/cbft/init_bleve.go:110 +0xd1 |
github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).fireAsyncError(...)
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240213172332-8b9206aeab25/index/scorch/scorch.go:188 |
github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).mergerLoop.func1()
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240213172332-8b9206aeab25/index/scorch/merge.go:34 +0xaf |
panic({0x1509260?, 0xc0009c1cc8?}) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.6/go/src/runtime/panic.go:914 +0x21f |
github.com/blevesearch/zapx/v16.(*vectorIndexOpaque).mergeAndWriteVectorIndexes(0xc0007f8be0?, {0xc000938cd0, 0x2, 0xc000835760?}, {0xc000938ce0, 0x2, 0xc00024b720?}, 0xc000a4a160?, 0x40?) |
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v16@v16.0.7/section_faiss_vector_index.go:285 +0xd7d |
github.com/blevesearch/zapx/v16.(*faissVectorIndexSection).Merge(0x0?, 0xc0009ea1e0?, {0xc0009ea200, 0x2, 0x2?}, {0xc0009ea1e0?, 0x2?, 0x2?}, {0xc000a027e0, 0x11, ...}, ...) |
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v16@v16.0.7/section_faiss_vector_index.go:172 +0x345 |
github.com/blevesearch/zapx/v16.MergeToWriter({0xc0009ea200, 0x2, 0x2}, {0xc0009ea1e0, 0x2, 0x2}, 0x90?, 0xc000749a70?, 0x4145c5?) |
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v16@v16.0.7/merge.go:150 +0x7bc |
github.com/blevesearch/zapx/v16.mergeSegmentBases({0xc0009ea200, 0x2, 0x2}, {0xc0009ea1e0, 0x2, 0x2}, {0xc000a39050, 0x82}, 0x0?, 0xc00023d140, ...) |
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v16@v16.0.7/merge.go:77 +0x245 |
github.com/blevesearch/zapx/v16.(*ZapPlugin).Merge(0x1dccb88, {0xc000862580, 0x2, 0x0?}, {0xc0009ea1e0, 0x2, 0x2}, {0xc000a39050, 0x82}, 0xc00023d140, ...) |
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v16@v16.0.7/merge.go:52 +0x152 |
github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).planMergeAtSnapshot(0xc0005f4480, {0x1de3470?, 0x2c5db20}, 0x1?, 0xc000067630?) |
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240213172332-8b9206aeab25/index/scorch/merge.go:331 +0x716 |
github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).mergerLoop(0xc0005f4480) |
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240213172332-8b9206aeab25/index/scorch/merge.go:78 +0x28c |
created by github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).Open in goroutine 45 |
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240213172332-8b9206aeab25/index/scorch/scorch.go:206 +0x145 |
Attachments
Issue Links
- relates to
-
MB-60781 [Rebalance] : Rebalance exited with reason {service_rebalance_failed,fts,{agent_died,<35544.11597.3>,{lost_connection,{'ns_1@172.23.105.199',shutdown}}}}.
- Closed
-
MB-60816 [1536 dim]: 4million FTS docs with 120G total memory quota & 6 fts nodes. FTS crashed due to merger error.
- Closed