Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60808

[FTS] : FTS service with 10k documents crashing with [FATA] scorch AsyncError, path: /opt/couchbase/var/lib/couchbase/data/@fts/test_bucket._default.test_index_22a14bd6eebde5b2_aa574717.pindex/store, treating thisasfatal,err: merger panic when processing

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.6.0
    • 7.6.0
    • fts
    • Couchbase Enterprise Edition 7.6.0 build 2140

    Description

      1. Created a cluster with 3 kv nodes and 3 fts nodes
      2. Created a magma bucket named "test_bucket"
      3. Created a vector index - definition attached
      4. Started loading data with different dimension vector in field "vec" - A snapshot of dataset attached
      5. Ran a query - Query attached

      FTS service crashed

       

      2024-02-15T06:52:53.245-08:00 [FATA] scorch AsyncError, path: /opt/couchbase/var/lib/couchbase/data/@fts/test_bucket._default.test_index_22a14bd6eebde5b2_aa574717.pindex/store, treating this as fatal, err: merger panic when processing /opt/couchbase/var/lib/couchbase/data/@fts/test_bucket._default.test_index_22a14bd6eebde5b2_aa574717.pindex/store, stack dump: /opt/couchbase/var/lib/couchbase/data/@fts/dumps/1708008773.fts.stack.dump.txt -- main.initBleveOptions.func2() at init_bleve.go:113 

      Observe this stacktrace from /opt/couchbase/var/lib/couchbase/data/@fts/dumps/1708008773.fts.stack.dump.txt

       

      goroutine 147 [running]:
      runtime/pprof.writeGoroutineStacks({0x1dd45a0, 0xc00071c2c0})
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.6/go/src/runtime/pprof/pprof.go:703 +0x6a
      runtime/pprof.writeGoroutine({0x1dd45a0?, 0xc00071c2c0?}, 0xc000800b00?)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.6/go/src/runtime/pprof/pprof.go:692 +0x25
      runtime/pprof.(*Profile).WriteTo(0x15d2d01?, {0x1dd45a0?, 0xc00071c2c0?}, 0x121?)
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.6/go/src/runtime/pprof/pprof.go:329 +0x146
      main.dumpStack({0x7ffe418c3364?, 0x24?}, {0xc0001a2360, 0x11f})
      	cbft/cmd/cbft/stack_dump.go:59 +0x47e
      main.initBleveOptions.func2({0x1dd4b60?, 0xc000796380}, {0xc0001a6680, 0x71})
      	cbft/cmd/cbft/init_bleve.go:110 +0xd1
      github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).fireAsyncError(...)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240213172332-8b9206aeab25/index/scorch/scorch.go:188
      github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).mergerLoop.func1()
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240213172332-8b9206aeab25/index/scorch/merge.go:34 +0xaf
      panic({0x1509260?, 0xc0009c1cc8?})
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.6/go/src/runtime/panic.go:914 +0x21f
      github.com/blevesearch/zapx/v16.(*vectorIndexOpaque).mergeAndWriteVectorIndexes(0xc0007f8be0?, {0xc000938cd0, 0x2, 0xc000835760?}, {0xc000938ce0, 0x2, 0xc00024b720?}, 0xc000a4a160?, 0x40?)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v16@v16.0.7/section_faiss_vector_index.go:285 +0xd7d
      github.com/blevesearch/zapx/v16.(*faissVectorIndexSection).Merge(0x0?, 0xc0009ea1e0?, {0xc0009ea200, 0x2, 0x2?}, {0xc0009ea1e0?, 0x2?, 0x2?}, {0xc000a027e0, 0x11, ...}, ...)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v16@v16.0.7/section_faiss_vector_index.go:172 +0x345
      github.com/blevesearch/zapx/v16.MergeToWriter({0xc0009ea200, 0x2, 0x2}, {0xc0009ea1e0, 0x2, 0x2}, 0x90?, 0xc000749a70?, 0x4145c5?)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v16@v16.0.7/merge.go:150 +0x7bc
      github.com/blevesearch/zapx/v16.mergeSegmentBases({0xc0009ea200, 0x2, 0x2}, {0xc0009ea1e0, 0x2, 0x2}, {0xc000a39050, 0x82}, 0x0?, 0xc00023d140, ...)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v16@v16.0.7/merge.go:77 +0x245
      github.com/blevesearch/zapx/v16.(*ZapPlugin).Merge(0x1dccb88, {0xc000862580, 0x2, 0x0?}, {0xc0009ea1e0, 0x2, 0x2}, {0xc000a39050, 0x82}, 0xc00023d140, ...)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v16@v16.0.7/merge.go:52 +0x152
      github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).planMergeAtSnapshot(0xc0005f4480, {0x1de3470?, 0x2c5db20}, 0x1?, 0xc000067630?)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240213172332-8b9206aeab25/index/scorch/merge.go:331 +0x716
      github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).mergerLoop(0xc0005f4480)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240213172332-8b9206aeab25/index/scorch/merge.go:78 +0x28c
      created by github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).Open in goroutine 45
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240213172332-8b9206aeab25/index/scorch/scorch.go:206 +0x145 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              thejas.orkombu Thejas Orkombu
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty