Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60352

FTS rebalance operations are failing due to crash in scorch.IndexSnapshotVectorReader

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 7.6.0
    • 7.6.0
    • fts
    • Enterprise Edition 7.6.0 build 2001

    Description

      QE Test

      ./sequoia -client 172.23.104.254:2375 -provider file:centos_third_cluster.yml -test tests/fts/cheshire-cat/test_fts_clusterops_coll_crud_magma_vector_search.yml -scope tests/fts/cheshire-cat/scope_fts_magma_vector_search.yml -scale 3 -repeat 0 -log_level 0 -version 7.6.0-2001 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
      

      All FTS rebalance operations are failing due to crash in indexImpl.SearchInContext hence marking as test blocker.

      FTS service crash

      2024-01-11T18:32:25.504-08:00, ns_log:0:info:message(ns_1@172.23.108.145) - Service 'fts' exited with status 2. Restarting. Messages:
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/index_impl.go:440 +0x49
      github.com/blevesearch/bleve/v2.(*indexImpl).SearchInContext(0xc01ab41260, {0x1de6ea0, 0xc02096ac00}, 0xc020b8c2d0)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/index_impl.go:479 +0x2a5
      github.com/couchbase/cbft.(*cacheBleveIndex).SearchInContext(0xc02096ab70, {0x1de6ea0?, 0xc02096ac00?}, 0x0?)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/cache_bleve.go:125 +0x31b
      github.com/blevesearch/bleve/v2.preSearchDataSearch.func1({0x1df5db0, 0xc02096ab70}, 0x1?)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/index_alias_impl.go:558 +0xee
      created by github.com/blevesearch/bleve/v2.preSearchDataSearch in goroutine 1825
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/index_alias_impl.go:565 +0x125
      

      Panic on FTS node 172.23.108.136

      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x7821ed]
       
      goroutine 1712 [running]:
      github.com/blevesearch/bleve/v2/index/scorch.(*IndexSnapshotVectorReader).Next(0xc000ae09a0, 0x1387820?)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/index/scorch/snapshot_index_vr.go:80 +0x6d
      github.com/blevesearch/bleve/v2/search/searcher.(*KNNSearcher).Next(0xc01a37aa00, 0xc01a866398?)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/search/searcher/search_knn.go:106 +0xc8
      github.com/blevesearch/bleve/v2/search/searcher.(*DisjunctionSliceSearcher).initSearchers(0xc01a85e840, 0xc01a85d290)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/search/searcher/search_disjunction_slice.go:152 +0xe7
      github.com/blevesearch/bleve/v2/search/searcher.(*DisjunctionSliceSearcher).Next(0xc01a85e840, 0xc01a85d290)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/search/searcher/search_disjunction_slice.go:215 +0x38
      github.com/blevesearch/bleve/v2/search/collector.(*KNNCollector).Collect(0xc01a859300, {0x1de6ea0, 0xc01a85d020}, {0x1deeb40, 0xc01a85e840}, {0x1df1b00?, 0xc000851780})
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/search/collector/knn.go:181 +0x304
      github.com/blevesearch/bleve/v2.(*indexImpl).runKnnCollector(0xc0006f16c0, {0x1de6ea0, 0xc01a85d020}, 0xc0006193b0, {0x1df1b00, 0xc000851780}, 0x1)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/search_knn.go:261 +0x177
      github.com/blevesearch/bleve/v2.(*indexImpl).preSearch(0xc01a85d020?, {0x1de6ea0?, 0xc01a85d020?}, 0x33b?, {0x1df1b00?, 0xc000851780?})
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/index_impl.go:440 +0x49
      github.com/blevesearch/bleve/v2.(*indexImpl).SearchInContext(0xc0006f16c0, {0x1de6ea0, 0xc01a85d020}, 0xc0006193b0)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/index_impl.go:479 +0x2a5
      github.com/couchbase/cbft.(*cacheBleveIndex).SearchInContext(0xc01a85cfc0, {0x1de6ea0?, 0xc01a85d020?}, 0x1388180?)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/cache_bleve.go:125 +0x31b
      github.com/blevesearch/bleve/v2.preSearchDataSearch.func1({0x1df5db0, 0xc01a85cfc0}, 0x2?)
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/index_alias_impl.go:558 +0xee
      created by github.com/blevesearch/bleve/v2.preSearchDataSearch in goroutine 1710
      	/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20240110165502-473beac8c5eb/index_alias_impl.go:565 +0x125
      

      Note

      I have reopened MB-60241 for the other panic seen on this cluster.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sujay.gad Sujay Gad
            sujay.gad Sujay Gad
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty