Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60241

FTS crashed on an incoming node during rebalance. Panic observed: panic: runtime error: index out of range [-1]

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 7.6.0
    • 7.6.0
    • fts
    • Enterprise Edition 7.6.0 build 18211 - Toy build

    Description

      1. Create a cluster with 1 kv, 2 fts nodes. Create magma bucket, 2 collections. Load 10k items with embeddings.
      2. Start a vector search query load.
      3. Do a rebalance IN of a fts node
      4. Do a rebalance OUT of a fts node
      5. Do a rebalance 2 IN & 1 OUT of a fts node. Fts crashed on .232 node

        KV nodes in cluster: ['172.23.107.76']
        CBAS nodes in cluster: []
        INDEX nodes in cluster: []
        FTS nodes in cluster: ['172.23.107.237', '172.23.107.126']
        QUERY nodes in cluster: []
        EVENTING nodes in cluster: []
        Servers coming in : ['172.23.107.240', '172.23.107.232'] with services: ['fts', 'fts']
        Servers going out : ['172.23.107.237']
        

      6. Rebalance failed

        Node 232

        Service 'fts' exited with status 2. Restarting. Messages:
        /home/couchbase/jenkins/workspace/toy-unix-simple/cbft/protobuf/search.pb.go:949 +0xd0
        github.com/couchbase/cbft.serverInterceptor({0x1435000, 0xc000524170}, {0x1dea630, 0xc0006f02d0}, 0xc00627ff38, 0x19f0910)
        /home/couchbase/jenkins/workspace/toy-unix-simple/cbft/grpc_server.go:365 +0x278
        google.golang.org/grpc.(*Server).processStreamingRPC(0xc0005ba000, {0x1def060, 0xc005de4b60}, 0xc0063ac6c0, 0xc000594120, 0x27bf860, 0x0)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1675 +0x12c7
        google.golang.org/grpc.(*Server).handleStream(0xc0005ba000, {0x1def060, 0xc005de4b60}, 0xc0063ac6c0, 0x0)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1755 +0x9a6
        google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:986 +0xbb
        created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 622
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:997 +0x145
        

      Panic on 232:

      panic: runtime error: index out of range [-1]
       
      goroutine 623 [running]:
      github.com/blevesearch/bleve/v2.distributeKNNHit(...)
              /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20231215161722-ba3fe9904583/search_knn.go:339
      github.com/blevesearch/bleve/v2.redistributeKNNPreSearchData(0xc0006f0d20, {0xc000681090, 0x1, 0xc000512000?})
              /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20231215161722-ba3fe9904583/search_knn.go:367 +0x439
      github.com/blevesearch/bleve/v2.(*indexAliasImpl).SearchInContext(0xc00626cdc0, {0x1de6f40, 0xc0065476b0}, 0xc0006f0d20)
              /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20231215161722-ba3fe9904583/index_alias_impl.go:189 +0x49d
      github.com/couchbase/cbft.(*SearchService).Search(0xc000524170, 0xc006392bd0, {0x1ded240?, 0xc0065142f0})
              /home/couchbase/jenkins/workspace/toy-unix-simple/cbft/grpc_server.go:267 +0x1364
      github.com/couchbase/cbft/protobuf._SearchService_Search_Handler({0x1435000?, 0xc000524170}, {0x1de9d48, 0xc0063ae280})
              /home/couchbase/jenkins/workspace/toy-unix-simple/cbft/protobuf/search.pb.go:949 +0xd0
      github.com/couchbase/cbft.serverInterceptor({0x1435000, 0xc000524170}, {0x1dea630, 0xc0006f02d0}, 0xc00627ff38, 0x19f0910)
              /home/couchbase/jenkins/workspace/toy-unix-simple/cbft/grpc_server.go:365 +0x278
      google.golang.org/grpc.(*Server).processStreamingRPC(0xc0005ba000, {0x1def060, 0xc005de4b60}, 0xc0063ac6c0, 0xc000594120, 0x27bf860, 0x0)
              /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1675 +0x12c7
      google.golang.org/grpc.(*Server).handleStream(0xc0005ba000, {0x1def060, 0xc005de4b60}, 0xc0063ac6c0, 0x0)
              /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1755 +0x9a6
      google.golang.org/grpc.(*Server).serveStreams.func1.1()
              /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:986 +0xbb
      created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 622
              /home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:997 +0x145
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            1 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty