Details
-
Bug
-
Resolution: Fixed
-
Test Blocker
-
7.6.0
-
Enterprise Edition 7.6.0 build 18211 - Toy build
-
Untriaged
-
0
-
Unknown
Description
- Create a cluster with 1 kv, 2 fts nodes. Create magma bucket, 2 collections. Load 10k items with embeddings.
- Start a vector search query load.
- Do a rebalance IN of a fts node
- Do a rebalance OUT of a fts node
- Do a rebalance 2 IN & 1 OUT of a fts node. Fts crashed on .232 node
KV nodes in cluster: ['172.23.107.76']
CBAS nodes in cluster: []
INDEX nodes in cluster: []
FTS nodes in cluster: ['172.23.107.237', '172.23.107.126']
QUERY nodes in cluster: []
EVENTING nodes in cluster: []
Servers coming in : ['172.23.107.240', '172.23.107.232'] with services: ['fts', 'fts']
Servers going out : ['172.23.107.237']
- Rebalance failed
Node 232
Service 'fts' exited with status 2. Restarting. Messages:
/home/couchbase/jenkins/workspace/toy-unix-simple/cbft/protobuf/search.pb.go:949 +0xd0
github.com/couchbase/cbft.serverInterceptor({0x1435000, 0xc000524170}, {0x1dea630, 0xc0006f02d0}, 0xc00627ff38, 0x19f0910)
/home/couchbase/jenkins/workspace/toy-unix-simple/cbft/grpc_server.go:365 +0x278
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0005ba000, {0x1def060, 0xc005de4b60}, 0xc0063ac6c0, 0xc000594120, 0x27bf860, 0x0)
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1675 +0x12c7
google.golang.org/grpc.(*Server).handleStream(0xc0005ba000, {0x1def060, 0xc005de4b60}, 0xc0063ac6c0, 0x0)
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1755 +0x9a6
google.golang.org/grpc.(*Server).serveStreams.func1.1()
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:986 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 622
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:997 +0x145
Panic on 232:
panic: runtime error: index out of range [-1]
|
|
goroutine 623 [running]:
|
github.com/blevesearch/bleve/v2.distributeKNNHit(...)
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20231215161722-ba3fe9904583/search_knn.go:339
|
github.com/blevesearch/bleve/v2.redistributeKNNPreSearchData(0xc0006f0d20, {0xc000681090, 0x1, 0xc000512000?})
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20231215161722-ba3fe9904583/search_knn.go:367 +0x439
|
github.com/blevesearch/bleve/v2.(*indexAliasImpl).SearchInContext(0xc00626cdc0, {0x1de6f40, 0xc0065476b0}, 0xc0006f0d20)
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.11-0.20231215161722-ba3fe9904583/index_alias_impl.go:189 +0x49d
|
github.com/couchbase/cbft.(*SearchService).Search(0xc000524170, 0xc006392bd0, {0x1ded240?, 0xc0065142f0})
|
/home/couchbase/jenkins/workspace/toy-unix-simple/cbft/grpc_server.go:267 +0x1364
|
github.com/couchbase/cbft/protobuf._SearchService_Search_Handler({0x1435000?, 0xc000524170}, {0x1de9d48, 0xc0063ae280})
|
/home/couchbase/jenkins/workspace/toy-unix-simple/cbft/protobuf/search.pb.go:949 +0xd0
|
github.com/couchbase/cbft.serverInterceptor({0x1435000, 0xc000524170}, {0x1dea630, 0xc0006f02d0}, 0xc00627ff38, 0x19f0910)
|
/home/couchbase/jenkins/workspace/toy-unix-simple/cbft/grpc_server.go:365 +0x278
|
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0005ba000, {0x1def060, 0xc005de4b60}, 0xc0063ac6c0, 0xc000594120, 0x27bf860, 0x0)
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1675 +0x12c7
|
google.golang.org/grpc.(*Server).handleStream(0xc0005ba000, {0x1def060, 0xc005de4b60}, 0xc0063ac6c0, 0x0)
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1755 +0x9a6
|
google.golang.org/grpc.(*Server).serveStreams.func1.1()
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:986 +0xbb
|
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 622
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:997 +0x145
|