Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
Untriaged
-
-
1
-
No
Description
Build : 7.1.0-2515
Test : -test tests/2i/neo/test_neo_idx_clusterops_recovery.yml -scope tests/2i/neo/scope_neo_plasma_idx_dgm.yml
Scale : 2
Iteration : 2
On indexer node 172.23.107.2, indexer crashed with the following error :
2022-03-23T17:40:55.019-07:00 [Info] send progress stats to clients
|
2022-03-23T17:40:55.019-07:00 [Error] Indexer::processBuildDoneCatchup Fail to find real index instance 3055985754217342362
|
panic: Fail to find real index instance 3055985754217342362
|
|
|
goroutine 1 [running]:
|
github.com/couchbase/indexing/secondary/common.CrashOnError(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:464
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).processBuildDoneCatchup(0xc0014b0000, 0xc0027f0003, 0xc010846cc0, 0x16, 0x13c89, 0xc0080272c0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:5670 +0x170a
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).handleInitialBuildDone(0xc0014b0000, 0x15a0840, 0xc00b5414a0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:5544 +0x54d
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).handleWorkerMsgs(0xc0014b0000, 0x15a0840, 0xc00b5414a0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1316 +0xa91
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc0014b0000)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1079 +0x773
|
github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc00018aff0, 0x4, 0x13e3c59, 0xf, 0xc000187e78)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:510 +0x26e5
|
main.main()
|
goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:182 +0x1565
|
Initializing write barrier = 8000
|
2022-03-23T17:40:55.668-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem --caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=30468e9d5c984f2ed347fabe8e7376bd -isEnterprise=true]
|
In the test, node 172.23.107.2 was added at 2022-03-23T17:33:52 and a rebalance operation was in progress at this time. A test step is to kill indexer process on another node, so that rebalance fails and can be automatically retried. So, indexer process on 172.23.97.216 was killed at 2022-03-23T17:40:34 and rebalance failed at 2022-03-23T17:40:36.
This seems to be a regression or an edge case as the GSI component test has been run almost every week, but the issue was never seen. The most recent successful runs were with builds 7.1.0-2506 (RC2) and 7.1.0-2475 (RC1)
Attachments
Issue Links
- is a backport of
-
-
- Resolved
-