Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.0.0-3291
Test : -test tests/2i/test_idx_clusterops_cheshire_cat.yml -scope tests/2i/scope_idx_cheshire_cat.yml
Scale : 2
Iteration : 1st
This is a new system test with indexes on collections, and cluster ops. There are 20 scopes, 200 collections, and ~1000 index instances in the cluster. The first rebalance operation was to add a new index node to the cluster. This failed after ~12 mins of rebalance as indexer on 172.23.96.31 crashed with the following error :
Stack trace :
2020-10-07T19:43:01.702-07:00 [Info] TK StreamBegin INIT_STREAM bucket2:scope_6:coll_-8 786 235805425695305 0 4
|
2020-10-07T19:43:01.706-07:00 [Info] Rebalancer::waitForIndexBuild Index bucket1:idx2 (replica 3): State INDEX_STATE_CATCHUP Pending 597 EstTime 0
|
2020-10-07T19:43:01.706-07:00 [Info] Rebalancer::waitForIndexBuild Index bucket2:idx1: State INDEX_STATE_INITIAL Pending 1.1005699e+07 EstTime 517
|
panic: Timestamp mismatch between snapshot
|
target bucket: bucket3, scopeId: , collectionId: , vbuckets: 1024 Crc64: 15138562774735479631 snapType INMEM_SNAP -
|
{vbno, vbuuid, manifest, seqno, snapshot-start, snapshot-end}
|
{ 0 fda43f9cf56e 3b 16719 16719 16719}
|
{ 1 9c48e1b7ae45 3b 16597 16597 16597}
|
...
|
...
|
goroutine 15996303 [running]:
|
github.com/couchbase/indexing/secondary/common.CrashOnError(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:413
|
github.com/couchbase/indexing/secondary/indexer.(*Rebalancer).tokenMergeOrReady.func1(0xc005d1b4a0, 0xc00679b194, 0x24, 0xc018fe3980, 0x20, 0xc018fe39a0, 0x20, 0xc018fe39c0, 0x20, 0xc018fe39e0, ...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalancer.go:1126 +0x4d8
|
created by github.com/couchbase/indexing/secondary/indexer.(*Rebalancer).tokenMergeOrReady
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalancer.go:1096 +0x10a
|