Details
-
Bug
-
Resolution: Won't Fix
-
Critical
-
5.5.0
-
Untriaged
-
-
Unknown
Description
Build : 5.5.0-2884
Test : GSI component test : -test tests/2i/test_idx_rebalance_replica_vulcan_kv_opt.yml -scope tests/2i/scope_idx_rebalance_replica_vulcan.yml
Scale : 3
Iteration : 3rd iteration (~20 hrs of run)
In the 3rd iteration, the step in the test to rebalance out a KV node fails when memcached crashes on the master node. The rebalance operation after that is stuck with indexer nodes not completing rebalance. Index building is stuck on all indexer nodes. The following messages can be seen in the indexer logs:
2018-06-12T00:08:28.582-07:00 [Info] Timekeeper::checkInitStreamReadyToMerge MAINT_STREAM In Repair.INIT_STREAM cannot be merged. Continue both streams for bucket other-3.
|
2018-06-12T00:08:28.587-07:00 [Info] Timekeeper::checkInitStreamReadyToMerge MAINT_STREAM In Repair.INIT_STREAM cannot be merged. Continue both streams for bucket other-3.
|
2018-06-12T00:08:28.651-07:00 [Error] KVSender::restartVbuckets MAINT_STREAM default Error in fetching cluster info Node is not a member of bucket: 172.23.104.17:8091
|
2018-06-12T00:08:28.657-07:00 [Error] Timekeeper::sendRestartMsg Error Response from KV Node is not a member of bucket: 172.23.104.17:8091 For Request
|
...
|
...
|
2018-06-12T00:08:28.677-07:00 [Info] StreamState::getRepairTsForBucket
|
Bucket other-2 StreamId MAINT_STREAM Vbucket 342 repair is being retried for 6 times.
|
2018-06-12T00:08:28.677-07:00 [Info] StreamState::getRepairTsForBucket
|
Bucket other-2 StreamId MAINT_STREAM Vbucket 343 repair is being retried for 6 times.
|
2018-06-12T00:08:28.677-07:00 [Info] StreamState::getRepairTsForBucket
|
Bucket other-2 StreamId MAINT_STREAM Vbucket 344 repair is being retried for 6 times.
|
2018-06-12T00:08:28.677-07:00 [Info] StreamState::getRepairTsForBucket
|