Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.5.0
-
centos2 cluster
-
Untriaged
-
-
Unknown
Description
Build : 5.5.0-2954 (RC3)
Test : -test tests/2i/test_idx_rebalance_replica_vulcan_kv_opt.yml -scope tests/2i/scope_idx_rebalance_replica_vulcan_new.yml
Iteration : 2 (~15 hrs of test run)
In the system test, there are steps to rebalance in and out the data nodes. As per the fix for MB-29937 (https://github.com/couchbase/indexing/commit/933727fc4a114bb9e73e30f26072af5a6d0a69b5), If the rebalance operation doesnt involve an indexer node, the indexes should not be rebalanced. But in this case, the indexes are still getting rebalanced and replicated. The replication also is either stuck or very slow, causing the rebalance to be stuck at 99.1% since 10+ hrs now.
Supportal link : http://supportal.couchbase.com/snapshot/ed4575668dc70f54449225a9f04e9441::0
The indexer logs are filled with following errors :
018-07-04T18:12:36.831-07:00 [Info] StreamState::getRepairTsForBucket
Bucket default StreamId MAINT_STREAM previous repair fails. Retry using previous repairTs
2018-07-04T18:12:36.831-07:00 [Info] StreamState::adjustNonSnapAlignedVbs
Bucket default StreamId MAINT_STREAM Vbuckets [956].
2018-07-04T18:12:36.912-07:00 [Error] KVSender::restartVbuckets MAINT_STREAM default Error in fetching cluster info Node is not a member of bucket: 172.23.96.207:8091
2018-07-04T18:12:36.977-07:00 [Info] default/default_result_ratings_claims_pindex/Backstore#13033604016120019334:1 Plasma: logCleaner: starting... frag 31, data: 18426522, used: 26708277 log:(689325327 - 716484608)
Here, 172.23.96.207 is the kv node that is being rebalanced out.
Marking this issue as a regression since we had successfully completed 3 iterations of the same test on RC1 build.
Attachments
Issue Links
- relates to
-
MB-30457 backport MB-30376 - [System Test] Indexes being rebalanced even if the rebalance operation is for a data node
- Closed