Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30376

[System Test] Indexes being rebalanced even if the rebalance operation is for a data node

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.5.0
    • 5.5.0
    • secondary-index
    • centos2 cluster

    Description

      Build : 5.5.0-2954 (RC3)
      Test : -test tests/2i/test_idx_rebalance_replica_vulcan_kv_opt.yml -scope tests/2i/scope_idx_rebalance_replica_vulcan_new.yml
      Iteration : 2 (~15 hrs of test run)

      In the system test, there are steps to rebalance in and out the data nodes. As per the fix for MB-29937 (https://github.com/couchbase/indexing/commit/933727fc4a114bb9e73e30f26072af5a6d0a69b5), If the rebalance operation doesnt involve an indexer node, the indexes should not be rebalanced. But in this case, the indexes are still getting rebalanced and replicated. The replication also is either stuck or very slow, causing the rebalance to be stuck at 99.1% since 10+ hrs now.

      Supportal link : http://supportal.couchbase.com/snapshot/ed4575668dc70f54449225a9f04e9441::0

      The indexer logs are filled with following errors :
      018-07-04T18:12:36.831-07:00 [Info] StreamState::getRepairTsForBucket
      Bucket default StreamId MAINT_STREAM previous repair fails. Retry using previous repairTs
      2018-07-04T18:12:36.831-07:00 [Info] StreamState::adjustNonSnapAlignedVbs
      Bucket default StreamId MAINT_STREAM Vbuckets [956].
      2018-07-04T18:12:36.912-07:00 [Error] KVSender::restartVbuckets MAINT_STREAM default Error in fetching cluster info Node is not a member of bucket: 172.23.96.207:8091
      2018-07-04T18:12:36.977-07:00 [Info] default/default_result_ratings_claims_pindex/Backstore#13033604016120019334:1 Plasma: logCleaner: starting... frag 31, data: 18426522, used: 26708277 log:(689325327 - 716484608)

      Here, 172.23.96.207 is the kv node that is being rebalanced out.

      Marking this issue as a regression since we had successfully completed 3 iterations of the same test on RC1 build.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-30457 backport MB-30376 - [System Test] Indexes being rebalanced even if the rebalance operation is for a data node

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-30376
          # Subject Branch Project Status CR V
          96635,1 MB-30376 exclude projectors not owning a vbucket during restartVb unstable indexing Status: MERGED +2 +1
          97211,1 MB-30457 exclude projectors not owning a vbucket during restartVb spock indexing Status: MERGED +2 +1

          Activity

            People

              deepkaran.salooja Deepkaran Salooja
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty