Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46582

[System Test] Rebalance in for 2 KV nodes took over 12 hours in indexer rebalance

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-5219 (RC2)
      Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale : 3
      Iteration : 2nd

      There is a step to add 2 KV nodes to the cluster in a rebalance operation. This rebalance took 12h 15m to complete, and just index rebalance took 12h 8m.

      Test console : 

      [2021-05-26T09:31:33-07:00, sequoiatools/couchbase-cli:7.0:cd12df] server-add -c 172.23.97.74:8091 --server-add https://172.23.121.77 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      		 
      [2021-05-26T09:32:35-07:00, sequoiatools/couchbase-cli:7.0:416b6d] server-add -c 172.23.97.74:8091 --server-add https://172.23.123.25 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      		 
      [2021-05-26T09:32:48-07:00, sequoiatools/couchbase-cli:7.0:0166b0] rebalance -c 172.23.97.74:8091 -u Administrator -p password
      		 
      [2021-05-26T21:47:23-07:00, sequoiatools/cmd:d082b2] 60

      Indexer nodes in the cluster : 172.23.120.81, 172.23.123.33, 172.23.96.243, 172.23.97.105, 172.23.97.110, 172.23.97.148

      The logs were collected just a few mins before the rebalance completed.

      Not sure if this is a regression, because the same symptom (slow rebalance) has been showing up in various tests, but the underlying issue could be different.

      Attachments

        For Gerrit Dashboard: MB-46582
        # Subject Branch Project Status CR V
        154191,4 MB-46582: Do not throttle LSS on recoveryLSS unstable plasma Status: MERGED +2 +1
        154921,6 MB-46582 Increase the size of smallSnapshotThreshold for INIT_STREAM unstable indexing Status: MERGED +2 +1
        155487,2 Disabling the legacy n1qlRBAC workload as it potentially causes MB-46582 master sequoia Status: MERGED +2 +1

        Activity

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty