Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46582

[System Test] Rebalance in for 2 KV nodes took over 12 hours in indexer rebalance

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-5219 (RC2)
      Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale : 3
      Iteration : 2nd

      There is a step to add 2 KV nodes to the cluster in a rebalance operation. This rebalance took 12h 15m to complete, and just index rebalance took 12h 8m.

      Test console :

      [2021-05-26T09:31:33-07:00, sequoiatools/couchbase-cli:7.0:cd12df] server-add -c 172.23.97.74:8091 --server-add https://172.23.121.77 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2021-05-26T09:32:35-07:00, sequoiatools/couchbase-cli:7.0:416b6d] server-add -c 172.23.97.74:8091 --server-add https://172.23.123.25 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2021-05-26T09:32:48-07:00, sequoiatools/couchbase-cli:7.0:0166b0] rebalance -c 172.23.97.74:8091 -u Administrator -p password
      [2021-05-26T21:47:23-07:00, sequoiatools/cmd:d082b2] 60
      

      Indexer nodes in the cluster : 172.23.120.81, 172.23.123.33, 172.23.96.243, 172.23.97.105, 172.23.97.110, 172.23.97.148

      The logs were collected just a few mins before the rebalance completed.

      Not sure if this is a regression, because the same symptom (slow rebalance) has been showing up in various tests, but the underlying issue could be different.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty