Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46869

[System Test] Index swap rebalance stuck at 72%

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-5277
      Test : -test tests/2i/cheshirecat/test_idx_clusterops_cheshire_cat_recovery.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat_dgm.yml
      Scale : 2
      Iteration : 4th
      Day : 2nd

      The test has a step to swap rebalance a pair of index nodes - add 172.23.97.216 and remove 172.23.97.217. The rebalance started at 2021-06-09T04:02:02. Then indexer on 172.23.97.216 was killed at 2021-06-09T04:08:31. So, the rebalance failed. This rebalance was automatically retried after 5 mins. It appears that the rebalance is stuck since then with 72% progress.

      curl  -u Administrator:password 172.23.97.215:8091/pools/default/rebalanceProgress | jq
      {
        "status": "running",
        "ns_1@172.23.97.232": {
          "progress": 1
        },
        "ns_1@172.23.107.2": {
          "progress": 0.7222222222222223
        },
        "ns_1@172.23.107.3": {
          "progress": 0.7222222222222223
        },
        "ns_1@172.23.107.4": {
          "progress": 0.7222222222222223
        },
        "ns_1@172.23.97.235": {
          "progress": 1
        },
        "ns_1@172.23.97.215": {
          "progress": 1
        },
        "ns_1@172.23.107.5": {
          "progress": 0.7222222222222223
        },
        "ns_1@172.23.97.216": {
          "progress": 0.7222222222222223
        },
        "ns_1@172.23.97.236": {
          "progress": 0
        },
        "ns_1@172.23.97.237": {
          "progress": 1
        },
        "ns_1@172.23.97.217": {
          "progress": 0.7222222222222223
        },
        "ns_1@172.23.97.227": {
          "progress": 0
        }
      }

      The MetadataStore files from 172.23.97.216 are attached.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              srinath.duvuru Srinath Duvuru
              deepkaran.salooja Deepkaran Salooja
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty