Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48024

[System Test] Index rebalance took 13+ hrs to complete

    XMLWordPrintable

Details

    • Bug
    • Resolution: User Error
    • Critical
    • 7.1.0
    • 7.0.2
    • qe

    Description

      Build : 7.0.2-6522
      Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale : 3
      Iteration : 1st

      Rebalance to failover one KV node and add 2 KV nodes is stuck for 7+ hrs in the index phase of the rebalance. This rebalance started at 2021-08-18T08:46:14-07:00

      Upon checking the getIndexStatus output, there is 1 index that is stuck in moving state.

      {
               "bucket" : "default",
               "collection" : "_default",
               "completion" : 100,
               "definition" : "CREATE INDEX `default_result` ON `default`(`result`) WITH {  \"defer_build\":true, \"nodes\":[ \"172.23.97.105:8091\",\"172.23.97.148:8091\" ] }",
               "defnId" : 4476407847701622807,
               "hosts" : [
                  "172.23.97.105:8091",
                  "172.23.97.148:8091"
               ],
               "indexName" : "default_result",
               "indexType" : "plasma",
               "instId" : 3902550320708673201,
               "lastScanTime" : "NA",
               "name" : "default_result",
               "numPartition" : 2,
               "numReplica" : 0,
               "partitionMap" : {
                  "172.23.97.105:8091" : [
                     0
                  ],
                  "172.23.97.148:8091" : [
                     0
                  ]
               },
               "partitioned" : false,
               "progress" : 100,
               "replicaId" : 0,
               "scheduled" : false,
               "scope" : "_default",
               "secExprs" : [
                  "`result`"
               ],
               "stale" : false,
               "status" : "Moving"
            }
      

      Not sure if this is the same root cause as MB-46582 or MB-46323, or if this is a different issue.

      Attached logs were collected around 11 AM on 8/18. Latest logs will be attached soon.

      Indexer nodes in cluster : 172.23.120.81, 172.23.96.243, 172.23.96.254, 172.23.97.105, 172.23.97.110, 172.23.97.148

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty