Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45631

[System Test] Index rebalance stuck due to 1 index in moving state

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-4916
      Test : -test tests/2i/cheshirecat/test_idx_clusterops_cheshire_cat.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat_dgm.yml
      Scale : 2
      Iteration : 2nd

      This issue could be similar to MB-45329 or MB-44976, or could be a new issue. But since both tickets are resolved, I have opened up a new ticket.

      In the second iteration of the test, there is a step to rebalance out 2 indexer nodes.

      [2021-04-12T18:55:47-07:00, sequoiatools/couchbase-cli:7.0:87607c] rebalance -c 172.23.97.215:8091 --server-remove 172.23.97.216,172.23.107.2 -u Administrator -p password
      

      This rebalance operation is stuck as 1 index `idx2_9WLDF75` is stuck in moving state. These are the details of the index from getIndexStatus. The full output of getIndexStatus is attached.

      {
               "lastScanTime" : "NA",
               "defnId" : 16478650294981984358,
               "indexName" : "idx2_9WLDF75",
               "partitionMap" : {
                  "172.23.107.5:8091" : [
                     2,
                     1
                  ],
                  "172.23.97.216:8091" : [
                     1,
                     2
                  ]
               },
               "progress" : 100,
               "partitioned" : true,
               "replicaId" : 0,
               "name" : "idx2_9WLDF75",
               "numPartition" : 4,
               "instId" : 7281156889785812375,
               "collection" : "coll_3",
               "hosts" : [
                  "172.23.107.5:8091",
                  "172.23.97.216:8091"
               ],
               "secExprs" : [
                  "`free_breakfast`",
                  "`type`",
                  "`free_parking`",
                  "array_count(`public_likes`)",
                  "`price`",
                  "`country`"
               ],
               "bucket" : "bucket3",
               "stale" : false,
               "indexType" : "plasma",
               "scope" : "scope_0",
               "numReplica" : 1,
               "definition" : "CREATE INDEX `idx2_9WLDF75` ON `bucket3`.`scope_0`.`coll_3`(`free_breakfast`,`type`,`free_parking`,array_count(`public_likes`),`price`,`country`) PARTITION BY hash((meta().`id`)) WITH {  \"defer_build\":true, \"nodes\":[ \"172.23.107.3:8091\",\"172.23.107.5:8091\",\"172.23.97.216:8091\" ], \"num_replica\":1, \"num_partition\":2 }",
               "scheduled" : false,
               "completion" : 100,
               "status" : "Moving"
            }
      

      Attached logs are from around 2021-04-12 21:24. Let me know if you need logs from before or after.

      Indexer nodes : 172.23.107.2, 172.23.107.3, 172.23.107.4, 172.23.107.5, 172.23.97.216

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              deepkaran.salooja Deepkaran Salooja
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty