Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46056

[System Test] Index rebalance stuck due to 1 index in moving state

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-5075
      Test : -test tests/2i/cheshirecat/test_idx_clusterops_cheshire_cat_moi.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat_moi.yml
      Scale : 2
      Iteration : 3rd

      If this issue is similar to MB-45631, this ticket can be closed and MB-45631 can be reopened again.

      In the MOI system test, a rebalance operation to remove 2 indexer nodes started at 2021-05-02T14:33:26, and is stuck for ~8 hrs now.

      [2021-05-02T14:33:26-07:00, sequoiatools/couchbase-cli:7.0:4a28e9] rebalance -c 172.23.97.215:8091 --server-remove 172.23.97.216,172.23.107.2 -u Administrator -p password

      The following index is stuck in Moving state :

      {
               "bucket" : "bucket2",
               "progress" : 100,
               "partitionMap" : {
                  "172.23.107.2:8091" : [
                     4
                  ],
                  "172.23.107.4:8091" : [
                     2
                  ],
                  "172.23.107.3:8091" : [
                     4
                  ],
                  "172.23.97.216:8091" : [
                     3,
                     1
                  ]
               },
               "completion" : 100,
               "numReplica" : 2,
               "defnId" : 7271921765741110904,
               "indexType" : "memory_optimized",
               "numPartition" : 5,
               "scheduled" : false,
               "collection" : "coll_4",
               "partitioned" : true,
               "indexName" : "idx1_fvPdGxM",
               "definition" : "CREATE INDEX `idx1_fvPdGxM` ON `bucket2`.`scope_3`.`coll_4`(`country`,(distinct (array ((`r`.`ratings`).`Check in / front desk`) for `r` in `reviews` end)),array_count(`public_likes`),array_count(`reviews`) DESC,`type`,`phone`,`price`,`email`,`address`,`name`,`url`) PARTITION BY hash((meta().`id`)) WITH {  \"defer_build\":true, \"nodes\":[ \"172.23.107.2:8091\",\"172.23.107.3:8091\",\"172.23.107.4:8091\",\"172.23.107.5:8091\",\"172.23.97.216:8091\" ], \"num_replica\":2, \"num_partition\":4 }",
               "name" : "idx1_fvPdGxM (replica 2)",
               "hosts" : [
                  "172.23.107.2:8091",
                  "172.23.107.3:8091",
                  "172.23.107.4:8091",
                  "172.23.97.216:8091"
               ],
               "instId" : 16303863785724576625,
               "status" : "Moving",
               "secExprs" : [
                  "`country`",
                  "(distinct (array ((`r`.`ratings`).`Check in / front desk`) for `r` in `reviews` end))",
                  "array_count(`public_likes`)",
                  "array_count(`reviews`)",
                  "`type`",
                  "`phone`",
                  "`price`",
                  "`email`",
                  "`address`",
                  "`name`",
                  "`url`"
               ],
               "replicaId" : 2,
               "lastScanTime" : "Sun May  2 22:28:20 PDT 2021",
               "stale" : false,
              "scope" : "scope_3"
            }
      

      Full output of /getIndexStatus is attached. 107.2_gis.json

      All indexer nodes : 172.23.107.2, 172.23.107.3, 172.23.107.4, 172.23.107.5, 172.23.97.216

      Latest logs (from around 5/2 8.30 PM are attached)

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              deepkaran.salooja Deepkaran Salooja
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty