Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60579

[System Test] Rebalance failure - Cannot find an available shard, all candidate shards (num: 1) are under transfer or restore

    XMLWordPrintable

Details

    • Triaged
    • 0
    • Unknown

    Description

      There have been quite a few rebalance failures with a similar error -

      Seen during iteration iteration 22

      [user:error,2024-01-25T18:00:39.823-08:00,ns_1@172.23.97.67:<0.22016.0>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index,
                                    {worker_died,
                                     {'EXIT',<0.15135.308>,
                                      {task_failed,rebalance,
                                       {service_error,
                                        <<"Error creating slice Unable to initialize /data/@2i/bucket1_idx8_smeBu_12604540048321060787_3.index/mainIndex, err = Cannot find an available shard, all candidate shards (num: 1) are under transfer or restore, retry later">>}}}}}.
       
      [user:error,2024-01-25T18:08:12.030-08:00,ns_1@172.23.97.67:<0.22016.0>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index,
                                    {worker_died,
                                     {'EXIT',<0.31589.310>,
                                      {task_failed,rebalance,
                                       {service_error,
                                        <<"Error creating slice Unable to initialize /data/@2i/bucket1_idx8_smeBu_12604540048321060787_3.index/mainIndex, err = Cannot find an available shard, all candidate shards (num: 1) are under transfer or restore, retry later">>}}}}}.
      Rebalance Operation Id = ec5693f15bfb725e2009b681bee8850b
      

      cbcollect ->

      Cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.105.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.106.171.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.106.176.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.106.30.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.97.108.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.97.109.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.97.67.zip

      Unsure if it has anything to do with the earlier issue https://issues.couchbase.com/browse/MB-60575 filed on the same run.

      Attachments

        Activity

          People

            pavan.pb Pavan PB
            pavan.pb Pavan PB
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty