Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.0
-
7.6.0-2052
-
Triaged
-
0
-
Unknown
Description
There have been quite a few rebalance failures with a similar error -
Seen during iteration iteration 22
[user:error,2024-01-25T18:00:39.823-08:00,ns_1@172.23.97.67:<0.22016.0>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index, |
{worker_died,
|
{'EXIT',<0.15135.308>, |
{task_failed,rebalance,
|
{service_error,
|
<<"Error creating slice Unable to initialize /data/@2i/bucket1_idx8_smeBu_12604540048321060787_3.index/mainIndex, err = Cannot find an available shard, all candidate shards (num: 1) are under transfer or restore, retry later">>}}}}}. |
|
|
[user:error,2024-01-25T18:08:12.030-08:00,ns_1@172.23.97.67:<0.22016.0>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index, |
{worker_died,
|
{'EXIT',<0.31589.310>, |
{task_failed,rebalance,
|
{service_error,
|
<<"Error creating slice Unable to initialize /data/@2i/bucket1_idx8_smeBu_12604540048321060787_3.index/mainIndex, err = Cannot find an available shard, all candidate shards (num: 1) are under transfer or restore, retry later">>}}}}}. |
Rebalance Operation Id = ec5693f15bfb725e2009b681bee8850b
|
cbcollect ->
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1706253279/collectinfo-2024-01-26T072607-ns_1%40172.23.97.67.zip
Unsure if it has anything to do with the earlier issue https://issues.couchbase.com/browse/MB-60575 filed on the same run.
Attachments
| For Gerrit Dashboard: MB-60579 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 204677,3 | MB-60579 Process ShardTokenRecoveryShard for master and source nodes | unstable | indexing | Status: MERGED | +2 | +1 |
| 204833,1 | Merging fixes for MB-60575, MB-60579, MB-60585, MB-60521 | master | indexing | Status: MERGED | +2 | +1 |