Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.0
-
7.6.0-2095
-
Untriaged
-
0
-
Unknown
Description
Rebalance failures of interest -
Seen during iteration 65 -
[user:error,2024-02-07T05:05:43.592-08:00,ns_1@172.23.97.67:<0.10153.366>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index, |
{agent_died,<34855.11419.1620>, |
{linked_process_died,<34855.13063.1620>, |
{'ns_1@172.23.97.108', |
{timeout,
|
{gen_server,call,
|
[<34855.9403.1620>, |
{call,"ServiceAPI.GetTaskList", |
#Fun<json_rpc_connection.0.36915653>, |
#{timeout => 60000}}, |
60000]}}}}}}. |
Rebalance Operation Id = 96cf5475af598d1eb0d00b34e2bb1e38
|
|
[user:error,2024-02-07T05:08:09.104-08:00,ns_1@172.23.97.67:<0.10153.366>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index, |
{agent_died,<34859.13197.1364>, |
{linked_process_died,<34859.26957.1567>, |
{'ns_1@172.23.106.171', |
{timeout,
|
{gen_server,call,
|
[<34859.8648.1364>, |
{call,"ServiceAPI.StartTopologyChange", |
#Fun<json_rpc_connection.0.36915653>, |
#{timeout => 60000}}, |
60000]}}}}}}. |
|
[user:error,2024-02-07T07:37:46.598-08:00,ns_1@172.23.97.67:<0.10153.366>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index, |
{worker_died,
|
{'EXIT',<0.3945.1638>, |
{task_failed,rebalance,
|
{service_error,
|
<<"shard copy aborted: LSS /data/@2i/shards/shard6124636750988113296/data metadata copy failed :config.json copy failed src :/data/@2i/shards/shard6124636750988113296/config.json dst :https://172.23.106.30:9104//plasma_storage_v1/7b1871d5533d3945a3bc73a93c51704e_ShardTokenbf_c7_1f_16_95_4_5c_a4/6124636750988113296/shards/shard6124636750988113296/config.json error :CopyBytes error : rpc remote file not open, shardId :6124636750988113296">>}}}}}. |
Rebalance Operation Id = 82584f2818e09e8f0b2b39160324f562
|
|
|
Rest of the rebalance failures are either test-induced or already have bugs associated with them.
Iteration 66 -
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707319962/collectinfo-2024-02-07T160007-ns_1%40172.23.97.67.zip
Iteration 65 -
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707314201/collectinfo-2024-02-07T143243-ns_1%40172.23.97.67.zip
Iteration 64 -
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707308284/collectinfo-2024-02-07T125644-ns_1%40172.23.97.67.zip
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707303043/collectinfo-2024-02-07T111805-ns_1%40172.23.97.67.zip