Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
7.6.0
-
7.6.0-2132
-
Untriaged
-
0
-
Unknown
Description
Rebalance hasn't completed and appears to be stuck at 99%. The rebalance has been running for almost 83 hours now -
{"stageInfo":{"index":{"totalProgress":99,"perNodeProgress":{"ns_1@172.23.106.171":0.99,"ns_1@172.23.105.122":0.99,"ns_1@172.23.106.176":0.99,"ns_1@172.23.97.108":0.99,"ns_1@172.23.97.109":0.99},"startTime":"2024-02-16T09:09:28.979-08:00","completedTime":false,"timeTaken":1669674},"data":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.96.230":1,"ns_1@172.23.96.245":1,"ns_1@172.23.97.66":1,"ns_1@172.23.96.198":1},"startTime":"2024-02-16T09:09:25.836-08:00","completedTime":"2024-02-16T09:09:28.970-08:00","timeTaken":3134},"query":{"totalProgress":100,"perNodeProgress":{"ns_1@172.23.97.100":1,"ns_1@172.23.97.67":1},"startTime":"2024-02-16T09:09:28.970-08:00","completedTime":"2024-02-16T09:09:28.979-08:00","timeTaken":9}},"rebalanceId":"af6aa131e3e9d6f0c9dabd2bf73bf2af","nodesInfo":{"active_nodes":["ns_1@172.23.105.122","ns_1@172.23.106.171","ns_1@172.23.106.176","ns_1@172.23.96.198","ns_1@172.23.96.230","ns_1@172.23.96.245","ns_1@172.23.97.100","ns_1@172.23.97.108","ns_1@172.23.97.109","ns_1@172.23.97.66","ns_1@172.23.97.67"],"keep_nodes":["ns_1@172.23.105.122","ns_1@172.23.106.171","ns_1@172.23.106.176","ns_1@172.23.96.198","ns_1@172.23.96.230","ns_1@172.23.96.245","ns_1@172.23.97.100","ns_1@172.23.97.108","ns_1@172.23.97.109","ns_1@172.23.97.66","ns_1@172.23.97.67"],"eject_nodes":[],"delta_nodes":[],"failed_nodes":[]},"masterNode":"ns_1@172.23.97.67","startTime":"2024-02-16T09:09:25.126-08:00","reportingTime":"2024-02-16T09:37:18.652-08:00","timeTaken":1673526} |
cbcollect around the time of the issue ->
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708104282/collectinfo-2024-02-16T173455-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708104282/collectinfo-2024-02-16T173455-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708104282/collectinfo-2024-02-16T173455-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708104282/collectinfo-2024-02-16T173455-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708104282/collectinfo-2024-02-16T173455-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708104282/collectinfo-2024-02-16T173455-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708104282/collectinfo-2024-02-16T173455-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708104282/collectinfo-2024-02-16T173455-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708104282/collectinfo-2024-02-16T173455-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708104282/collectinfo-2024-02-16T173455-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708104282/collectinfo-2024-02-16T173455-ns_1%40172.23.97.67.zip
latest cbcollect ->
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708399008/collectinfo-2024-02-20T032654-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708399008/collectinfo-2024-02-20T032654-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708399008/collectinfo-2024-02-20T032654-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708399008/collectinfo-2024-02-20T032654-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708399008/collectinfo-2024-02-20T032654-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708399008/collectinfo-2024-02-20T032654-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708399008/collectinfo-2024-02-20T032654-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708399008/collectinfo-2024-02-20T032654-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708399008/collectinfo-2024-02-20T032654-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708399008/collectinfo-2024-02-20T032654-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708399008/collectinfo-2024-02-20T032654-ns_1%40172.23.97.67.zip
Please let me know if you need logs from a specific timeframe as I've just attached what I've thought are pertinent logs.
cc Varun Velamuri.