[BP 6.6.3][System Test] Analytics rebalance stuck

Build : 7.0.0-3653
Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Scale : 3
Day : 2nd

A rebalance operation to remove analytics node 172.23.121.77 from the cluster is stuck for ~44 hours now.

The rebalance operation started at 2:30:09 PM on 7 Nov, 2020. The analytics nodes in the cluster are :
172.23.120.81, 172.23.121.77, 172.23.96.122, 172.23.97.112

172.23.97.74 is the master node in the cluster.

The following logs were taken shortly after the rebalance operation started (at 14:43 ):
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.106.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.120.58.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.120.73.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.120.75.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.120.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.120.81.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.120.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.121.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.123.24.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.123.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.123.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.123.31.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.123.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.123.33.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.96.14.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.96.18.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.96.243.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.96.254.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.96.48.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.97.105.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.97.110.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.97.112.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.97.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.97.149.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.97.150.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.97.151.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1604788995/collectinfo-2020-11-07T224316-ns_1%40172.23.97.74.zip

These logs were captured at around 9:45 AM on 11/09 (after ~44 hrs of rebalance start)
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.106.134.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.120.58.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.120.73.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.120.75.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.120.77.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.120.81.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.120.86.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.121.77.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.123.24.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.123.25.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.123.26.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.123.31.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.123.32.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.123.33.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.96.122.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.96.14.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.96.18.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.96.243.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.96.254.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.96.48.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.97.105.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.97.110.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.97.112.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.97.148.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.97.149.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.97.150.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.97.151.zip
https://uploads.couchbase.com/analytics_reb_stuck_1109/collectinfo-2020-11-09T174647-ns_1%40172.23.97.74.zip