[System Test] Indexer rebalance taking a lot of time

Build : 7.0.0-4955
Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Scale : 3
Iteration : 2nd

In the second iteration of the test, it has been observed that the rebalances are taking a very long time to complete, especially in the indexer phase. They complete successfully though. But this should be investigated and if there are avenues of optimization to make the rebalance quicker, it should be understood. Here are a few examples from the test console log.

[2021-04-17T21:44:46-07:00, sequoiatools/couchbase-cli:7.0:f0b17e] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.137 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index

[2021-04-17T21:45:04-07:00, sequoiatools/couchbase-cli:7.0:76de88] rebalance -c 172.23.108.103:8091 -u Administrator -p password

[2021-04-18T04:25:27-07:00, sequoiatools/cmd:615b5d] 60

Total Rebalance time : 6h 41m

[2021-04-19T00:52:31-07:00, sequoiatools/couchbase-cli:7.0:18eb89] failover -c 172.23.108.103:8091 --server-failover 172.23.106.100:8091 -u Administrator -p password --hard

[2021-04-19T00:53:09-07:00, sequoiatools/couchbase-cli:7.0:af5532] rebalance -c 172.23.108.103:8091 -u Administrator -p password

[2021-04-19T05:59:43-07:00, sequoiatools/cmd:a7c152] 60

Total Rebalance time : 5h 6m

[2021-04-19T08:11:20-07:00, sequoiatools/couchbase-cli:7.0:5370be] server-add -c 172.23.108.103:8091 --server-add https://172.23.97.239 -u Administrator -p password --server-add-username Administrator --server-add-password password --services fts

[2021-04-19T08:11:43-07:00, sequoiatools/couchbase-cli:7.0:c07417] rebalance -c 172.23.108.103:8091 -u Administrator -p password

[2021-04-19T15:48:50-07:00, sequoiatools/cmd:44ee2e] 60

Total Rebalance time : 7h 37m

For the last instance, even though it was not a rebalance operation to add/remove an indexer node, it took 7+ hours just in the index rebalance stage.

"index" : {

         "totalProgress" : 100,

         "perNodeProgress" : {

            "ns_1@172.23.104.137" : 1,

            "ns_1@172.23.96.253" : 1,

            "ns_1@172.23.96.252" : 1,

            "ns_1@172.23.99.11" : 1,

            "ns_1@172.23.121.117" : 1,

            "ns_1@172.23.105.107" : 1

         },

         "startTime" : "2021-04-19T08:30:53.894-07:00",

         "completedTime" : "2021-04-19T15:48:21.392-07:00",

         "timeTaken" : 26247498

      },

*Logs *
=========================================================
1st rebalance mentioned above
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.97.242.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618753864/collectinfo-2021-04-18T135107-ns_1%40172.23.99.25.zip

=========================================================
+2nd rebalance mentioned above +
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.121.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.97.242.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618845660/collectinfo-2021-04-19T152104-ns_1%40172.23.99.25.zip

=========================================================
3rd rebalance mentioned above
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.121.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.97.242.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1618873613/collectinfo-2021-04-19T230659-ns_1%40172.23.99.25.zip