[System Test] GSI rebalance ongoing for more than 80 hours, progressing very slowly

Test Details

Longevity test

./sequoia -client 172.23.104.254:2375 -provider file:centos_third_cluster.yml -test tests/integration/7.2/test_7.2.yml -scope tests/integration/7.2/scope_7.2_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.2.0-5324 -skip_setup=true -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=true -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true

Day - 3
Cycle - 1
Scale - 3

Test step

1. Perform swap rebalance of KV nodes.

   [2023-05-13T09:40:23-07:00, sequoiatools/couchbase-cli:7.1:db507d] server-add -c 172.23.108.139:8091 --server-add https://172.23.108.141 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data

   [2023-05-13T09:40:39-07:00, sequoiatools/couchbase-cli:7.1:7d8fad] rebalance -c 172.23.108.139:8091 --server-remove 172.23.107.236 -u Administrator -p password

2. This rebalance was stuck for FTS service due to MB-56272.
3. Stopped the rebalance and applied the workaround for MB-56272.
4. After restart GSI rebalance has been ongoing for more then 80 hours now, progressing very slowly - currently at 90%. I have been monitoring the progress of the rebalance during last couple of days and at no point rebalance has got stuck, it is just that it has been progressing very slowly.

Note

GSI rebalance operations in Magma longevity test for 7.1.x releases usually took at most couple of days to complete. The number of Indexes in the test have remained same compared to 7.1.x test but amount of Indexed data is higher.