[Upgrade] Slow rebalance of cappella cluster from 7.2.3 to 7.6.0 due to stream requests failing with TMPFAIL error at memcached

Dhruvil has noticed that during the rebalance progress DocsPending didn't change for quite some time. I'm attaching the snippet he pointed out.

// node-028

2024-01-17T09:14:56.811+00:00 [Info] Rebalancer::waitForIndexBuild: Index: default4:scope_0:coll_3:idx3_mYEFrUOK State: INDEX_STATE_INITIAL DocsPending: 636753 DocsQueued: 0 DocsProcessed: 20754807, Rate: 14987.074081632523 Remaining: 636753 EstTime: 42 Partns: [2] DestAddr: 127.0.0.1:9102 

logs shared at that time 

https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan/collectinfo-2024-01-17T093257-ns_1%40svc-d-node-025.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan/collectinfo-2024-01-17T093257-ns_1%40svc-d-node-026.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan/collectinfo-2024-01-17T093257-ns_1%40svc-d-node-027.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan/collectinfo-2024-01-17T093257-ns_1%40svc-i-node-019.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan/collectinfo-2024-01-17T093257-ns_1%40svc-i-node-020.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan/collectinfo-2024-01-17T093257-ns_1%40svc-i-node-023.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan/collectinfo-2024-01-17T093257-ns_1%40svc-i-node-024.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan/collectinfo-2024-01-17T093257-ns_1%40svc-i-node-028.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan/collectinfo-2024-01-17T093257-ns_1%40svc-q-node-017.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan/collectinfo-2024-01-17T093257-ns_1%40svc-q-node-018.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip

I see that the rebalance is making progress, however, it's quite slow. This is the fresh logs collected after 5hrs of rebalance progress.

 
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan_set_2/collectinfo-2024-01-17T131947-ns_1%40svc-d-node-025.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan_set_2/collectinfo-2024-01-17T131947-ns_1%40svc-d-node-026.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan_set_2/collectinfo-2024-01-17T131947-ns_1%40svc-d-node-027.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan_set_2/collectinfo-2024-01-17T131947-ns_1%40svc-i-node-019.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan_set_2/collectinfo-2024-01-17T131947-ns_1%40svc-i-node-020.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan_set_2/collectinfo-2024-01-17T131947-ns_1%40svc-i-node-023.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan_set_2/collectinfo-2024-01-17T131947-ns_1%40svc-i-node-024.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan_set_2/collectinfo-2024-01-17T131947-ns_1%40svc-i-node-028.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan_set_2/collectinfo-2024-01-17T131947-ns_1%40svc-q-node-017.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/gsi_upgrade_7.2.3_to_7.6.0_17thJan_set_2/collectinfo-2024-01-17T131947-ns_1%40svc-q-node-018.d4mwovaz9xo8ildy.sandbox.nonprod-project-avengers.com.zip