Slow KV requests to some nodes

Script to Repro

./testrunner -i /tmp/win10-bucket-ops.ini rerun=False -t volumetests.test_system_orchestrator_heartbeats_and_timeouts.volume.test_volume_MB_41562,nodes_init=7,initial_load=3000000,replicas=2

 
This is a new system test written to test MB-41562.

Steps to Repro
1. Create a 7 node cluster as shown below
------------------------++-------------

------------------------++-------------

------------------------++-------------

2. Set non default orchestrator heartbeats and timeouts.

2020-11-01 22:12:51,023 | test  | INFO    | MainThread | [test_system_orchestrator_heartbeats_and_timeouts:test_volume_MB_41562:639] Step 1: Set Non default orchestrator heartbeats and timeouts

curl http://localhost:9000/diag/eval -u Administrator:asdasd -d 'ns_config:set({mb_master, heartbeat_interval}, 500).'

curl http://localhost:9000/diag/eval -u Administrator:asdasd -d 'ns_config:set({mb_master, timeout_interval_count}, 3).’

curl http://localhost:9000/diag/eval -u Administrator:asdasd -d 'ns_config:set({leader_lease_acquire_worker, lease_time}, 5000).'

curl http://localhost:9000/diag/eval -u Administrator:asdasd -d 'ns_config:set({leader_lease_acquire_worker, lease_grace_time}, 2000).'

curl http://localhost:9000/diag/eval -u Administrator:asdasd -d 'ns_config:set({leader_lease_acquire_worker, lease_renew_after}, 500).'

3. Do initial Data load and start running n1ql queries in the background.
2020-11-13 18:49:55,753 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
---------------------------------------------------------------------------

---------------------------------------------------------------------------

---------------------------------------------------------------------------

4. Do a rebalance in
2020-11-13 18:49:58,676 | test | INFO | pool-1-thread-3 | [table_view:display:72] Rebalance Overview
------------------------------------

------------------------------------

------------------------------------

5. Find the orchestrator node, kill babysitter on orchestrator, do a hard failover, start couchbase-server, start delta recovery and rebalance. This is step is repeated 5 times in a loop.

6. Do a rebalance out
2020-11-13 21:29:57,996 | test | INFO | pool-1-thread-10 | [table_view:display:72] Rebalance Overview
------------------------------------

------------------------------------

------------------------------------

7. Flush all the buckets.

Repeat steps 3-7 multiple times.

I had done cbcollect to share with Ns_serv team. I saw the following line in the nutshell.

• See CBSE-4320 for details about checkpoint usage.*

Wanted to check if there is something of concern here.