[Magma, 10TB-VolumeTest, CBAS]: Delta recovery of a KV node is stuck since days.

1. Create a 3 node cluster: 2KV, 1 CBAS
2. Create magma bucket and 50 collections.
3. Create 50000000 items in each collection
4. Update 50000000 RandonKey keys to create 50 percent fragmentation
5. Create 50000000 items in each collection
6. Update 50000000 RandonKey keys to create 50 percent fragmentation
7. Total Items in the bucket = 5B Active, 5B replica, 50% fragmentation
8. Create 5 CBAS datasets on 5 different collections and wait for the ingestion to complete
9. Rebalance in with Loading of docs
10. Rebalance Out with Loading of docs
11. Rebalance In_Out with Loading of docs
12. Swap with Loading of docs
13. Failover 1 node and RebalanceOut that node with loading in parallel
14. Failover a node and FullRecovery that node
15. Failover a node and DeltaRecovery that node with loading in parallel. Delta recovery is stuck since a few days.

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job2.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.ClusterOpsVolume,nodes_init=2,graceful=True,skip_cleanup=True,num_items=50000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=2,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=5,pc=10,index_nodes=0,cbas_nodes=1,fts_nodes=0,ops_rate=200000,ramQuota=51200,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=30000,key_type=RandomKey -m rest'

Live cluster is available at: http://172.23.110.127:8091/ui/index.html#/servers/list?commonBucket=GleamBookUsers0&scenarioZoom=minute&scenario=f9fqsgwb1

Note: The same test was previously ran and passed on 7.1.1 when DCP services were earlier enabled there and plan was to release magma support for CBAS in 7.1.1 which was later changes to 7.1.2