[CDC] Perpetual Backfilling loop, suspect eager cp removal. was: Rebalance hung seen in Graceful failover + Delta Recovery

1. Create a 2 KV and 1 index/query node cluster.
2. Create a magma bucket(replicas=1) and collections(total collection count including default collections is 51)
3. Create 500000000 items sequentially(After creation of few thousands of documents set bucket_history_retention_seconds=259200,bucket_history_retention_bytes=12000000000000)
4. Update 500000000 created in above step
5. Create 500000000 items sequentially
6. Update 500000000 created in above step
7. Create five indexes Wait for index building.
8. Rebalance in KV with Loading of docs.
9. Rebalance Out KV with Loading of docs.(Rebalance completed successfully)
10. Rebalance In_Out KV with Loading of docs. (Rebalance completed successfully)
11. Swap Rebalance with Loading of docs (Rebalance completed successfully)
12. Graceful Failover 1 node and RebalanceOut that node with loading in parallel (Rebalance completed successfully)
13. Graceful Failover a node and FullRecovery that node with loading in parallel(Rebalance completed successfully)
14. Graeful Failover a node and DeltaRecovery that node with loading in parallel
15. Observed Rebalance was hung , After starting rebalance not even a single doc has moved (Wiated for close to one hour before reporting this)

Note:

Cluster is still in same state:
http://172.23.110.64:8091/ui/index.html#/logs?commonBucket=GleamBookUsers0&scenarioZoom=minute&scenario=7udweu5h0

QE-TEST:

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ankush_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.ClusterOpsVolume,nodes_init=2,graceful=True,skip_cleanup=True,num_items=50000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=info,infra_log_level=error,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=5,pc=10,index_nodes=1,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=30000,key_type=RandomKey -m rest'