[CDC] Rebalance exited with reason {service_rebalance_failed,index, {agent_died,

Steps:

1. Create a 2 KV and 1 index/query node cluster.
2. Create a magma bucket(replicas=1) and collections(total collection count including default collections is 51)
3. Create 500000000 items sequentially(After creation of few thousands of documents update
4. Update 500000000 created in above step
5. Create 500000000 items sequentially
6. Update 500000000 created in above step
7. Create five indexes Wait for index building.
8. Rebalance in KV with Loading of docs. (Rebalance completed successfully)
9. Rebalance Out KV with Loading of docs.(Rebalance completed successfully)
10. Rebalance In_Out KV with Loading of docs.
11. Pause the rebalance and Enable CDC bucket_history_retention_seconds=259200,bucket_history_retention_bytes=10000000000000)
12. Again trigger rebalance in_out KV with loading of docs (Rebalance completed successfully)
13. Gracefull failover a node , Add a node and trigger rebalance(A swap rebalance)
14. Rebalance exited with reason {service_rebalance_failed,index,
    {agent_died,<34340.5783.0>,

Note:
Cluster is still in same state
http://172.23.110.64:8091/ui/index.html#/servers/list?commonBucket=GleamBookUsers0&scenarioZoom=minute&scenario=6fitqfa3g
This looks similar to MB-56170, But since it failed at a different step, Logging a new bug

Error:

Rebalance exited with reason {service_rebalance_failed,index,

{agent_died,<34340.5783.0>,

{linked_process_died,<34340.2799.253>,

{'ns_1@172.23.110.67',

{timeout,

{gen_server,call,

[<34340.5835.0>,

{call,"ServiceAPI.StartTopologyChange",

#Fun<json_rpc_connection.0.86436583>,

#{timeout => 60000}},

60000]}}}}}}.

QE-TEST:

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ankush_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.ClusterOpsVolume,nodes_init=2,graceful=True,skip_cleanup=True,num_items=50000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=info,infra_log_level=error,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=5,pc=10,index_nodes=1,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=30000,key_type=RandomKey -m rest'