Rebalance in of a node failed due to wait_seqno_persisted_failed.

1. Step 1: Create a 3 node cluster
   2022-03-09 20:05:53,509 | test | INFO | pool-3-thread-26 | [task:check:474] Rebalance completed with progress: 100% in 25.114000082 sec
2. Step 1*: Create a 3 node XDCR remote cluster
   2022-03-09 20:06:23,374 | test | INFO | pool-3-thread-28 | [task:check:474] Rebalance completed with progress: 100% in 15.0910000801 sec
3. Step 2: Create required buckets and collections.
4. Step 2*: Create required buckets and collections on XDCR remote.
5. Step 1: Create 10000000 items sequentially
6. Step 2: Update 10000000 RandonKey keys to create 50 percent fragmentation
7. Step 3: Create 10000000 items sequentially
8. Step 4: Update 10000000 RandonKey keys to create 50 percent fragmentation
9. Step 5: Rebalance in with Loading of docs on xdcr src cluster. Rebalance failed on ns_1@172.23.105.175 at 9:41:02 PM 10 Mar, 2022:

   Rebalance exited with reason {mover_crashed,

   {unexpected_exit,

   {'EXIT',<0.1673.161>,

   {{wait_seqno_persisted_failed,

   "GleamBookUsers0",110,2123076,

   [{'ns_1@172.23.121.78',

   {'EXIT',

   {socket_closed,

   {gen_server,call,

   [{'janitor_agent-GleamBookUsers0',

   'ns_1@172.23.121.78'},

   {if_rebalance,<0.23281.135>,

   {wait_seqno_persisted,110,2123076}},

   infinity]}}}}]},

   [{ns_single_vbucket_mover,

   '-wait_seqno_persisted_many/5-fun-2-',5,

   [{file,"src/ns_single_vbucket_mover.erl"},

   {line,474}]},

   {proc_lib,init_p,3,

   [{file,"proc_lib.erl"},{line,211}]}]}}}}.

   Rebalance Operation Id = 40310280f91beb85863c6d434dcf1be0 hide

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job1.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.ClusterOpsVolume,nodes_init=3,graceful=True,skip_cleanup=True,num_items=10000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=5,pc=25,index_nodes=0,xdcr_collections=50,xdcr_remote_nodes=3,cbas_nodes=0,fts_nodes=0,ops_rate=40000,ramQuota=10240,doc_ops=create:update:delete:read,rebl_ops_rate=10000,key_type=RandomKey,vbuckets=1024,mutation_perc=30,replicas=2 -m rest'

Last working build was 2438