[CouchstoreToMagmaMigrationTest] Rebalance exited with reason {pre_rebalance_janitor_run_failed, "GleamBookUsers0", {error,wait_for_memcached_failed,

Steps To Recreate:

1. Create a 4 node cluster.
2. Create a couchstore bucket(replicas=1, ram_quota =10 GiB per node, bucket_eviction_policy=fullEviction, bucket-name=GleamBookUsers0)
3. Create 50 non default collections
4. Load 50000000 docs of size 512 bytes in each of the newly created non default collections
5. Change storage mode from couchstore to magma
6. Start doc:ops(update:read)
7. Trigger a swap rebalance(one node coming in one going out)
8. Swap rebalance was successfull
9. Trigger graceful failover fullrecovery rebalance (failed over node 172.23.107.220) while data loading is going on
10. Graceful failover+fullrecovery was successfull
11. Trigger hard failover + fullrecovery+ rebalance (failed over node 172.23.107.97) while data loading is going on
12. Trigger hard failover + fullrecovery+ rebalance (failed over node 172.23.107.232) while data loading is going on
13. Update maxTTL value of VolumeCollection0 and VolumeCollection1
14. Delete one collection(VolumeCollection 10)
15. Create a collection(with name VolumeCollection10) with history=true
16. Stop rebalance and create a new bucket( with historyRetentionCollectionDefault=true -d historyRetentionBytes=8446744073709551615 -d historyRetentionSeconds=3600)
17. Update num replicas = 2 , durability=majority and bucket priority to high for bucket GleamBookUsers0
18. Start rebalance
19. Rebalance exited with reason {pre_rebalance_janitor_run_failed, "GleamBookUsers0", {error,wait_for_memcached_failed,

Restarted rebalance multiple times but it was failing always

Rebalance Failute:

Rebalance exited with reason {pre_rebalance_janitor_run_failed,

"GleamBookUsers0",

{error,wait_for_memcached_failed,

['ns_1@172.23.107.220',

'ns_1@172.23.107.239',

'ns_1@172.23.107.97']}}.

Rebalance Operation Id = 0d32f3032d8813abee3b8aca0cc87262

QE-TEST:

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/temp_vol.ini bucket_storage=couchstore,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.StorageMigrationTestHappyPath,nodes_init=4,graceful=True,skip_cleanup=True,num_items=50000000,num_buckets=1,bucket_names=GleamBook,doc_size=512,bucket_type=membase,bucket_eviction_policy=valueOnly,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=22,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=0,pc=10,indexer_mem_quota=0,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=100000,ramQuota=10240,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey,revert_migration=True'