Details
-
Task
-
Resolution: Done
-
Major
-
7.1.0
-
7.1.0-2093
Description
Steps to repro:
- Create a 2 node cluster(172.23.122.245, 172.23.122.246(bucket_ram_quota = 2056 MB/node), replicas=0)
- Create 100 non default collections in default scope(collection count is 101(including default collection)
- Start Loading 500k docs in each of 100 non default collections
- While doc loading is going on drop few collections recreate few collections
- Remove node 172.23.122.246 and trigger rebalance
- Rebalance was successful
- Change replicas to 1 and again add back node 172.23.122.246
- Trigger Full compaction
- While rebalance is going on observed node 172.23.122.245 turned in to amber on UI (UI was showing 1 node in pending state)
Note:
Though I couldn't find anything weird in memcached logs
Ben Huddleston also had a look in to the cluster, but he also didn't notice anything in memcached logs. So we wanted ns serv or UI team to have a look in to the logs.
Also this issue is not consistent, I ran this test few times, but have encountered this issue only twice.
QE-Test:
git fetch "https://review.couchbase.org/TAF" refs/changes/88/166488/1 && git checkout FETCH_HEAD
|
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/qe_r.ini -p bucket_storage=couchstore,rerun=false,bucket_eviction_policy=fullEviction,init_loading=False -t storage.magma.magma_rebalance.MagmaRebalance.test_data_load_collections_with_rebalance_in,num_items=500000,doc_size=256,nodes_init=2,nodes_in=1,standard_buckets=1,magma_buckets=0,bucket_storage=couchstore,data_load_stage=before,sdk_timeout=60,vbuckets=1024,key_size=12,replicas=0,infra_log_level=debug,log_level=debug,skip_cleaup=True,randomize_value=True,bucket_eviction_policy=fullEviction,infra_log_level=debug,log_level=debug,init_loading=False,fragmentation=30,skip_cleanup=True,autoCompactionDefined=true,iterations=1,enable_dp=True,num_collections=100,num_scopes=1,bucket_ram_quota=2056,skip_cleanup=True,sdk_client_pool=False,ops_rate=12000,doc_ops=create,create_perc=100,delete_perc=0,update_perc=0,num_collections_to_drop=0,get-cbcollect-info=True -m rest'
|
