Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
5.0.0-5032
-
Untriaged
-
-
1
-
Unknown
Description
Steps:
Step 1: Create a 8 kv , 2 query, 4 cbas, 4 index node cluster
Step 2: Create required buckets and collections. Create 50 indexes.
Step 3: Create 500000000 items sequentially with ttl=10
Step 4.1: Rebalance IN while loading of docs in is progress
Step 5.1: Rebalance OUT while loading of docs in is progress
Rebalance out failed after ~80 mins with the below error:
|
172.23.121.139 |
Service 'indexer' exited with status 137. Restarting. Messages:
|
2021-04-28T09:21:15.400-07:00 [Info] GleamBookUsers0/idx43/Mainstore#3972447244979960998:0 Plasma: shortMemStats = {"memory_size":64,"memory_size_index":0,"buf_memused":15117661,"mvcc_purge_ratio":0.00000,"resident_ratio":0.00000,"alloc_size":1424,"free_size":1360,"items_count":0,"recs_in_mem":0,"reclaimed":1360,"reclaim_pending":0}
|
2021-04-28T09:21:15.400-07:00 [Info] GleamBookUsers0/idx28/Mainstore#10260842048718521879:0 Plasma: shortMemStats = {"memory_size":64,"memory_size_index":0,"buf_memused":15117661,"mvcc_purge_ratio":0.00000,"resident_ratio":0.00000,"alloc_size":1288,"free_size":1224,"items_count":0,"recs_in_mem":0,"reclaimed":1224,"reclaim_pending":0}
|
2021-04-28T09:21:22.657-07:00 [Info] ServiceMgr::GetTaskList []
|
2021-04-28T09:21:15.401-07:00 [Warn] Timekeeper::flushMonitor Waiting for flush to finish for 870 seconds. Stream MAINT_STREAM KeyspaceId GleamBookUsers0.
|
2021-04-28T09:21:22.669-07:00 [Warn] Timekeeper::flushMonitor Waiting for flush to finish for 880 seconds. Stream MAINT_STREAM KeyspaceId GleamBookUsers0.
|
2021-04-28T09:21:22.671-07:00 [Info] ServiceMgr::rebalanceJanitor Running Periodic Cleanup
|
2021-04-28T09:21:22.662-07:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 3 154] []}
|
2021-04-28T09:21:22.753-07:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 3 154]
|
Indexes are still building:
|
QE Test |
git fetch "http://review.couchbase.org/TAF" refs/changes/34/151634/57 && git checkout FETCH_HEAD
|
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p bucket_storage=couchstore,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.test_rebalance,nodes_init=8,graceful=True,skip_cleanup=True,num_items=500000000,num_buckets=1,bucket_names=GleamBook,doc_size=256,bucket_type=membase,eviction_policy=fullEviction,iterations=100,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=2,maxttl=1000,num_indexes=50,pc=1,sdk_client_pool=True,max_clients=1,index_nodes=4,query_nodes=2,cbas_nodes=4,fts_nodes=0 -m rest'
|
