Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.2.0
-
7.2.0-5161
-
Untriaged
-
Centos 64-bit
-
-
0
-
Unknown
-
KV 2023-2
Description
Steps To Recreate:
- Create a 3 node cluster
Create a magma buckets with (history_retention_collection_default=true,bucket_history_retention_seconds=0,bucket_history_retention_bytes=96636764160, magma_key_tree_data_block_size=131072,magma_seq_tree_data_block_size=9092) (vbuckets = 16, replicas = 2)
Create fifty collections(total collections including default are fifty one)
Create 100000 docs in each of the non default collections
Update all the docs created in above step(16 times)
Change bucket replicas == 1 and rebalance(rebalance is successfull)
Again change replicas == 2 and start rebalance
while rebalance is going on create/delete few collections
Trigger Compaction
Stop rebalance and change enable bucket level ttl and set it to 10seconds
Resume rebalance
Stop Rebalance and change bucket durability level to majorityandPersistActive
Resume Rebalance
Stop rebalance and increase bucket history_retention_bytes to 960GB
Resume Rebalance
Stop rebalance and decrease bucket_history_retention_bytes to 2GB(total data on disk at this stage was 110 GB)
Resume Rebalance
Rebalance failed with dcp_wait_for_data_move_failed
Rebalance Failure:
Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.10923.99>,
|
{{dcp_wait_for_data_move_failed,"default",3,
|
'ns_1@172.23.107.232',
|
['ns_1@172.23.107.97','ns_1@172.23.107.240'],
|
{error,no_stats_for_this_vbucket}},
|
[{ns_single_vbucket_mover,
|
'-wait_dcp_data_move/5-fun-0-',5,
|
[{file,"src/ns_single_vbucket_mover.erl"},
|
{line,451}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,211}]}]}}}}.
|
Rebalance Operation Id = 52b233838e709215285e35761577a4f7
|
|
|
Worker <0.10912.99> (for action {move,{3,
|
['ns_1@172.23.107.232',
|
'ns_1@172.23.107.97'],
|
['ns_1@172.23.107.232',
|
'ns_1@172.23.107.97',
|
'ns_1@172.23.107.240'],
|
[]}}) exited with reason {unexpected_exit,
|
{'EXIT',
|
<0.10923.99>,
|
{{dcp_wait_for_data_move_failed,
|
"default",
|
3,
|
'ns_1@172.23.107.232',
|
['ns_1@172.23.107.97',
|
'ns_1@172.23.107.240'],
|
{error,
|
no_stats_for_this_vbucket}},
|
[{ns_single_vbucket_mover,
|
'-wait_dcp_data_move/5-fun-0-',
|
5,
|
[{file,
|
"src/ns_single_vbucket_mover.erl"},
|
{line,
|
451}]},
|
{proc_lib,
|
init_p,3,
|
[{file,
|
"proc_lib.erl"},
|
{line,
|
211}]}]}}}
|
QE-TEST:
git fetch https://review.couchbase.org/TAF refs/changes/62/186462/5 && git checkout -b change-186462 FETCH_HEAD
|
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/temp_vol_3.ini -p bucket_storage=magma,bucket_ram_quota=256,init_loading=False,bucket_eviction_policy=fullEviction,rerun=False -t storage.magma.magma_crash_recovery.MagmaCrashTests.test_magma_failures_with_CDC,nodes_init=3,skip_cleanup=True,num_items=100000,doc_size=1024,batch_size=1000,sdk_timeout=60,log_level=info,infra_log_level=info,key_size=22,num_collections=51,ops_rate=20000,key_type=SimpleKey,vbuckets=16,replicas=2,doc_ops=update,test_itr=50,bucket_history_retention_seconds=0,bucket_history_retention_bytes=96636764160,standard_buckets=1,magma_buckets=1,num_scopes=1,induce_failures=False,history_for_collections=True,magma_key_tree_data_block_size=131072,magma_seq_tree_data_block_size=9092 -m rest'
|