Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
Enterprise Edition 7.1.0 build 2071
-
Untriaged
-
1
-
Unknown
Description
Steps to repro:
- Create a 4 node cluster(replicas=3)
- Created 5 million items(doc size =1024 bytes)
- Stopped persistence on one of the nodes
- Started doc ops(create:expiry:update) on node(where persistence is stopped)
- Killed memcached on node(where persistence is stopped) to trigger rollback on other nodes
- During rollback keep killing memcached on all the nodes(which are rolling back)
- Repeat above steps multiple times.
QE Test |
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.91973.ini bucket_storage=magma,rerun=false,GROUP=P2;crash,randomize_value=True,bucket_eviction_policy=fullEviction,get-cbcollect-info=True,infra_log_level=debug,log_level=debug,enable_dp=false,collect_pcaps=True,get-cbcollect-info=True,autoCompactionDefined=true,upgrade_version=7.1.0-2071 -t storage.magma.magma_rollback.MagmaRollbackTests.test_crash_during_rollback,num_items=5000000,doc_size=1024,nodes_init=4,num_rollbacks=5,vbuckets=1024,rollback_items=500,replicas=3,key_size=12,init_loading=False,doc_ops=expiry:create:update:delete,num_collections=0,process_concurrency=2,collections_for_rollback=1,threads_to_use=1000,target_active_nodes=3,GROUP=P2;crash'
|
Note: 165 items couldn't be persisted in 1200s.
Found these messages in the logs:
2022-01-17T05:50:22.448585-08:00 CRITICAL (default) MagmaKVStore::saveDocs vb:540 WriteDocs failed. Status:Invalid: kvstore-540/rev-000000001: seqno (2) should always increase monotonically between write batches (164) numCommits:0
Ankush Sharma, can you please update the test steps?
Here is the test details: http://qe-jenkins.sc.couchbase.com/job/test_suite_executor-TAF/6951/artifact/job_logs/testrunner-22-Jan-17_04-37-09/test_2/