Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
7.1.0-1190
-
Untriaged
-
1
-
Unknown
-
Magma 2021-Sep-6, Magma May 2022
Description
1. Created a 3 node cluster (replicas =2)
2. Loaded 5 million items(doc size = 1024)
3. Stopped persistence on one node.
4. Started New doc ops (creates:update:expriry:delete).
5. Kill memcached on node where persistence is stopped to trigger rollbacks on other nodes
6. While rollback is going on, kill memcached on all nodes.
7. Repeat the rollback process multiple times.
8. Observed CRITICAL msg in memcached as mentioned below.
|
172.23.106.222 |
Found ' CRITICAL ' logs - ['[ns_server:info,2021-08-25T02:55:33.297-07:00,babysitter_of_ns_1@cb.local:<0.1070.0>:ns_port_server:log:221]memcached<0.1070.0>: 2021-08-25T02:55:33.097175-07:00 CRITICAL (default) MagmaKVStore::saveDocs vb:368 WriteDocs failed. Status:Invalid: kvstore-368/rev-000000001: seqno (1928) should always increase monotonically between write batches (4002) numCommits:0\n', '[ns_server:info,2021-08-25T02:55:38.223-07:00,babysitter_of_ns_1@cb.local:<0.1070.0>:ns_port_server:log:221]memcached<0.1070.0>:
|
|
QE Test |
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.40027.ini bucket_storage=magma,rerun=false,GROUP=P2;crash,randomize_value=True,bucket_eviction_policy=fullEviction,get-cbcollect-info=True,infra_log_level=debug,log_level=debug,enable_dp=false,collect_pcaps=True,upgrade_version=7.1.0-1190 -t storage.magma.magma_rollback.MagmaRollbackTests.test_crash_during_rollback,num_items=4000000,doc_size=1024,nodes_init=3,num_rollbacks=5,vbuckets=1024,rollback_items=600,replicas=2,key_size=12,init_loading=False,doc_ops=expiry:create:update:delete,num_collections=0,process_concurrency=2,collections_for_rollback=1,threads_to_use=1000,target_active_nodes=2,GROUP=P2;crash'
|
Jenkins Job: http://qe-jenkins1.sc.couchbase.com/job/test_suite_executor-TAF/307/console