Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
7.1.0-1190
-
Untriaged
-
1
-
Unknown
-
Magma 2021-Sep-6, Magma May 2022
Description
1. Created a 3 node cluster (replicas =2)
2. Loaded 5 million items(doc size = 1024)
3. Stopped persistence on one node.
4. Started New doc ops (creates:update:expriry:delete).
5. Kill memcached on node where persistence is stopped to trigger rollbacks on other nodes
6. While rollback is going on, kill memcached on all nodes.
7. Repeat the rollback process multiple times.
8. Observed CRITICAL msg in memcached as mentioned below.
172.23.106.222 |
Found ' CRITICAL ' logs - ['[ns_server:info,2021-08-25T02:55:33.297-07:00,babysitter_of_ns_1@cb.local:<0.1070.0>:ns_port_server:log:221]memcached<0.1070.0>: 2021-08-25T02:55:33.097175-07:00 CRITICAL (default) MagmaKVStore::saveDocs vb:368 WriteDocs failed. Status:Invalid: kvstore-368/rev-000000001: seqno (1928) should always increase monotonically between write batches (4002) numCommits:0\n', '[ns_server:info,2021-08-25T02:55:38.223-07:00,babysitter_of_ns_1@cb.local:<0.1070.0>:ns_port_server:log:221]memcached<0.1070.0>:
|
QE Test |
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.40027.ini bucket_storage=magma,rerun=false,GROUP=P2;crash,randomize_value=True,bucket_eviction_policy=fullEviction,get-cbcollect-info=True,infra_log_level=debug,log_level=debug,enable_dp=false,collect_pcaps=True,upgrade_version=7.1.0-1190 -t storage.magma.magma_rollback.MagmaRollbackTests.test_crash_during_rollback,num_items=4000000,doc_size=1024,nodes_init=3,num_rollbacks=5,vbuckets=1024,rollback_items=600,replicas=2,key_size=12,init_loading=False,doc_ops=expiry:create:update:delete,num_collections=0,process_concurrency=2,collections_for_rollback=1,threads_to_use=1000,target_active_nodes=2,GROUP=P2;crash'
|
Jenkins Job: http://qe-jenkins1.sc.couchbase.com/job/test_suite_executor-TAF/307/console
Attachments
For Gerrit Dashboard: MB-48166 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
162623,4 | MB-48166 wal: Remove unused Flush wal API call | master | magma | Status: MERGED | +2 | +1 |
162624,1 | MB-48166 util: Remove locking from sync points unless using Dependencies | master | magma | Status: ABANDONED | +1 | -1 |
162661,5 | MB-48166 wal: Remove GroupComit code | master | magma | Status: MERGED | +2 | +1 |
162666,6 | MB-48166 wal: Move sync to endTx | master | magma | Status: MERGED | +2 | +1 |