Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.1
-
7.1.2-3358
-
Untriaged
-
-
1
-
Yes
Description
STEPS TO RECREATE:
- Create a 4 node cluster
- Create a magma bucket (bucket_ram_quota = 1024 MB)
- Create 40 million items(item size = 1024 bytes)
- Start new doc ops (Create:delete:expiry,)
- During doc ops, keep killing memcached on all nodes (sigkill memcached)
- Observed CRITICAL Caught unhandled std::exception-derived exception. what(): non-monotonic kvstore revision for kvstore:kvstore-462/rev-000000001 existing revision:2 cbcollect_info_ns_1@172.23.106.116_20220709-234607/ns_server.babysitter.log:13131:memcached<0.1942.0>: what(): non-monotonic kvstore revision for kvstore:kvstore-462/rev-000000001 existing revision:2
(Observed on node 172.23.106.116)
+Babysitter Logs: +
cbcollect_info_ns_1@172.23.106.116_20220709-234607/ns_server.babysitter.log:13129:memcached<0.1942.0>: 2022-07-09T20:52:32.151478+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): non-monotonic kvstore revision for kvstore:kvstore-462/rev-000000001 existing revision:2
|
QE-Test:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.62147.ini bucket_storage=magma,rerun=false,GROUP=P0;recovery_1024MB,randomize_value=true,bucket_eviction_policy=fullEviction,replicas=1,nodes_init=4,enable_dp=false,collect_pcaps=True,key_size=12,ops_rate=10000,num_collections=2,sdk_client_pool=False,init_loading=False,get-cbcollect-info=True,autoCompactionDefined=true,get-cbcollect-info=True,infra_log_level=info,log_level=info,bucket_storage=magma,upgrade_version=7.1.2-3358 -t storage.magma.magma_crash_recovery.MagmaCrashTests.test_crash_during_recovery_new,bucket_ram_quota=1024,multiplier=1,batch_size=500,num_items=40000000,doc_size=1024,doc_ops=create:delete:expiry,create_perc=50,delete_perc=25,expiry_perc=25,GROUP=P0;recovery_1024MB'
|
Attachments
| For Gerrit Dashboard: MB-52902 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 177346,3 | MB-52902: Populate kvstore rev if no vbstate found | neo | kv_engine | Status: MERGED | +2 | +1 |
| 177399,1 | MB-52902: Populate kvstore rev if no vbstate found | master | kv_engine | Status: ABANDONED | 0 | 0 |
| 178259,2 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |