Details
-
Bug
-
Resolution: Known Error
-
Critical
-
7.2.0
-
7.2.0-5225
-
Untriaged
-
0
-
Unknown
Description
Steps to Recreate
- Create a 4 node cluster
- Create a magma buckets with (bucket_history_retention_seconds=0,bucket_history_retention_bytes=200000000000,)
replicas = 2, just default collections - Create 30000000 docs(doc_size = 256 bytes)
- Start upserting all the documents
- Start bucket compaction(Keep calling api to compact bucket with a sleep of 20 seconds, but next iteration of bucket compaction call waits for previous compact call to finish)
- Once bucket compaction finishes
- While doc ops and compactions are going on keep killing memcached (sleep between two SIGKILLS is 60 to 90 seconds, but test ensures cluster warmup finishes before next sigkill)
- Observed ERROR (default) EPBucket::compactInternal: compaction failed for vb:568(looks like Vb 568 got deleted, same happened for vb548 as well)
Snippet from Memcached Logs on 172.23.122.161:
023-03-03T22:57:08.981968-08:00 INFO (default) Compaction of vb:568, task:6581, purge_before_ts:1677652803, purge_before_seq:0, drop_deletes:false, internal:false created with delay:0s (awaiting completion).
|
2023-03-03T22:57:08.982011-08:00 INFO (default) MagmaKVStore::compactDBInternal: vb:568 purge_before_ts:1677652803 purge_before_seq:0 drop_deletes:false retain_erroneous_tombstones:true
|
2023-03-03T22:57:08.982071-08:00 WARNING (default) MagmaKVStore::getDbFileInfo failed status:NotExists: KVStore ID:568 does not exist
|
2023-03-03T22:57:08.982207-08:00 WARNING (default) MagmaKVStore::readVBStateFromDisk: vb:568 readLocalDoc returned status NotExists: MagmaKVStore::readLocalDoc vb:568 key:_local/vbstate NotExists: KVStore ID:568 does not exist
|
2023-03-03T22:57:08.982229-08:00 WARNING (default) MagmaKVStore::compactDBInternal: trying to run compaction on vb:568 but can't read vbstate. Status:Error
|
2023-03-03T22:57:08.982241-08:00 ERROR (default) EPBucket::compactInternal: compaction failed for vb:568
|
2023-03-03T22:57:08.982268-08:00 INFO (default) Compaction of vb:568 done (2). purged tombstones:0, prepares:0, prepareBytes:0 collection_items_erased:alive:0,deleted:0, collections_erased:0, size/items/tombstones/purge_seqno pre{0, 0, 0, 0}, post{0, 0, 0, 0}
|
QE-TEST:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.121213.ini bucket_storage=magma,rerun=false,GROUP=P0;kill,randomize_value=true,doc_size=256,bucket_eviction_policy=fullEviction,replicas=1,nodes_init=4,enable_dp=false,collect_pcaps=True,get-cbcollect-info=True,autoCompactionDefined=true,bucket_history_retention_bytes=200000000000,upgrade_version=7.2.0-5225 -t storage.magma.magma_compaction.MagmaCompactionTests.test_crash_during_compaction,num_items=30000000,doc_size=256,graceful=False,doc_ops=update,replicas=1,GROUP=P0;kill'
|