Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
7.1.0-2478
-
Untriaged
-
Centos 64-bit
-
1
-
Unknown
-
KV March-22, KV May 22
Description
STEPS TO RECREATE:
DISK FULL TEST
- Create a 4 node cluster
- Create 5 million items (doc size = 2048) and replicas =1
- Fill entire disk , ( "fallocate -l <space left on disk> <file_name>", e.g "fallocate -l 84716M /data/full_disk_84716MB_1647101247.94")
- After Disk is full, Start doc ops (create docs) until ep_data_write_failed > 0(ensured using cbstats)
- Kill memcached on all nodes (kill -9 $(pgrep memcached) Time difference between sigKill on each node was three seconds
- Observed "2022-03-12T09:08:26.230072-08:00 CRITICAL (default) WarmupBackfillTask::run(): caught exception while running backfill - aborting warmup: WarmupVbucketVisitor::visit(): vb:107 shardId:3 failed to create BySeqnoScanContext, for backfill task:'Warmup - loading KV Pairs shard 3'"
(Observed on node 172.23.122.247)
QE-TEST:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.33408.ini bucket_storage=magma,rerun=false,bucket_eviction_policy=fullEviction,randomize_value=True,enable_dp=false,GROUP=P0,get-cbcollect-info=True,upgrade_version=7.1.0-1671 -t storage.magma.magma_disk_full.MagmaDiskFull.test_crash_recovery_disk_full,nodes_init=4,num_items=5000000,doc_size=2048,sdk_timeout=60,replicas=1,GROUP=P0'
|
Note:
- After the above failure, in tear down we clear the disk space, by removing the file created to fill up the disk (step 3 mentioned above (using rm -rf /data/full_disk_*)). But even after creating disk space all nodes on UI stays in amber state.
- This issue is not easily reproducible . I ran this test many times on the same build, but was able to hit this issue only once.
Cluster details: http://172.23.122.245:8091/ui/index.html#/buckets?commonBucket=default&scenarioZoom=minute&scenario=oombr8sk5
Attachments
For Gerrit Dashboard: MB-51413 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
172221,7 | MB-51413: Create MockCouchKVStore in replaceCouchKVStore | master | kv_engine | Status: MERGED | +2 | +1 |
172222,3 | MB-51413: test | master | kv_engine | Status: ABANDONED | -1 | 0 |
172225,2 | MB-51413: Don't bundle rev with state in Magma readVBStateFromDisk | master | kv_engine | Status: MERGED | +2 | +1 |
172252,9 | MB-51413: Add NotFound status to CouchKVStore readVBState | master | kv_engine | Status: MERGED | +2 | +1 |
172254,8 | MB-51413: Move ReadVBStateStatus to KVStoreIface | master | kv_engine | Status: MERGED | +2 | +1 |
172264,10 | MB-51413: Return ReadVBStateResult from readVBStateFromDisk(Vbid) | master | kv_engine | Status: MERGED | +2 | +1 |
172266,10 | MB-51413: Return ReadVBStateResult from snapshot magma readVBState | master | kv_engine | Status: ABANDONED | +2 | -1 |
172267,9 | MB-51413: Magma snapshot readVBState check snap bounds | master | kv_engine | Status: ABANDONED | 0 | -1 |
172276,9 | MB-51413: Return ReadVBStateResult in RocksDBKVStore readVBState | master | kv_engine | Status: ABANDONED | 0 | -1 |
172277,9 | MB-51413: Make getPersistedVBucketState return ReadVBStateResult | master | kv_engine | Status: ABANDONED | 0 | -1 |
172336,11 | MB-51413: Don't warmup vBuckets without state | master | kv_engine | Status: ABANDONED | 0 | -1 |
172720,3 | MB-51413: Move StatDef to separate files | master | kv_engine | Status: MERGED | +2 | +1 |
172733,6 | MB-51413: Generate Configuration::addStats directly | master | kv_engine | Status: MERGED | +2 | +1 |
174466,2 | [BP] MB-51413: Don't bundle rev with state in Magma readVBStateFromDisk | neo | kv_engine | Status: MERGED | +2 | +1 |
174467,2 | [BP] MB-51413: Create MockCouchKVStore in replaceCouchKVStore | neo | kv_engine | Status: MERGED | +2 | +1 |
174468,2 | [BP] MB-51413: Add NotFound status to CouchKVStore readVBState | neo | kv_engine | Status: MERGED | +2 | +1 |
174469,2 | [BP] MB-51413: Move ReadVBStateStatus to KVStoreIface | neo | kv_engine | Status: MERGED | +2 | +1 |
174470,2 | [BP] MB-51413: Return ReadVBStateResult from readVBStateFromDisk(Vbid) | neo | kv_engine | Status: MERGED | +2 | +1 |
174471,2 | MB-51413: Return ReadVBStateResult from snapshot magma readVBState | neo | kv_engine | Status: MERGED | +2 | +1 |
174472,2 | MB-51413: Magma snapshot readVBState check snap bounds | neo | kv_engine | Status: MERGED | +2 | +1 |
174473,2 | MB-51413: Return ReadVBStateResult in RocksDBKVStore readVBState | neo | kv_engine | Status: MERGED | +2 | +1 |
174474,2 | MB-51413: Make getPersistedVBucketState return ReadVBStateResult | neo | kv_engine | Status: MERGED | +2 | +1 |
174475,2 | MB-51413: Don't warmup vBuckets without state | neo | kv_engine | Status: MERGED | +2 | +1 |
174521,1 | Merge branch 'neo' | neo | kv_engine | Status: ABANDONED | 0 | 0 |
174523,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
174572,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
174636,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
174637,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
174671,2 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
174681,2 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
174682,2 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
174729,1 | Merge branch 'neo' | master | kv_engine | Status: ABANDONED | 0 | 0 |
175102,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
175115,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |