Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
7.1.0-1985
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
-
KV 2022-Jan, KV 2022-Feb
Description
Steps to repro:
- Create a 4 node cluster.
- Create magma bucket
- Load num_items=40000000,doc_size=1024. Let the load finish.
- Started new doc_ops=create:update:expiry
- Trigger bucket compaction
- Start killing memcahced on all nodes. Wait for bucket warmup to complete. Repeat this step in a thread with a random wait of 30-60s
- Observed after 5th iteration bucket got stuck in warmup state (observed for 1200 seconds)
QE-TEST:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ankush_temp_job.ini bucket_storage=magma,rerun=false,GROUP=P0;kill,randomize_value=true,doc_size=256,bucket_eviction_policy=fullEviction,replicas=1,nodes_init=4,enable_dp=false,collect_pcaps=True,get-cbcollect-info=True,autoCompactionDefined=true,upgrade_version=7.1.0-1985 -t storage.magma.magma_compaction.MagmaCompactionTests.test_crash_during_compaction,doc_size=256,upgrade_version=7.1.0-1985,graceful=False,rerun=false,GROUP=P0;kill,enable_dp=false,doc_ops=create:update:expiry,get-cbcollect-info=True,replicas=1,bucket_storage=magma,bucket_eviction_policy=fullEviction,nodes_init=4,num_items=20000000,autoCompactionDefined=true,collect_pcaps=True,randomize_value=true -m rest'
|
Ritesh Agarwal has also logged a similar bug some time back(MB-49003), But this time we have tried on better machines(that we use for volume testing), but still we are hitting this issue.
