Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
7.0.0-2017
-
Untriaged
-
-
1
-
Yes
-
Magma-July-19-2021, Magma-Aug-2-2021, Magma-Aug-9-2021, Magma-Aug-16-2021, Magma 2021-Aug-23, Magma 2021-Sep-6
Description
Expected Behavour
If KV-Engine attempts to open a Magma instance (via MagmaKVStore::MagmaKVStore calling Magma::Open()), that should succeed even if the disk cannot be written to (e.g. disk full), to allow read-availability even if data cannot be written.
Actual Behaviour
Magma::Open fails (see below backtrace).
Test Details
./testrunner -i /tmp/ankush.ini -t magma.magma_basic_crud.BasicCrudTests.test_multi_update_delete,nodes_init=4,replicas=0,num_items=7000000,rerun=False,magma_storage=True,bucket_storage=magma,sdk_timeout=60,infra_log_level=debug,log_level=debug,skip_cleanup=True,randomize_value=True,vbuckets=128,disable_magma_commit_points=True,fragmentation=50,update_itr=2,doc_size=4096,test_itr=2,bucket_eviction_policy=fullEviction,replicas=0
|
Steps to repro
|
1) Had a four node cluster
|
2) Was Loading 7M items (each of size 4096 bytes)
|
3) Disk became full on one of node(172.23.122.217)
|
4) Observer core , Below are Backtraces and logs
|
Below are the back trace for same
|
|
|
(gdb) bt
|
#0 0x00007f6422c8b337 in raise () from /lib64/libc.so.6
|
#1 0x00007f6422c8ca28 in abort () from /lib64/libc.so.6
|
#2 0x00007f64237e9195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#3 0x000000000054a6b2 in backtrace_terminate_handler() ()
|
#4 0x00007f64237e6f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#5 0x00007f64237e6fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#6 0x00007f64237e7213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#7 0x00007f642709b421 in MagmaKVStore::MagmaKVStore(MagmaKVStoreConfig&) () from /opt/couchbase/bin/../lib/libep.so
|
#8 0x00007f6426fdfa23 in KVStoreFactory::create(KVStoreConfig&) () from /opt/couchbase/bin/../lib/libep.so
|
#9 0x00007f6426ffbcd4 in KVShard::KVShard(unsigned short, unsigned short, Configuration&) () from /opt/couchbase/bin/../lib/libep.so
|
#10 0x00007f6427044b23 in VBucketMap::VBucketMap(Configuration&, KVBucket&) () from /opt/couchbase/bin/../lib/libep.so
|
#11 0x00007f6426fe9bae in KVBucket::KVBucket(EventuallyPersistentEngine&) () from /opt/couchbase/bin/../lib/libep.so
|
#12 0x00007f6426f6ff76 in EPBucket::EPBucket(EventuallyPersistentEngine&) () from /opt/couchbase/bin/../lib/libep.so
|
#13 0x00007f6426f87f09 in EventuallyPersistentEngine::makeBucket(Configuration&) () from /opt/couchbase/bin/../lib/libep.so
|
#14 0x00007f6426f99d6c in EventuallyPersistentEngine::initialize(char const*) () from /opt/couchbase/bin/../lib/libep.so
|
#15 0x0000000000421e80 in CreateBucketThread::create() ()
|
#16 0x0000000000422399 in CreateBucketThread::run() ()
|
#17 0x00007f64259f84bf in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
|
#18 0x00007f64259ed827 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
|
#19 0x00007f642302ae65 in start_thread () from /lib64/libpthread.so.0
|
#20 0x00007f6422d5388d in clone () from /lib64/libc.so.6
|
Logs:
|
2020-05-13T04:41:36.010224-07:00 CRITICAL kvstore-51/rev-000000002/seqIndex Error occured during memtable flush (IOError: /opt/couchbase/var/lib/couchbase/data/default/magma.3/kvstore-51/rev-000000002/seqIndex/sstable.143.data: Unable to open file err=No space left on device).
|
2020-05-13T04:41:36.010242-07:00 CRITICAL kvstore-34/rev-000000002/seqIndex Error occured during memtable flush (IOError: /opt/couchbase/var/lib/couchbase/data/default/magma.2/kvstore-34/rev-000000002/seqIndex/sstable.141.data: Unable to open file err=No space left on device).
|
2020-05-13T04:41:36.010290-07:00 CRITICAL (default) magma_2 MagmaKVStore::saveDocs vb:34 WriteDocs failed. Status:IOError: /opt/couchbase/var/lib/couchbase/data/default/magma.2/kvstore-34/rev-000000002/seqIndex/sstable.141.data: Unable to open file err=No space left on device
|
2020-05-13T04:41:36.010301-07:00 CRITICAL (default) magma_3 MagmaKVStore::saveDocs vb:51 WriteDocs failed. Status:IOError: /opt/couchbase/var/lib/couchbase/data/default/magma.3/kvstore-51/rev-000000002/seqIndex/sstable.143.data: Unable to open file err=No space left on device
|
2020-05-13T04:41:36.012188-07:00 CRITICAL kvstore-39/rev-000000002/seqIndex Error occured during memtable flush (IOError: /opt/couchbase/var/lib/couchbase/data/default/magma.3/kvstore-39/rev-000000002/seqIndex/sstable.143.data: Unable to open file err=No space left on device).
|