Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
Cheshire-Cat
-
Enterprise Edition 7.0.0 build 2422
-
Untriaged
-
-
1
-
Unknown
Description
1. Create a 2 node cluster: .37, .39
2. Create a bucket with replica=1 and Storage=Magma
3. After 60s of load, rebalance In 1 node: .40
4. Rebalance proceeded till 36% and then it failed.
Magma started seeing these failures:
172.23.97.38 : Found message in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000065.txt
|
2020-06-26T04:53:25.856338-07:00 CRITICAL Failed to write tree state /data/magma/magma.8/kvstore-8/rev-000000004/seqIndex/state.6: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.856734-07:00 CRITICAL Failed to write tree state /data/magma/magma.4/kvstore-4/rev-000000004/seqIndex/state.5: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.856776-07:00 CRITICAL Failed to write tree state /data/magma/magma.8/kvstore-64/rev-000000002/seqIndex/state.6: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.856805-07:00 CRITICAL Failed to write tree state /data/magma/magma.10/kvstore-10/rev-000000002/seqIndex/state.5: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.857168-07:00 CRITICAL Failed to write tree state /data/magma/magma.10/kvstore-66/rev-000000002/seqIndex/state.6: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.857421-07:00 CRITICAL Failed to write tree state /data/magma/magma.2/kvstore-2/rev-000000004/seqIndex/state.5: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.857490-07:00 CRITICAL Failed to write tree state /data/magma/magma.7/kvstore-7/rev-000000004/seqIndex/state.5: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.857503-07:00 CRITICAL Failed to write tree state /data/magma/magma.4/kvstore-60/rev-000000002/seqIndex/state.5: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.857542-07:00 CRITICAL Failed to write tree state /data/magma/magma.10/kvstore-122/rev-000000002/seqIndex/state.5: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.857766-07:00 CRITICAL Failed to write tree state /data/magma/magma.15/kvstore-15/rev-000000002/seqIndex/state.5: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.857829-07:00 CRITICAL Failed to write tree state /data/magma/magma.2/kvstore-58/rev-000000002/seqIndex/state.5: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.857906-07:00 CRITICAL Failed to write tree state /data/magma/magma.12/kvstore-68/rev-000000002/seqIndex/state.5: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.858119-07:00 CRITICAL Failed to write tree state /data/magma/magma.10/kvstore-178/rev-000000002/seqIndex/state.6: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.886898-07:00 CRITICAL kvstore-12/rev-000000002/seqIndex Error occured during memtable flush (IOError: /data/magma/magma.12/kvstore-12/rev-000000002/seqIndex/sstable.153.data: Unable to open file err=No such file or directory).
|
2020-06-26T04:53:25.886947-07:00 CRITICAL kvstore-120/rev-000000002/seqIndex Error occured during memtable flush (IOError: /data/magma/magma.8/kvstore-120/rev-000000002/seqIndex/sstable.155.data: Unable to open file err=No such file or directory).
|
2020-06-26T04:53:25.887517-07:00 CRITICAL Failed to write tree state /data/magma/magma.12/kvstore-124/rev-000000002/seqIndex/state.5: Unable to open file err=No such file or directory
|
2020-06-26T04:53:25.887665-07:00 CRITICAL kvstore-114/rev-000000002/seqIndex Error occured during memtable flush (IOError: /data/magma/magma.2/kvstore-114/rev-000000002/seqIndex/sstable.155.data: Unable to open file err=No such file or directory).
|
2020-06-26T04:53:25.887677-07:00 CRITICAL kvstore-234/rev-000000002/seqIndex Error occured during memtable flush (IOError: /data/magma/magma.10/kvstore-234/rev-000000002/seqIndex/sstable.157.data: Unable to open file err=No such file or directory).
|
2020-06-26T04:53:25.887806-07:00 CRITICAL kvstore-176/rev-000000002/seqIndex Error occured during memtable flush (IOError: /data/magma/magma.8/kvstore-176/rev-000000002/seqIndex/sstable.154.data: Unable to open file err=No such file or directory).
|
2020-06-26T04:53:25.887837-07:00 CRITICAL kvstore-116/rev-000000002/seqIndex Error occured during memtable flush (IOError: /data/magma/magma.4/kvstore-116/rev-000000002/seqIndex/sstable.155.data: Unable to open file err=No such file or directory).
|
Mini core dump shows nothing:
172.23.97.38 : Stack Trace of 45bc65e3-8b8f-4ef9-8567369b-7decc099.dmp
|
rm -rf /opt/couchbase/var/lib/couchbase/crash/45bc65e3-8b8f-4ef9-8567369b-7decc099.core
|
//opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/45bc65e3-8b8f-4ef9-8567369b-7decc099.dmp > /opt/couchbase/var/lib/couchbase/crash/45bc65e3-8b8f-4ef9-8567369b-7decc099.core
|
gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/45bc65e3-8b8f-4ef9-8567369b-7decc099.core -ex bt full -ex quit
|
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
|
#0 0x00007f8176133387 in ?? ()
|
#0 0x00007f8176133387 in ?? ()
|
#1 0x00007f8176134a78 in ?? ()
|
#2 0x0000000000000020 in ?? ()
|
#3 0x0000000000000000 in ?? ()
|
Note: Data is kept at root@172.23.97.38:/root/magma8-kvstore8.tar.gz
QE Test:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/dp_magma_job.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t volumetests.Magma.volume.SystemTestMagma,nodes_init=2,replicas=1,skip_cleanup=True,num_items=1000000000,num_buckets=1,bucket_names=GleamBook,doc_size=4096,bucket_type=membase,compression_mode=off,iterations=1,batch_size=10,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,maxttl=60,process_concurrency=1 -m rest'
|
