Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
Cheshire-Cat
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.0.0-3814
Test : -test tests/2i/cheshirecat/test_idx_clusterops_cheshire_cat_basic.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat.yml
Scale : 2
Iteration : 1st
This is the GSI test where there are KV, GSI and Query nodes. The test run mutations on KV collections continuously while there are GSI indexes consuming them, and N1QL queries running constantly. The test also does cluster ops for indexer nodes.
The test started rebalance operation to remove 2 indexer nodes - 172.23.97.77,172.23.97.83 out of the cluster at 2020-11-22T00:42.
Starting 2020-11-22T03:56, memcached crashes were observed across various KV nodes - 172.23.104.16, 172.23.104.17, 172.23.104.18 & 172.23.104.19 multiple times.
Here is an excerpt from the memcached logs from 172.23.104.16 -
2020-11-22T03:56:44.625015-08:00 CRITICAL (bucket3) Flush::saveCollectionStats caught exception ThrowExceptionUnderflowPolicy current:0 arg:-1
|
2020-11-22T03:56:44.625051-08:00 CRITICAL (bucket3) Flush::saveCollectionStats items:78 + -78, highSeq:92967, diskSize:55148 + -55149
|
2020-11-22T03:56:44.625238-08:00 CRITICAL (bucket3) CouchKVStore::saveDocs vb:4 saveCollectionStats exception ThrowExceptionUnderflowPolicy current:0 arg:-1 docs:1170
|
2020-11-22T03:56:44.625261-08:00 CRITICAL (bucket3) CouchKVStore::saveDocs info key:cid:0xe:doc_27528, seq:92078, rev:10, deleted:1, cm:3, value_size:0
|
2020-11-22T03:56:44.625270-08:00 CRITICAL (bucket3) CouchKVStore::saveDocs info key:cid:0xe:doc_30163, seq:92393, rev:10, deleted:1, cm:3, value_size:0
|
2020-11-22T03:56:44.625278-08:00 CRITICAL (bucket3) CouchKVStore::saveDocs info key:cid:0xe:doc_30611, seq:92712, rev:10, deleted:1, cm:3, value_size:0
|
2020-11-22T03:56:44.625284-08:00 CRITICAL (bucket3) CouchKVStore::saveDocs info key:cid:0xe:doc_30781, seq:92132, rev:10, deleted:1, cm:3, value_size:0
|
...
|
...
|
...
|
2020-11-22T03:56:44.711107-08:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2020-11-22T03:56:44.711166-08:00 CRITICAL Caught unhandled std::exception-derived exception. what(): ThrowExceptionUnderflowPolicy current:0 arg:-1
|
2020-11-22T03:56:44.713005-08:00 WARNING 176: (bucket3) DCP (Producer) eq_dcpq:secidx:proj-bucket3:scope_5:coll_-2-INIT_STREAM_TOPIC_b84632af24fa7cc5c2eae00f2b54f5dc-9508391014086615316/1 - (vb:165) processItems() for vbucket which does not exist
|
2020-11-22T03:56:44.719162-08:00 WARNING 176: (bucket3) DCP (Producer) eq_dcpq:secidx:proj-bucket3:scope_5:coll_-2-INIT_STREAM_TOPIC_b84632af24fa7cc5c2eae00f2b54f5dc-9508391014086615316/1 - (vb:165) processItems() for vbucket which does not exist
|
2020-11-22T03:56:45.280865-08:00 WARNING (No Engine) Slow runtime for 'DurabilityTimeoutVisitor on vb:115' on thread NonIO_worker_1: 210 ms
|
...
|
...
|
2020-11-22T03:56:45.964001-08:00 CRITICAL (bucket3) Flush::saveCollectionStats caught exception ThrowExceptionUnderflowPolicy current:0 arg:-1
|
2020-11-22T03:56:45.964120-08:00 CRITICAL (bucket3) Flush::saveCollectionStats items:84 + -84, highSeq:88138, diskSize:61451 + -61452
|
2020-11-22T03:56:45.964194-08:00 CRITICAL (bucket3) CouchKVStore::saveDocs vb:815 saveCollectionStats exception ThrowExceptionUnderflowPolicy current:0 arg:-1 docs:1260
|
2020-11-22T03:56:45.964267-08:00 CRITICAL (bucket3) CouchKVStore::saveDocs info key:cid:0xe:doc_29408, seq:87688, rev:10, deleted:1, cm:3, value_size:0
|
2020-11-22T03:56:45.964282-08:00 CRITICAL (bucket3) CouchKVStore::saveDocs info key:cid:0xe:doc_29598, seq:87091, rev:10, deleted:1, cm:3, value_size:0
|
...
|
...
|
...
|
2020-11-22T03:56:46.009077-08:00 CRITICAL (bucket3) CouchKVStore::saveDocs info key:cid:0x27:doc_99088, seq:87875, rev:10, deleted:1, cm:3, value_size:0
|
2020-11-22T03:56:46.009083-08:00 CRITICAL (bucket3) CouchKVStore::saveDocs info key:cid:0x27:doc_99118, seq:87049, rev:10, deleted:1, cm:3, value_size:0
|
2020-11-22T03:56:46.009099-08:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-3814). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/33a23dd1-04d9-4d99-42526db8-ced1571c.dmp before terminating.
|
2020-11-22T03:56:46.009115-08:00 CRITICAL Stack backtrace of crashed thread:
|
2020-11-22T03:56:46.009139-08:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2020-11-22T03:56:46.009430-08:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x19dffd]
|
2020-11-22T03:56:46.009513-08:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x1b377a]
|
2020-11-22T03:56:46.009549-08:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x1b3ab8]
|
2020-11-22T03:56:46.009575-08:00 CRITICAL /lib64/libpthread.so.0() [0x7fa716045000+0xf6d0]
|
2020-11-22T03:56:46.009618-08:00 CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7fa715c78000+0x36277]
|
2020-11-22T03:56:46.009662-08:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7fa715c78000+0x37968]
|
2020-11-22T03:56:46.009754-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7fa71677a000+0x91195]
|
2020-11-22T03:56:46.009788-08:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1ada22]
|
2020-11-22T03:56:46.009829-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa71677a000+0x8ef86]
|
2020-11-22T03:56:46.009868-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa71677a000+0x8efd1]
|
2020-11-22T03:56:46.009904-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(__cxa_rethrow+0x46) [0x7fa71677a000+0x8f266]
|
2020-11-22T03:56:46.009935-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fa719d67000+0x24c0c3]
|
2020-11-22T03:56:46.009960-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fa719d67000+0x24c498]
|
2020-11-22T03:56:46.009971-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fa719d67000+0x24ca45]
|
2020-11-22T03:56:46.009985-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fa719d67000+0x1247c2]
|
2020-11-22T03:56:46.010004-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fa719d67000+0x128104]
|
2020-11-22T03:56:46.010016-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fa719d67000+0x18216c]
|
2020-11-22T03:56:46.010028-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fa719d67000+0x183349]
|
2020-11-22T03:56:46.010036-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fa719d67000+0x186073]
|
2020-11-22T03:56:46.010048-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fa719d67000+0x872cf]
|
2020-11-22T03:56:46.010060-08:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fa7187fd000+0x10917]
|
2020-11-22T03:56:46.010069-08:00 CRITICAL /lib64/libpthread.so.0() [0x7fa716045000+0x7e25]
|
2020-11-22T03:56:46.010118-08:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fa715c78000+0xfebad]
|
2020-11-22T03:56:46.010490-08:00 WARNING 176: (bucket3) DCP (Producer) eq_dcpq:secidx:proj-bucket3:scope_5:coll_-2-INIT_STREAM_TOPIC_b84632af24fa7cc5c2eae00f2b54f5dc-9508391014086615316/1 - (vb:97) processItems() for vbucket which does not exist
|
2020-11-22T03:56:46.011600-08:00 WARNING 176: (bucket3) DCP (Producer) eq_dcpq:secidx:proj-bucket3:scope_5:coll_-2-INIT_STREAM_TOPIC_b84632af24fa7cc5c2eae00f2b54f5dc-9508391014086615316/1 - (vb:97) processItems() for vbucket which does not exist
|
2020-11-22T03:56:46.013363-08:00 INFO ---------- Closing logfile
|
Attachments
Issue Links
- duplicates
-
-
- Closed
-