2020-06-26T06:02:25.463510-07:00 INFO 61: (default) DCP (Producer) eq_dcpq:secidx:proj-default-MAINT_STREAM_TOPIC_7c2cc7d3ecd849d461c04f3f1c8157a3-15896629439356753117/3 - (vb:971) ActiveStream::registerCursor name "eq_dcpq:secidx:proj-default-MAINT_STREAM_TOPIC_7c2cc7d3ecd849d461c04f3f1c8157a3-15896629439356753117/3", backfill:true, seqno:3
|
2020-06-26T06:02:25.463616-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2020-06-26T06:02:25.463646-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/item.cc: 480
|
...
|
2020-06-26T06:02:25.487835-07:00 CRITICAL Breakpad caught a crash (Couchbase version 6.6.0-7845). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/51a0585f-c33d-ecc5-312b7470-097e5b60.dmp before terminating.
|
2020-06-26T06:02:25.487850-07:00 CRITICAL Stack backtrace of crashed thread:
|
2020-06-26T06:02:25.488176-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1352cd]
|
2020-06-26T06:02:25.488227-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14d45e]
|
2020-06-26T06:02:25.488272-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14d774]
|
2020-06-26T06:02:25.488312-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fa7d5dab000+0xf130]
|
2020-06-26T06:02:25.488360-07:00 CRITICAL /lib64/libc.so.6(gsignal+0x39) [0x7fa7d59ea000+0x355c9]
|
2020-06-26T06:02:25.488403-07:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7fa7d59ea000+0x36cd8]
|
2020-06-26T06:02:25.488448-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7fa7d64e0000+0x91195]
|
2020-06-26T06:02:25.488478-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x148d52]
|
2020-06-26T06:02:25.488511-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8ef86]
|
2020-06-26T06:02:25.488541-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8efd1]
|
2020-06-26T06:02:25.488569-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8f213]
|
2020-06-26T06:02:25.488591-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x13ff89]
|
2020-06-26T06:02:25.488611-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x140138]
|
2020-06-26T06:02:25.488628-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x6d4fd]
|
2020-06-26T06:02:25.488643-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x6d947]
|
2020-06-26T06:02:25.488657-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7d928]
|
2020-06-26T06:02:25.488695-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x1db972]
|
2020-06-26T06:02:25.488701-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so() [0x7fa7d0c4e000+0x1245a]
|
2020-06-26T06:02:25.488704-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so() [0x7fa7d0c4e000+0x111c6]
|
2020-06-26T06:02:25.488707-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so(couchstore_changes_since+0x1d8) [0x7fa7d0c4e000+0x19fa8]
|
2020-06-26T06:02:25.488712-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x1dc897]
|
2020-06-26T06:02:25.488715-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7eb12]
|
2020-06-26T06:02:25.488718-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7ec58]
|
2020-06-26T06:02:25.488721-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7c93a]
|
2020-06-26T06:02:25.488723-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7cb8e]
|
2020-06-26T06:02:25.488727-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x12b6a4]
|
2020-06-26T06:02:25.488729-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fa7d858f000+0x8f17]
|
2020-06-26T06:02:25.488733-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fa7d5dab000+0x7df3]
|
2020-06-26T06:02:25.488756-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fa7d59ea000+0xf61ad]
|
Grepping for CRITICAL in .106 logs shows:
2020-06-26T06:02:25.463510-07:00 INFO 61: (default) DCP (Producer) eq_dcpq:secidx:proj-default-MAINT_STREAM_TOPIC_7c2cc7d3ecd849d461c04f3f1c8157a3-15896629439356753117/3 - (vb:971) ActiveStream::registerCursor name "eq_dcpq:secidx:proj-default-MAINT_STREAM_TOPIC_7c2cc7d3ecd849d461c04f3f1c8157a3-15896629439356753117/3", backfill:true, seqno:3
2020-06-26T06:02:25.463616-07:00 CRITICAL *** Fatal error encountered during exception handling ***
2020-06-26T06:02:25.463646-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/item.cc: 480
...
2020-06-26T06:02:25.487835-07:00 CRITICAL Breakpad caught a crash (Couchbase version 6.6.0-7845). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/51a0585f-c33d-ecc5-312b7470-097e5b60.dmp before terminating.
2020-06-26T06:02:25.487850-07:00 CRITICAL Stack backtrace of crashed thread:
2020-06-26T06:02:25.488176-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1352cd]
2020-06-26T06:02:25.488227-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14d45e]
2020-06-26T06:02:25.488272-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14d774]
2020-06-26T06:02:25.488312-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fa7d5dab000+0xf130]
2020-06-26T06:02:25.488360-07:00 CRITICAL /lib64/libc.so.6(gsignal+0x39) [0x7fa7d59ea000+0x355c9]
2020-06-26T06:02:25.488403-07:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7fa7d59ea000+0x36cd8]
2020-06-26T06:02:25.488448-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7fa7d64e0000+0x91195]
2020-06-26T06:02:25.488478-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x148d52]
2020-06-26T06:02:25.488511-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8ef86]
2020-06-26T06:02:25.488541-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8efd1]
2020-06-26T06:02:25.488569-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8f213]
2020-06-26T06:02:25.488591-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x13ff89]
2020-06-26T06:02:25.488611-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x140138]
2020-06-26T06:02:25.488628-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x6d4fd]
2020-06-26T06:02:25.488643-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x6d947]
2020-06-26T06:02:25.488657-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7d928]
2020-06-26T06:02:25.488695-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x1db972]
2020-06-26T06:02:25.488701-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so() [0x7fa7d0c4e000+0x1245a]
2020-06-26T06:02:25.488704-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so() [0x7fa7d0c4e000+0x111c6]
2020-06-26T06:02:25.488707-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so(couchstore_changes_since+0x1d8) [0x7fa7d0c4e000+0x19fa8]
2020-06-26T06:02:25.488712-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x1dc897]
2020-06-26T06:02:25.488715-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7eb12]
2020-06-26T06:02:25.488718-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7ec58]
2020-06-26T06:02:25.488721-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7c93a]
2020-06-26T06:02:25.488723-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7cb8e]
2020-06-26T06:02:25.488727-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x12b6a4]
2020-06-26T06:02:25.488729-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fa7d858f000+0x8f17]
2020-06-26T06:02:25.488733-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fa7d5dab000+0x7df3]
2020-06-26T06:02:25.488756-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fa7d59ea000+0xf61ad]
The aforementioned failed expectation is related to
MB-37374(deleted user XATTRs):Item::removeUserXattrs()
Expects(isDeleted());
Expects(bodySize == 0);