Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
6.6.2
-
Enterprise Edition 6.6.2 build 1779316
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
ISSUE
Memcached process is crashing continuously with below mentioned error in the longevity test started against 6.6.2-1779316 toy build.
Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at kv_engine/engines/ep/src/dcp/active_stream.cc: 1321
Crash is related to https://review.couchbase.org/c/kv_engine/+/177931Â change added to the toy build.
Crash is occurring continuously from the first rebalance step of the test itself.
QE TEST
-test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
|
Day - 1
Cycle - 1
Scale - 3
CRASH
2022-07-28T03:40:28.715027-07:00 CRITICAL *** Fatal error encountered during exception handling *** |
2022-07-28T03:40:28.715115-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/toy-unix/kv_engine/engines/ep/src/dcp/active_stream.cc: 1321 |
2022-07-28T03:40:28.883852-07:00 WARNING (default) Slow runtime for 'DurabilityCompletionTask' on thread nonIO_worker_1: 110 ms |
2022-07-28T03:40:28.888118-07:00 INFO (DISTRICT) DCP (Producer) eq_dcpq:cbas:Default.Local:DISTRICT:518a5eec11eac81579e270d720bb7f2e:13 - Notifying paused connection now that DcpProducer::BufferLog is no longer full; ackedBytes:399860, bytesSent:104584, maxBytes:131072 |
2022-07-28T03:40:28.888679-07:00 INFO (DISTRICT) DCP (Producer) eq_dcpq:cbas:Default.Local:DISTRICT:518a5eec11eac81579e270d720bb7f2e:9 - Notifying paused connection now that DcpProducer::BufferLog is no longer full; ackedBytes:346354, bytesSent:105480, maxBytes:131072 |
2022-07-28T03:40:28.889340-07:00 INFO (DISTRICT) DCP (Producer) eq_dcpq:cbas:Default.Local:DISTRICT:518a5eec11eac81579e270d720bb7f2e:12 - Notifying paused connection now that DcpProducer::BufferLog is no longer full; ackedBytes:453644, bytesSent:105352, maxBytes:131072 |
2022-07-28T03:40:28.893584-07:00 INFO (DISTRICT) DCP (Producer) eq_dcpq:cbas:Default.Local:DISTRICT:518a5eec11eac81579e270d720bb7f2e:11 - Notifying paused connection now that DcpProducer::BufferLog is no longer full; ackedBytes:346482, bytesSent:105183, maxBytes:131072 |
2022-07-28T03:40:28.894294-07:00 INFO 588: (DISTRICT) DCP (Producer) eq_dcpq:cbas:Default.Local:DISTRICT:518a5eec11eac81579e270d720bb7f2e:13 - (vb:181) ActiveStream::transitionState: Transitioning from backfilling to in-memory |
2022-07-28T03:40:28.898490-07:00 CRITICAL Breakpad caught a crash (Couchbase version 6.6.2-1779316). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/15db4940-fe0b-5342-5e8c17d4-3aa3aa3e.dmp before terminating. |
2022-07-28T03:40:28.898530-07:00 CRITICAL Stack backtrace of crashed thread: |
2022-07-28T03:40:28.898774-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x136d3d] |
2022-07-28T03:40:28.898792-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14f86e] |
2022-07-28T03:40:28.898806-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14fb84] |
2022-07-28T03:40:28.898815-07:00 CRITICAL /lib64/libpthread.so.0() [0x7f91964f6000+0xf5d0] |
2022-07-28T03:40:28.898850-07:00 CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7f9196129000+0x36207] |
2022-07-28T03:40:28.898878-07:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7f9196129000+0x378f8] |
2022-07-28T03:40:28.898939-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f9196c2b000+0x91195] |
2022-07-28T03:40:28.898955-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x14a9c2] |
2022-07-28T03:40:28.898996-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f9196c2b000+0x8ef86] |
2022-07-28T03:40:28.899033-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f9196c2b000+0x8efd1] |
2022-07-28T03:40:28.899073-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f9196c2b000+0x8f213] |
2022-07-28T03:40:28.899087-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f919169c000+0x5ef98] |
2022-07-28T03:40:28.899097-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f919169c000+0x6f97b] |
2022-07-28T03:40:28.899106-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f919169c000+0x726af] |
2022-07-28T03:40:28.899166-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f919169c000+0x72b23] |
2022-07-28T03:40:28.899178-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f919169c000+0x72c04] |
2022-07-28T03:40:28.899187-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f919169c000+0x7beb9] |
2022-07-28T03:40:28.899195-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f919169c000+0x12ddd4] |
2022-07-28T03:40:28.899204-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f9198cda000+0x8ef7] |
2022-07-28T03:40:28.899211-07:00 CRITICAL /lib64/libpthread.so.0() [0x7f91964f6000+0x7dd5] |
2022-07-28T03:40:28.899245-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f9196129000+0xfdead] |
PFA stack trace of the crash as well as logs from all the nodes.