Details
-
Improvement
-
Resolution: Fixed
-
Major
-
Cheshire-Cat
-
1
-
KV-Engine 2021-Feb
Description
Related to MB-43818 / MB-43373, with the introduction of FollyExecutorPool we no longer have a minidump of where a fatal exception was thrown from, only where it was caught.
This is due to the fact that folly::ThreadPoolExecutor::runTask() has a catch(...) statement inside it, meaning uncaught exceptions are caught (and logged). This is in contrast to CB3ExecutorPool, which didn't have any catch blocks and hence such uncaught exceptions would trigger std::terminate, with a backtrace of where the exception was thrown.
To partly address this, in MB-43373 we added our own catch(...) inside GlobalTask::execute to avoid the Folly one which calls std::terminate. This restores the fatal behaviour; but leaves the backtrace pointing at where std::terminate was called - the frames relating to where the exception was thrown are lost.
Additionally, MB-43818 was raised to explicitly record the backtrace of select thrown exceptions via a new throwWithTrace function; which are printed (if present) as part of our terminate_handler. This restores a textual backtrace of where the exception was thrown from.
However, a couple of recent bugs (and their investigation) has highlighted the deficiencies in this approach - while we can see where the (fatal) exception was thrown from in terms of code lines, we have lost all additional context which was previous present in the minidump - local variables, stack variables, and some globals. This is a not-insignificant loss of information; particularly when considering customer-encountered bugs where we really need as much information as possible.
Attachments
Issue Links
For Gerrit Dashboard: MB-44253 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
145810,3 | MB-44253: Update folly to v2020.08.24.00-couchbase-cb4 | master | tlm | Status: MERGED | +2 | +1 |
146321,3 | Revert "MB-43818: GlobalTask::execute: Log task info if exception thrown" | master | kv_engine | Status: MERGED | +2 | +1 |
147330,2 | Revert "MB-43373: Abort memcached if exception thrown from GlobalTask::run v2" | master | kv_engine | Status: MERGED | +2 | +1 |
176913,5 | MB-44253: Folly: Remove try/catch from Executor*::runTask methods | master | tlm | Status: MERGED | +2 | +1 |
176929,4 | MB-44253: Test: Should crash if GlobalTask::run throws | master | kv_engine | Status: MERGED | +2 | +1 |