Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44253

Restore state of where fatal exceptions are thrown on bg threads to Breakpad minidump

    XMLWordPrintable

Details

    • 1
    • KV-Engine 2021-Feb

    Description

      Related to MB-43818 / MB-43373, with the introduction of FollyExecutorPool we no longer have a minidump of where a fatal exception was thrown from, only where it was caught.

      This is due to the fact that folly::ThreadPoolExecutor::runTask() has a catch(...) statement inside it, meaning uncaught exceptions are caught (and logged). This is in contrast to CB3ExecutorPool, which didn't have any catch blocks and hence such uncaught exceptions would trigger std::terminate, with a backtrace of where the exception was thrown.

      To partly address this, in MB-43373 we added our own catch(...) inside GlobalTask::execute to avoid the Folly one which calls std::terminate. This restores the fatal behaviour; but leaves the backtrace pointing at where std::terminate was called - the frames relating to where the exception was thrown are lost.

      Additionally, MB-43818 was raised to explicitly record the backtrace of select thrown exceptions via a new throwWithTrace function; which are printed (if present) as part of our terminate_handler. This restores a textual backtrace of where the exception was thrown from.

      However, a couple of recent bugs (and their investigation) has highlighted the deficiencies in this approach - while we can see where the (fatal) exception was thrown from in terms of code lines, we have lost all additional context which was previous present in the minidump - local variables, stack variables, and some globals. This is a not-insignificant loss of information; particularly when considering customer-encountered bugs where we really need as much information as possible.

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-44253
          # Subject Branch Project Status CR V

          Activity

            People

              drigby Dave Rigby (Inactive)
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty