Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35334

Snapshot range invariant (snapshot_range_t(a,b) requires start <= end) broken after crash/restart

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • None
    • 6.5.0
    • couchbase-bucket
    • 6.5.0-3894
    • Triaged
    • No

    Description

      Separated out from MB-35332, the snapshot range invariant is broken and results in an abort of memcached on node .104.203 after the "main" crash in MB-35332:

      2019-07-30T01:59:41.127509-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2019-07-30T01:59:41.127578-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): snapshot_range_t(126,0) requires start <= end
      ...
      2019-07-30T01:59:41.187086-07:00 CRITICAL Breakpad caught a crash (Couchbase version 6.5.0-3894). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/02fc502b-3f3f-7886-0abb7d7a-5fa5a08b.dmp before terminating.
      2019-07-30T01:59:41.187113-07:00 CRITICAL Stack backtrace of crashed thread:
      2019-07-30T01:59:41.187313-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x133d50]
      2019-07-30T01:59:41.187336-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14b28e]
      2019-07-30T01:59:41.187357-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14b5a4]
      2019-07-30T01:59:41.187368-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f24b9e03000+0xf100]
      2019-07-30T01:59:41.187405-07:00 CRITICAL     /lib64/libc.so.6(gsignal+0x37) [0x7f24b9a42000+0x355f7]
      2019-07-30T01:59:41.187441-07:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7f24b9a42000+0x36ce8]
      2019-07-30T01:59:41.187490-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f24ba538000+0x91195]
      2019-07-30T01:59:41.187525-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x146d6d]
      2019-07-30T01:59:41.187577-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f24ba538000+0x8ef86]
      2019-07-30T01:59:41.187622-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f24ba538000+0x8efd1]
      2019-07-30T01:59:41.187673-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f24ba538000+0x8f213]
      2019-07-30T01:59:41.187705-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f24b5367000+0x70a22]
      2019-07-30T01:59:41.187724-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f24b5367000+0xee86b]
      2019-07-30T01:59:41.187743-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f24b5367000+0x13cba5]
      2019-07-30T01:59:41.187762-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f24b5367000+0x13d06d]
      2019-07-30T01:59:41.187780-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f24b5367000+0x13644f]
      2019-07-30T01:59:41.187788-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f24bc3d3000+0x8f27]
      2019-07-30T01:59:41.187797-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f24b9e03000+0x7dc5]
      2019-07-30T01:59:41.187846-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7f24b9a42000+0xf621d]
      

      After this crash, the subsequent restart hits the same issue again (and again...):

      2019-07-30T01:59:46.010598-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): snapshot_range_t(126,0) requires start <= end
      2019-07-30T01:59:46.074329-07:00 CRITICAL Breakpad caught a crash (Couchbase version 6.5.0-3894). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/6d194160-7cbe-ab2d-35f2fbe5-61662911.dmp before terminating.
      ...
      2019-07-30T01:59:57.490175-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): snapshot_range_t(126,0) requires start <= end
      2019-07-30T01:59:57.553201-07:00 CRITICAL Breakpad caught a crash (Couchbase version 6.5.0-3894). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/5c8a056f-162a-0bd4-324e5d1a-2e2db4f2.dmp before terminating.
      ...
      

      Attachments

        Issue Links

          Activity

            People

              drigby Dave Rigby (Inactive)
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty