Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45752

[Magma/Couchstore]: memcached crashed - snapshot_range_t(25276,25252) requires start <= end

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • couchbase-bucket
    • 7.0.0-4960
    • Untriaged
    • 1
    • Yes
    • KV-Engine CC Final Sprint

    Description

      Crash Dump

      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
       #0  0x00007f33c6e4c387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
       55	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
       #0  0x00007f33c6e4c387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
               resultvar = 0
               pid = 17511
               selftid = 17547
       #1  0x00007f33c6e4da78 in __GI_abort () at abort.c:90
               save_stage = 2
               act = {__sigaction_handler = {sa_handler = 0x7f33c71de1c0 <_IO_2_1_stderr_>, sa_sigaction = 0x7f33c71de1c0 <_IO_2_1_stderr_>}, sa_mask = {__val = {139860357700137, 0, 139860357218915, 139856434661896, 139860360683968, 1, 139860360684099, 139860360668032, 139860357225086, 139860360683968, 10, 139858093804064, 4429344, 139858295048496, 139860357226355, 139860360683968}}, sa_flags = -946086896, sa_restorer = 0xd61620 <stderr>}
               sigs = {__val = {32, 0 <repeats 15 times>}}
       #2  0x00007f33c79b363c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
               terminating = false
               t = <optimized out>
       #3  0x00000000009131ef in backtrace_terminate_handler() ()
       No symbol table info available.
       #4  0x00007f33c79be8f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
       No locals.
       #5  0x00007f33c79be961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
       No locals.
       #6  0x00007f33c79bebf4 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0xd61a20 <typeinfo for std::runtime_error>, dest=0x439620 <_ZNSt13runtime_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95
               globals = <optimized out>
               header = 0x7f3340000e50
       #7  0x000000000076e6e6 in snapshot_range_t::checkInvariant() const ()
       No symbol table info available.
       #8  0x00000000007c5956 in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) ()
       No symbol table info available.
       #9  0x00000000007c682f in EPBucket::flushVBucket(Vbid) ()
       No symbol table info available.
       #10 0x000000000066dc50 in Flusher::flushVB() ()
       No symbol table info available.
       #11 0x000000000066e719 in Flusher::step(GlobalTask*) ()
       No symbol table info available.
       #12 0x0000000000670957 in GlobalTask::execute() ()
       No symbol table info available.
       #13 0x000000000066baa0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const ()
       No symbol table info available.
       #14 0x0000000000976050 in folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) ()
       No symbol table info available.
       #15 0x000000000095dd2a in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
       No symbol table info available.
       #16 0x0000000000979009 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
       No symbol table info available.
       #17 0x000000000066a02d in void folly::detail::function::FunctionTraits<void ()>::callBig<CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) ()
       No symbol table info available.
       #18 0x00007f33c79e7d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80
       No locals.
       #19 0x00007f33c71ebea5 in start_thread (arg=0x7f334bfff700) at pthread_create.c:307
               __res = <optimized out>
               pd = 0x7f334bfff700
               now = <optimized out>
               unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139858295125760, -6349787822976884426, 0, 8392704, 0, 139858295125760, 6451285948144329014, 6450997050070180150}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
               not_first_call = <optimized out>
               pagesize_m1 = <optimized out>
               sp = <optimized out>
               freesize = <optimized out>
       #20 0x00007f33c6f148dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
       No locals.
      

      172.23.96.90

      cbcollect_info_ns_1@172.23.96.90_20210418-024313/de850f4f-d1e2-4d57-bd0ea2b4-cfd2ac96.dmp
      2021-04-17T19:40:54.590759-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2021-04-17T19:40:54.590811-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): snapshot_range_t(13616,13115) requires start <= end
      2021-04-17T19:40:55.149221-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-4960). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/de850f4f-d1e2-4d57-bd0ea2b4-cfd2ac96.dmp before terminating.
      2021-04-17T19:40:55.149249-07:00 CRITICAL Stack backtrace of crashed thread:
      2021-04-17T19:40:55.149434-07:00 CRITICAL     #0  /opt/couchbase/bin/memcached() [0x400000+0x5066b3]
      2021-04-17T19:40:55.149455-07:00 CRITICAL     #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x52d3ea]
      2021-04-17T19:40:55.149472-07:00 CRITICAL     #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x52d728]
      2021-04-17T19:40:55.149479-07:00 CRITICAL     #3  /lib64/libpthread.so.0() [0x7fdf564e7000+0xf630]
      2021-04-17T19:40:55.149502-07:00 CRITICAL     #4  /lib64/libc.so.6(gsignal+0x37) [0x7fdf56119000+0x36387]
      2021-04-17T19:40:55.149522-07:00 CRITICAL     #5  /lib64/libc.so.6(abort+0x148) [0x7fdf56119000+0x37a78]
      2021-04-17T19:40:55.149544-07:00 CRITICAL     #6  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fdf56c1d000+0x9963c]
      2021-04-17T19:40:55.149562-07:00 CRITICAL     #7  /opt/couchbase/bin/memcached() [0x400000+0x5131ef]
      2021-04-17T19:40:55.149579-07:00 CRITICAL     #8  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fdf56c1d000+0xa48f6]
      2021-04-17T19:40:55.149598-07:00 CRITICAL     #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fdf56c1d000+0xa4961]
      2021-04-17T19:40:55.149614-07:00 CRITICAL     #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fdf56c1d000+0xa4bf4]
      2021-04-17T19:40:55.149631-07:00 CRITICAL     #11 /opt/couchbase/bin/memcached() [0x400000+0x36e6e6]
      2021-04-17T19:40:55.149646-07:00 CRITICAL     #12 /opt/couchbase/bin/memcached() [0x400000+0x3c5956]
      2021-04-17T19:40:55.149659-07:00 CRITICAL     #13 /opt/couchbase/bin/memcached() [0x400000+0x3c682f]
      2021-04-17T19:40:55.149673-07:00 CRITICAL     #14 /opt/couchbase/bin/memcached() [0x400000+0x26dc50]
      2021-04-17T19:40:55.149686-07:00 CRITICAL     #15 /opt/couchbase/bin/memcached() [0x400000+0x26e719]
      2021-04-17T19:40:55.149696-07:00 CRITICAL     #16 /opt/couchbase/bin/memcached() [0x400000+0x270957]
      2021-04-17T19:40:55.149706-07:00 CRITICAL     #17 /opt/couchbase/bin/memcached() [0x400000+0x26baa0]
      2021-04-17T19:40:55.149724-07:00 CRITICAL     #18 /opt/couchbase/bin/memcached() [0x400000+0x576050]
      2021-04-17T19:40:55.149748-07:00 CRITICAL     #19 /opt/couchbase/bin/memcached() [0x400000+0x55dd2a]
      2021-04-17T19:40:55.149764-07:00 CRITICAL     #20 /opt/couchbase/bin/memcached() [0x400000+0x579009]
      2021-04-17T19:40:55.149776-07:00 CRITICAL     #21 /opt/couchbase/bin/memcached() [0x400000+0x26a02d]
      2021-04-17T19:40:55.149805-07:00 CRITICAL     #22 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fdf56c1d000+0xcdd40]
      2021-04-17T19:40:55.149811-07:00 CRITICAL     #23 /lib64/libpthread.so.0() [0x7fdf564e7000+0x7ea5]
      2021-04-17T19:40:55.149841-07:00 CRITICAL     #24 /lib64/libc.so.6(clone+0x6d) [0x7fdf56119000+0xfe8dd]
      

      QE Tests:

      /data/workspace/centos-p0-magma-vset00-00-crash_during_recovery_7.0_P1/job_logs/testrunner-21-Apr-17_16-52-45/test_5
       
      /data/workspace/centos-p0-magma-vset00-00-crash_during_recovery_7.0_P1/job_logs/testrunner-21-Apr-17_16-52-45/test_6
       
      /data/workspace/centos-p0-magma-vset00-00-crash_during_recovery_with_replicas_7.0_P2/job_logs/testrunner-21-Apr-17_23-21-07/test_1
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              james.harrison James Harrison (Inactive)
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty