Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41649

Memcached crashed: ActiveDurabilityMonitor::commit vb:808 failed with status:1

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      /data/workspace/centos-p0-transaction-vset00-00-multi_bucket_durability_majority_6.5_P0_Set2/job_logs/testrunner-20-Sep-21_13-10-37/test_3/collectinfo-2020-09-21T222659-ns_1@172.23.105.246.zip
       
       
      cbcollect_info_ns_1@172.23.105.246_20200921-222659/0bad4d46-88de-44a6-d6ab2ebb-96b3c813.dmp
      2020-09-21T13:19:13.277508-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2020-09-21T13:19:13.277566-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): ActiveDurabilityMonitor::commit vb:808 failed with status:1
      2020-09-21T13:19:13.445686-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-3154). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/0bad4d46-88de-44a6-d6ab2ebb-96b3c813.dmp before terminating.
      2020-09-21T13:19:13.445719-07:00 CRITICAL Stack backtrace of crashed thread:
      2020-09-21T13:19:13.445936-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x193a6d]
      2020-09-21T13:19:13.445964-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x1a9dea]
      2020-09-21T13:19:13.445978-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x1aa128]
      2020-09-21T13:19:13.445992-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f70497de000+0xf5d0]
      2020-09-21T13:19:13.446016-07:00 CRITICAL     /lib64/libc.so.6(gsignal+0x37) [0x7f7049411000+0x36207]
      2020-09-21T13:19:13.446043-07:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7f7049411000+0x378f8]
      2020-09-21T13:19:13.446094-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f7049f13000+0x91195]
      2020-09-21T13:19:13.446108-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x1a47b2]
      2020-09-21T13:19:13.446125-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7049f13000+0x8ef86]
      2020-09-21T13:19:13.446232-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7049f13000+0x8efd1]
      2020-09-21T13:19:13.446256-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f7049f13000+0x8f213]
      2020-09-21T13:19:13.446276-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f704d4c6000+0x10d730]
      2020-09-21T13:19:13.446285-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f704d4c6000+0x108746]
      2020-09-21T13:19:13.446292-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f704d4c6000+0x108883]
      2020-09-21T13:19:13.446302-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f704d4c6000+0x1e3b33]
      2020-09-21T13:19:13.446311-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f704d4c6000+0x11039d]
      2020-09-21T13:19:13.446319-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f704d4c6000+0x17e713]
      2020-09-21T13:19:13.446329-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f704d4c6000+0x8244f]
      2020-09-21T13:19:13.446336-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f704bf90000+0x10947]
      2020-09-21T13:19:13.446343-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f70497de000+0x7dd5]
      2020-09-21T13:19:13.446373-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7f7049411000+0xfdead]
      

      QE test:

      Atomicity.basic_ops.basic_ops:
          test_basic_commit,nodes_init=2,replicas=1,transaction_commit=True,op_type=create;update;delete,bucket_type=ephemeral,num_buckets=2,GROUP=P0_Set2
      

      CC: Ashwin Govindarajulu

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            owend Daniel Owen added a comment - - edited

            Hey Ritesh Agarwal. Thanks for raising the defect.
            Do you have the full BT available that you could post in the ticket, also is this a new test, or is it a regression?
            If it is a new test be interested if you could run on 6.6.0 to see if its something new in CC.

            thanks

            owend Daniel Owen added a comment - - edited Hey Ritesh Agarwal . Thanks for raising the defect. Do you have the full BT available that you could post in the ticket, also is this a new test, or is it a regression? If it is a new test be interested if you could run on 6.6.0 to see if its something new in CC. thanks
            ritesh.agarwal Ritesh Agarwal added a comment - - edited

            Hi Daniel Owen: It seems to be an intermittent issue. On rerunning the same test passed on the same build.

            Found the minidump from the jenkins job:

            172.23.105.246 : Stack Trace of first crash: 0bad4d46-88de-44a6-d6ab2ebb-96b3c813.dmp

            Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
             #0  0x00007f7049447207 in raise () from /lib64/libc.so.6
             #0  0x00007f7049447207 in raise () from /lib64/libc.so.6
             #1  0x00007f70494488f8 in abort () from /lib64/libc.so.6
             #2  0x00007f7049fa4195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #3  0x00000000005a47b2 in backtrace_terminate_handler() ()
             #4  0x00007f7049fa1f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #5  0x00007f7049fa1fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #6  0x00007f7049fa2213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #7  0x00007f704d5d3730 in void ActiveDurabilityMonitor::throwException<std::logic_error>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /opt/couchbase/bin/../lib/libep.so
             #8  0x00007f704d5ce746 in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so
             #9  0x00007f704d5ce883 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () from /opt/couchbase/bin/../lib/libep.so
             #10 0x00007f704d6a9b33 in VBucket::processResolvedSyncWrites() () from /opt/couchbase/bin/../lib/libep.so
             #11 0x00007f704d5d639d in DurabilityCompletionTask::run() () from /opt/couchbase/bin/../lib/libep.so
             #12 0x00007f704d644713 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
             #13 0x00007f704d54844f in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
             #14 0x00007f704bfa0947 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
             #15 0x00007f70497e5dd5 in start_thread () from /lib64/libpthread.so.0
             #16 0x00007f704950eead in clone () from /lib64/libc.so.6
            

            Node 172.23.105.238 - Core dump seen: 1
            172.23.105.238 : Stack Trace of first crash: de856e74-8b16-4f05-c57aa9ae-e889bb98.dmp

            Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
             #0  0x00007f0ec1f48207 in raise () from /lib64/libc.so.6
             #0  0x00007f0ec1f48207 in raise () from /lib64/libc.so.6
             #1  0x00007f0ec1f498f8 in abort () from /lib64/libc.so.6
             #2  0x00007f0ec2aa5195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #3  0x00000000005a47b2 in backtrace_terminate_handler() ()
             #4  0x00007f0ec2aa2f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #5  0x00007f0ec2aa2fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #6  0x00007f0ec2aa3213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #7  0x00007f0ec60d4730 in void ActiveDurabilityMonitor::throwException<std::logic_error>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /opt/couchbase/bin/../lib/libep.so
             #8  0x00007f0ec60cf746 in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so
             #9  0x00007f0ec60cf883 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () from /opt/couchbase/bin/../lib/libep.so
             #10 0x00007f0ec61aab33 in VBucket::processResolvedSyncWrites() () from /opt/couchbase/bin/../lib/libep.so
             #11 0x00007f0ec60d739d in DurabilityCompletionTask::run() () from /opt/couchbase/bin/../lib/libep.so
             #12 0x00007f0ec6145713 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
             #13 0x00007f0ec604944f in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
             #14 0x00007f0ec4aa1947 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
             #15 0x00007f0ec22e6dd5 in start_thread () from /lib64/libpthread.so.0
             #16 0x00007f0ec200fead in clone () from /lib64/libc.so.6
            

            ritesh.agarwal Ritesh Agarwal added a comment - - edited Hi Daniel Owen : It seems to be an intermittent issue. On rerunning the same test passed on the same build. Found the minidump from the jenkins job: 172.23.105.246 : Stack Trace of first crash: 0bad4d46-88de-44a6-d6ab2ebb-96b3c813.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. # 0 0x00007f7049447207 in raise () from /lib64/libc.so. 6 # 0 0x00007f7049447207 in raise () from /lib64/libc.so. 6 # 1 0x00007f70494488f8 in abort () from /lib64/libc.so. 6 # 2 0x00007f7049fa4195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so. 6 # 3 0x00000000005a47b2 in backtrace_terminate_handler() () # 4 0x00007f7049fa1f86 in __cxxabiv1::__terminate( void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so. 6 # 5 0x00007f7049fa1fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so. 6 # 6 0x00007f7049fa2213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so. 6 # 7 0x00007f704d5d3730 in void ActiveDurabilityMonitor::throwException<std::logic_error>(std::__cxx11::basic_string< char , std::char_traits< char >, std::allocator< char > > const &, std::__cxx11::basic_string< char , std::char_traits< char >, std::allocator< char > > const &) const () from /opt/couchbase/bin/../lib/libep.so # 8 0x00007f704d5ce746 in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const &) () from /opt/couchbase/bin/../lib/libep.so # 9 0x00007f704d5ce883 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () from /opt/couchbase/bin/../lib/libep.so # 10 0x00007f704d6a9b33 in VBucket::processResolvedSyncWrites() () from /opt/couchbase/bin/../lib/libep.so # 11 0x00007f704d5d639d in DurabilityCompletionTask::run() () from /opt/couchbase/bin/../lib/libep.so # 12 0x00007f704d644713 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so # 13 0x00007f704d54844f in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so # 14 0x00007f704bfa0947 in platform_thread_wrap( void *) () from /opt/couchbase/bin/../lib/libplatform_so.so. 0.1 . 0 # 15 0x00007f70497e5dd5 in start_thread () from /lib64/libpthread.so. 0 # 16 0x00007f704950eead in clone () from /lib64/libc.so. 6 Node 172.23.105.238 - Core dump seen: 1 172.23.105.238 : Stack Trace of first crash: de856e74-8b16-4f05-c57aa9ae-e889bb98.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. # 0 0x00007f0ec1f48207 in raise () from /lib64/libc.so. 6 # 0 0x00007f0ec1f48207 in raise () from /lib64/libc.so. 6 # 1 0x00007f0ec1f498f8 in abort () from /lib64/libc.so. 6 # 2 0x00007f0ec2aa5195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so. 6 # 3 0x00000000005a47b2 in backtrace_terminate_handler() () # 4 0x00007f0ec2aa2f86 in __cxxabiv1::__terminate( void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so. 6 # 5 0x00007f0ec2aa2fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so. 6 # 6 0x00007f0ec2aa3213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so. 6 # 7 0x00007f0ec60d4730 in void ActiveDurabilityMonitor::throwException<std::logic_error>(std::__cxx11::basic_string< char , std::char_traits< char >, std::allocator< char > > const &, std::__cxx11::basic_string< char , std::char_traits< char >, std::allocator< char > > const &) const () from /opt/couchbase/bin/../lib/libep.so # 8 0x00007f0ec60cf746 in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const &) () from /opt/couchbase/bin/../lib/libep.so # 9 0x00007f0ec60cf883 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () from /opt/couchbase/bin/../lib/libep.so # 10 0x00007f0ec61aab33 in VBucket::processResolvedSyncWrites() () from /opt/couchbase/bin/../lib/libep.so # 11 0x00007f0ec60d739d in DurabilityCompletionTask::run() () from /opt/couchbase/bin/../lib/libep.so # 12 0x00007f0ec6145713 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so # 13 0x00007f0ec604944f in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so # 14 0x00007f0ec4aa1947 in platform_thread_wrap( void *) () from /opt/couchbase/bin/../lib/libplatform_so.so. 0.1 . 0 # 15 0x00007f0ec22e6dd5 in start_thread () from /lib64/libpthread.so. 0 # 16 0x00007f0ec200fead in clone () from /lib64/libc.so. 6
            owend Daniel Owen added a comment -

            Many thanks Ritesh Agarwal that's very useful. Interesting to see we are throwing a logic error.
            Looks like might be related to MB-41089

            owend Daniel Owen added a comment - Many thanks Ritesh Agarwal that's very useful. Interesting to see we are throwing a logic error. Looks like might be related to MB-41089
            ashwin.govindarajulu Ashwin Govindarajulu added a comment - - edited

            Not seeing this crash on 6.6.1-9072 using Transaction_client:1.1.0 and Java client: 3.0.8-SNAPSHOT.

            Able to reproduce this crash on CC build 7.0.0-2908 as well

             

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - - edited Not seeing this crash on 6.6.1-9072 using Transaction_client:1.1.0 and Java client: 3.0.8-SNAPSHOT. Able to reproduce this crash on CC build 7.0.0-2908 as well  

            Looks like a dupe of MB-41089. Ashwin Govindarajulu, could you please re-run with couchbase-server-6.6.1-9180 or newer and couchbase-server-7.0.0-3701 or newer?

            ben.huddleston Ben Huddleston added a comment - Looks like a dupe of MB-41089 . Ashwin Govindarajulu , could you please re-run with couchbase-server-6.6.1-9180 or newer and couchbase-server-7.0.0-3701 or newer?
            ashwin.govindarajulu Ashwin Govindarajulu added a comment - - edited Not seeing this crash on 7.0.0-3701 and 6.6.1-9182. Jenkins runs: 6.6.1-9182 - http://qa.sc.couchbase.com/job/oel6-4node-rebalance_in_jython/1359/console 7.0.0-3701 - http://qa.sc.couchbase.com/job/oel6-4node-rebalance_in_jython/1360/console

            Thanks Ashwin Govindarajulu. Closing as a dupe of MB-41089.

            ben.huddleston Ben Huddleston added a comment - Thanks Ashwin Govindarajulu . Closing as a dupe of MB-41089 .

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty