Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41089

Ephemeral HTCleaner may remove in-flight SyncWrite from the HashTable

    XMLWordPrintable

Details

    Description

      Node 172.23.107.63 - Core dump seen: 1
      172.23.107.63 : Stack Trace of first crash: c1d8239e-ad57-4f6d-74ec5b87-4acca00c.dmp
      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
       #0  0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
       #0  0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
       #1  0x00007efc2c8a98f8 in abort () from /lib64/libc.so.6
       #2  0x00007efc2d405195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6
       #3  0x000000000054ebb2 in backtrace_terminate_handler() ()
       #4  0x00007efc2d402f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
       #5  0x00007efc2d402fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
       #6  0x00007efc2d403213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
       #7  0x00007efc30c0cc60 in void ActiveDurabilityMonitor::throwException<std::logic_error>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /opt/couchbase/bin/../lib/libep.so
       #8  0x00007efc30c07546 in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so
       #9  0x00007efc30c07683 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () from /opt/couchbase/bin/../lib/libep.so
       #10 0x00007efc30cd88c0 in VBucket::processResolvedSyncWrites() () from /opt/couchbase/bin/../lib/libep.so
       #11 0x00007efc30c0e68d in DurabilityCompletionTask::run() () from /opt/couchbase/bin/../lib/libep.so
       #12 0x00007efc30c73583 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
       #13 0x00007efc30b867df in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
       #14 0x00007efc2f609777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
       #15 0x00007efc2cc46dd5 in start_thread () from /lib64/libpthread.so.0
       #16 0x00007efc2c96fead in clone () from /lib64/libc.so.6
      

      Memcached logs on .63:

      [root@sa1601-cnt76 ~]# grep CRITICAL /opt/couchbase/var/lib/couchbase/logs/memcached.log.00000*
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.039657-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.039712-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): ActiveDurabilityMonitor::commit vb:672 failed with status:1
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191278-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-2908). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/c1d8239e-ad57-4f6d-74ec5b87-4acca00c.dmp before terminating.
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191292-07:00 CRITICAL Stack backtrace of crashed thread:
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191470-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x13e11d]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191482-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x1541ea]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191491-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x154528]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191500-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7efc2cc3f000+0xf5d0]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191523-07:00 CRITICAL     /lib64/libc.so.6(gsignal+0x37) [0x7efc2c872000+0x36207]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191544-07:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7efc2c872000+0x378f8]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191575-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7efc2d374000+0x91195]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191585-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x14ebb2]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191601-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7efc2d374000+0x8ef86]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191616-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7efc2d374000+0x8efd1]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191632-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7efc2d374000+0x8f213]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191643-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xf0c60]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191652-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xeb546]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191658-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xeb683]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191668-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0x1bc8c0]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191675-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xf268d]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191683-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0x157583]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191690-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0x6a7df]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191696-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7efc2f5f9000+0x10777]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191704-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7efc2cc3f000+0x7dd5]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191736-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7efc2c872000+0xfdead]
      

      QE Test/Notes:

      num_items=5000,doc_size=512,transaction_timeout=30,nodes_init=2,replicas=1,bucket_size=100,GROUP=P0_Set1,durability=MAJORITY,get-cbcollect-info=False,infra_log_level=critical,log_level=error,upgrade_version=7.0.0-2908 -t Atomicity.basic_ops.basic_ops.test_basic_commit,nodes_init=2,replicas=1,transaction_commit=True,op_type=create;update,bucket_type=ephemeral,num_buckets=2,default_bucket=False,GROUP=P0_Set1
      

      Jenkin Job: centos-p0-transaction-vset00-00-multi_bucket_durability_majority_6.5_P0_Set1
      http://qa.sc.couchbase.com/job/test_suite_executor-TAF/52139/consoleFull

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            ritesh.agarwal Ritesh Agarwal created issue -
            ritesh.agarwal Ritesh Agarwal made changes -
            Field Original Value New Value
            Link to Log File, atop/blg, CBCollectInfo, Core dump https://uploads.couchbase.com/MB-41089/collectinfo-2020-08-23T032311-ns_1%40172.23.107.59.zip
            https://uploads.couchbase.com/MB-41089/collectinfo-2020-08-23T032311-ns_1%40172.23.107.63.zip

            Description {code}
            Node 172.23.107.63 - Core dump seen: 1
            172.23.107.63 : Stack Trace of first crash: c1d8239e-ad57-4f6d-74ec5b87-4acca00c.dmp
            Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
             #0 0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
             #0 0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
             #1 0x00007efc2c8a98f8 in abort () from /lib64/libc.so.6
             #2 0x00007efc2d405195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #3 0x000000000054ebb2 in backtrace_terminate_handler() ()
             #4 0x00007efc2d402f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #5 0x00007efc2d402fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #6 0x00007efc2d403213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #7 0x00007efc30c0cc60 in void ActiveDurabilityMonitor::throwException<std::logic_error>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /opt/couchbase/bin/../lib/libep.so
             #8 0x00007efc30c07546 in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so
             #9 0x00007efc30c07683 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () from /opt/couchbase/bin/../lib/libep.so
             #10 0x00007efc30cd88c0 in VBucket::processResolvedSyncWrites() () from /opt/couchbase/bin/../lib/libep.so
             #11 0x00007efc30c0e68d in DurabilityCompletionTask::run() () from /opt/couchbase/bin/../lib/libep.so
             #12 0x00007efc30c73583 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
             #13 0x00007efc30b867df in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
             #14 0x00007efc2f609777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
             #15 0x00007efc2cc46dd5 in start_thread () from /lib64/libpthread.so.0
             #16 0x00007efc2c96fead in clone () from /lib64/libc.so.6
            {code}

            QE Test:
            {code}
            num_items=5000,doc_size=512,transaction_timeout=30,nodes_init=2,replicas=1,bucket_size=100,GROUP=P0_Set1,durability=MAJORITY,get-cbcollect-info=False,infra_log_level=critical,log_level=error,upgrade_version=7.0.0-2908 -t Atomicity.basic_ops.basic_ops.test_basic_commit,nodes_init=2,replicas=1,transaction_commit=True,op_type=create;update,bucket_type=ephemeral,num_buckets=2,default_bucket=False,GROUP=P0_Set1
            {code}
            {code}
            Node 172.23.107.63 - Core dump seen: 1
            172.23.107.63 : Stack Trace of first crash: c1d8239e-ad57-4f6d-74ec5b87-4acca00c.dmp
            Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
             #0 0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
             #0 0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
             #1 0x00007efc2c8a98f8 in abort () from /lib64/libc.so.6
             #2 0x00007efc2d405195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #3 0x000000000054ebb2 in backtrace_terminate_handler() ()
             #4 0x00007efc2d402f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #5 0x00007efc2d402fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #6 0x00007efc2d403213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #7 0x00007efc30c0cc60 in void ActiveDurabilityMonitor::throwException<std::logic_error>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /opt/couchbase/bin/../lib/libep.so
             #8 0x00007efc30c07546 in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so
             #9 0x00007efc30c07683 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () from /opt/couchbase/bin/../lib/libep.so
             #10 0x00007efc30cd88c0 in VBucket::processResolvedSyncWrites() () from /opt/couchbase/bin/../lib/libep.so
             #11 0x00007efc30c0e68d in DurabilityCompletionTask::run() () from /opt/couchbase/bin/../lib/libep.so
             #12 0x00007efc30c73583 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
             #13 0x00007efc30b867df in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
             #14 0x00007efc2f609777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
             #15 0x00007efc2cc46dd5 in start_thread () from /lib64/libpthread.so.0
             #16 0x00007efc2c96fead in clone () from /lib64/libc.so.6
            {code}

            QE Test/Notes:
            {code}
            num_items=5000,doc_size=512,transaction_timeout=30,nodes_init=2,replicas=1,bucket_size=100,GROUP=P0_Set1,durability=MAJORITY,get-cbcollect-info=False,infra_log_level=critical,log_level=error,upgrade_version=7.0.0-2908 -t Atomicity.basic_ops.basic_ops.test_basic_commit,nodes_init=2,replicas=1,transaction_commit=True,op_type=create;update,bucket_type=ephemeral,num_buckets=2,default_bucket=False,GROUP=P0_Set1
            {code}
            Jenkin Job: _centos-p0-transaction-vset00-00-multi_bucket_durability_majority_6.5_P0_Set1_
            http://qa.sc.couchbase.com/job/test_suite_executor-TAF/52139/consoleFull
            ritesh.agarwal Ritesh Agarwal made changes -
            Summary Crash observed in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so Memcached crash observed in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so
            ritesh.agarwal Ritesh Agarwal made changes -
            Description {code}
            Node 172.23.107.63 - Core dump seen: 1
            172.23.107.63 : Stack Trace of first crash: c1d8239e-ad57-4f6d-74ec5b87-4acca00c.dmp
            Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
             #0 0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
             #0 0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
             #1 0x00007efc2c8a98f8 in abort () from /lib64/libc.so.6
             #2 0x00007efc2d405195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #3 0x000000000054ebb2 in backtrace_terminate_handler() ()
             #4 0x00007efc2d402f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #5 0x00007efc2d402fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #6 0x00007efc2d403213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #7 0x00007efc30c0cc60 in void ActiveDurabilityMonitor::throwException<std::logic_error>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /opt/couchbase/bin/../lib/libep.so
             #8 0x00007efc30c07546 in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so
             #9 0x00007efc30c07683 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () from /opt/couchbase/bin/../lib/libep.so
             #10 0x00007efc30cd88c0 in VBucket::processResolvedSyncWrites() () from /opt/couchbase/bin/../lib/libep.so
             #11 0x00007efc30c0e68d in DurabilityCompletionTask::run() () from /opt/couchbase/bin/../lib/libep.so
             #12 0x00007efc30c73583 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
             #13 0x00007efc30b867df in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
             #14 0x00007efc2f609777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
             #15 0x00007efc2cc46dd5 in start_thread () from /lib64/libpthread.so.0
             #16 0x00007efc2c96fead in clone () from /lib64/libc.so.6
            {code}

            QE Test/Notes:
            {code}
            num_items=5000,doc_size=512,transaction_timeout=30,nodes_init=2,replicas=1,bucket_size=100,GROUP=P0_Set1,durability=MAJORITY,get-cbcollect-info=False,infra_log_level=critical,log_level=error,upgrade_version=7.0.0-2908 -t Atomicity.basic_ops.basic_ops.test_basic_commit,nodes_init=2,replicas=1,transaction_commit=True,op_type=create;update,bucket_type=ephemeral,num_buckets=2,default_bucket=False,GROUP=P0_Set1
            {code}
            Jenkin Job: _centos-p0-transaction-vset00-00-multi_bucket_durability_majority_6.5_P0_Set1_
            http://qa.sc.couchbase.com/job/test_suite_executor-TAF/52139/consoleFull
            {code}
            Node 172.23.107.63 - Core dump seen: 1
            172.23.107.63 : Stack Trace of first crash: c1d8239e-ad57-4f6d-74ec5b87-4acca00c.dmp
            Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
             #0 0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
             #0 0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
             #1 0x00007efc2c8a98f8 in abort () from /lib64/libc.so.6
             #2 0x00007efc2d405195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #3 0x000000000054ebb2 in backtrace_terminate_handler() ()
             #4 0x00007efc2d402f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #5 0x00007efc2d402fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #6 0x00007efc2d403213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
             #7 0x00007efc30c0cc60 in void ActiveDurabilityMonitor::throwException<std::logic_error>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /opt/couchbase/bin/../lib/libep.so
             #8 0x00007efc30c07546 in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so
             #9 0x00007efc30c07683 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () from /opt/couchbase/bin/../lib/libep.so
             #10 0x00007efc30cd88c0 in VBucket::processResolvedSyncWrites() () from /opt/couchbase/bin/../lib/libep.so
             #11 0x00007efc30c0e68d in DurabilityCompletionTask::run() () from /opt/couchbase/bin/../lib/libep.so
             #12 0x00007efc30c73583 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
             #13 0x00007efc30b867df in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
             #14 0x00007efc2f609777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
             #15 0x00007efc2cc46dd5 in start_thread () from /lib64/libpthread.so.0
             #16 0x00007efc2c96fead in clone () from /lib64/libc.so.6
            {code}

            Memcached logs on .63:
            {code}
            [root@sa1601-cnt76 ~]# grep CRITICAL /opt/couchbase/var/lib/couchbase/logs/memcached.log.00000*
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.039657-07:00 CRITICAL *** Fatal error encountered during exception handling ***
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.039712-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): ActiveDurabilityMonitor::commit vb:672 failed with status:1
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191278-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-2908). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/c1d8239e-ad57-4f6d-74ec5b87-4acca00c.dmp before terminating.
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191292-07:00 CRITICAL Stack backtrace of crashed thread:
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191470-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x13e11d]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191482-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x1541ea]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191491-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x154528]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191500-07:00 CRITICAL /lib64/libpthread.so.0() [0x7efc2cc3f000+0xf5d0]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191523-07:00 CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7efc2c872000+0x36207]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191544-07:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7efc2c872000+0x378f8]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191575-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7efc2d374000+0x91195]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191585-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x14ebb2]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191601-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7efc2d374000+0x8ef86]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191616-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7efc2d374000+0x8efd1]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191632-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7efc2d374000+0x8f213]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191643-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xf0c60]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191652-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xeb546]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191658-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xeb683]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191668-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0x1bc8c0]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191675-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xf268d]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191683-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0x157583]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191690-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0x6a7df]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191696-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7efc2f5f9000+0x10777]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191704-07:00 CRITICAL /lib64/libpthread.so.0() [0x7efc2cc3f000+0x7dd5]
            /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191736-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7efc2c872000+0xfdead]
            {code}

            QE Test/Notes:
            {code}
            num_items=5000,doc_size=512,transaction_timeout=30,nodes_init=2,replicas=1,bucket_size=100,GROUP=P0_Set1,durability=MAJORITY,get-cbcollect-info=False,infra_log_level=critical,log_level=error,upgrade_version=7.0.0-2908 -t Atomicity.basic_ops.basic_ops.test_basic_commit,nodes_init=2,replicas=1,transaction_commit=True,op_type=create;update,bucket_type=ephemeral,num_buckets=2,default_bucket=False,GROUP=P0_Set1
            {code}
            Jenkin Job: _centos-p0-transaction-vset00-00-multi_bucket_durability_majority_6.5_P0_Set1_
            http://qa.sc.couchbase.com/job/test_suite_executor-TAF/52139/consoleFull
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Ritesh Agarwal [ ritesh.agarwal ]
            ritesh.agarwal Ritesh Agarwal made changes -
            Assignee Ritesh Agarwal [ ritesh.agarwal ] Daniel Owen [ owend ]
            ritesh.agarwal Ritesh Agarwal made changes -
            Is this a Regression? Unknown [ 10452 ] Yes [ 10450 ]
            ritesh.agarwal Ritesh Agarwal made changes -
            Is this a Regression? Yes [ 10450 ] Unknown [ 10452 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            owend Daniel Owen made changes -
            Rank Ranked higher
            ritesh.agarwal Ritesh Agarwal made changes -
            Labels Durability functional-test 7.0mustpass Durability functional-test
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Paolo Cocchi [ paolo.cocchi ]
            paolo.cocchi Paolo Cocchi made changes -
            Affects Version/s 6.6.0 [ 16787 ]
            Affects Version/s 6.5.1 [ 16622 ]
            Affects Version/s 6.5.0 [ 15037 ]
            paolo.cocchi Paolo Cocchi made changes -
            Affects Version/s Cheshire-Cat [ 15915 ]
            paolo.cocchi Paolo Cocchi made changes -
            Affects Version/s Cheshire-Cat [ 15915 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 6.6.1 [ 17002 ]
            lynn.straus Lynn Straus made changes -
            Labels 7.0mustpass Durability functional-test 7.0mustpass Durability approved-for-6.6.1 functional-test
            owend Daniel Owen made changes -
            Link This issue blocks MB-40528 [ MB-40528 ]
            paolo.cocchi Paolo Cocchi made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            VERIFICATION STEPS Atomicity.basic_ops.basic_ops:
              test_basic_commit,nodes_init=2,replicas=1,transaction_commit=True,op_type=create;update,num_items=50000,doc_size=512,bucket_type=ephemeral
            Assignee Paolo Cocchi [ paolo.cocchi ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
            Status Resolved [ 5 ] Closed [ 6 ]
            ben.huddleston Ben Huddleston made changes -
            Link This issue is duplicated by MB-41649 [ MB-41649 ]
            paolo.cocchi Paolo Cocchi made changes -
            Labels 7.0mustpass Durability approved-for-6.6.1 functional-test 7.0mustpass Durability approved-for-6.6.1 ephemeral-bucket functional-test
            paolo.cocchi Paolo Cocchi made changes -
            Summary Memcached crash observed in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so Memcached crash in ActiveDurabilityMonitor::commit - no HashTable item found with key
            drigby Dave Rigby made changes -
            Link This issue relates to MB-43120 [ MB-43120 ]
            paolo.cocchi Paolo Cocchi made changes -
            Summary Memcached crash in ActiveDurabilityMonitor::commit - no HashTable item found with key Ephemeral HTCleaner may remove in-flight SyncWrite from the HashTable
            drigby Dave Rigby made changes -
            Labels 7.0mustpass Durability approved-for-6.6.1 ephemeral-bucket functional-test 7.0mustpass Durability approved-for-6.6.1 ephemeral-bucket functional-test releasenote
            owend Daniel Owen made changes -
            Link This issue causes CBSE-9551 [ CBSE-9551 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.0 [ 17233 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s Cheshire-Cat [ 15915 ]

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty