Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41089

Ephemeral HTCleaner may remove in-flight SyncWrite from the HashTable

    XMLWordPrintable

    Details

      Description

      Node 172.23.107.63 - Core dump seen: 1
      172.23.107.63 : Stack Trace of first crash: c1d8239e-ad57-4f6d-74ec5b87-4acca00c.dmp
      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
       #0  0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
       #0  0x00007efc2c8a8207 in raise () from /lib64/libc.so.6
       #1  0x00007efc2c8a98f8 in abort () from /lib64/libc.so.6
       #2  0x00007efc2d405195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6
       #3  0x000000000054ebb2 in backtrace_terminate_handler() ()
       #4  0x00007efc2d402f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
       #5  0x00007efc2d402fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
       #6  0x00007efc2d403213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
       #7  0x00007efc30c0cc60 in void ActiveDurabilityMonitor::throwException<std::logic_error>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /opt/couchbase/bin/../lib/libep.so
       #8  0x00007efc30c07546 in ActiveDurabilityMonitor::commit(DurabilityMonitor::ActiveSyncWrite const&) () from /opt/couchbase/bin/../lib/libep.so
       #9  0x00007efc30c07683 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue() () from /opt/couchbase/bin/../lib/libep.so
       #10 0x00007efc30cd88c0 in VBucket::processResolvedSyncWrites() () from /opt/couchbase/bin/../lib/libep.so
       #11 0x00007efc30c0e68d in DurabilityCompletionTask::run() () from /opt/couchbase/bin/../lib/libep.so
       #12 0x00007efc30c73583 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
       #13 0x00007efc30b867df in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
       #14 0x00007efc2f609777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
       #15 0x00007efc2cc46dd5 in start_thread () from /lib64/libpthread.so.0
       #16 0x00007efc2c96fead in clone () from /lib64/libc.so.6
      

      Memcached logs on .63:

      [root@sa1601-cnt76 ~]# grep CRITICAL /opt/couchbase/var/lib/couchbase/logs/memcached.log.00000*
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.039657-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.039712-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): ActiveDurabilityMonitor::commit vb:672 failed with status:1
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191278-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-2908). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/c1d8239e-ad57-4f6d-74ec5b87-4acca00c.dmp before terminating.
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191292-07:00 CRITICAL Stack backtrace of crashed thread:
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191470-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x13e11d]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191482-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x1541ea]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191491-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x154528]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191500-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7efc2cc3f000+0xf5d0]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191523-07:00 CRITICAL     /lib64/libc.so.6(gsignal+0x37) [0x7efc2c872000+0x36207]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191544-07:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7efc2c872000+0x378f8]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191575-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7efc2d374000+0x91195]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191585-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x14ebb2]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191601-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7efc2d374000+0x8ef86]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191616-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7efc2d374000+0x8efd1]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191632-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7efc2d374000+0x8f213]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191643-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xf0c60]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191652-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xeb546]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191658-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xeb683]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191668-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0x1bc8c0]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191675-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0xf268d]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191683-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0x157583]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191690-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7efc30b1c000+0x6a7df]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191696-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7efc2f5f9000+0x10777]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191704-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7efc2cc3f000+0x7dd5]
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt:2020-08-22T18:42:52.191736-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7efc2c872000+0xfdead]
      

      QE Test/Notes:

      num_items=5000,doc_size=512,transaction_timeout=30,nodes_init=2,replicas=1,bucket_size=100,GROUP=P0_Set1,durability=MAJORITY,get-cbcollect-info=False,infra_log_level=critical,log_level=error,upgrade_version=7.0.0-2908 -t Atomicity.basic_ops.basic_ops.test_basic_commit,nodes_init=2,replicas=1,transaction_commit=True,op_type=create;update,bucket_type=ephemeral,num_buckets=2,default_bucket=False,GROUP=P0_Set1
      

      Jenkin Job: centos-p0-transaction-vset00-00-multi_bucket_durability_majority_6.5_P0_Set1
      http://qa.sc.couchbase.com/job/test_suite_executor-TAF/52139/consoleFull

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            ashwin.govindarajulu Ashwin Govindarajulu added a comment -

            Validated the fix using Enterprise Edition 6.6.1 build 9182.

            Closing this ticket.

            Show
            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Validated the fix using Enterprise Edition 6.6.1 build 9182. Closing this ticket.
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-3701 contains kv_engine commit 80cc532 with commit message:
            MB-41089: HTTombstonePurger skips Pending stored-values

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3701 contains kv_engine commit 80cc532 with commit message: MB-41089 : HTTombstonePurger skips Pending stored-values
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-3860 contains kv_engine commit 67cd140 with commit message:
            MB-41089: HTTombstonePurger skips Pending stored-values

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3860 contains kv_engine commit 67cd140 with commit message: MB-41089 : HTTombstonePurger skips Pending stored-values
            Hide
            drigby Dave Rigby added a comment -

            Paolo Cocchi Please can you update the title of this MB to describe the actual bug. Additionally, I think this should be included in the 6.6.1 release notes, so can you also add an appropriate comment here - see https://hub.internal.couchbase.com/confluence/display/PM/Release+Notes+-+How+to for details.

            Show
            drigby Dave Rigby added a comment - Paolo Cocchi Please can you update the title of this MB to describe the actual bug. Additionally, I think this should be included in the 6.6.1 release notes, so can you also add an appropriate comment here - see https://hub.internal.couchbase.com/confluence/display/PM/Release+Notes+-+How+to for details.
            Hide
            paolo.cocchi Paolo Cocchi added a comment - - edited

            Description for release notes
            Summary: The HTCleaner in Ephemeral is responsible for purging tombstones and also Completed (Committed / Aborted) SyncWrites. A bug in that component may lead to removing in-flight SyncWrites from internal data-structures, which would cause a crash on the node when/if it tries to complete the SyncWrite.

            Show
            paolo.cocchi Paolo Cocchi added a comment - - edited Description for release notes Summary: The HTCleaner in Ephemeral is responsible for purging tombstones and also Completed (Committed / Aborted) SyncWrites. A bug in that component may lead to removing in-flight SyncWrites from internal data-structures, which would cause a crash on the node when/if it tries to complete the SyncWrite.

              People

              Assignee:
              ashwin.govindarajulu Ashwin Govindarajulu
              Reporter:
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty