Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40286

VBucket::commit (vb:924) failed as no HashTable item found with key

    XMLWordPrintable

Details

    Description

      Script to Repro

      ./testrunner -i /tmp/win10-bucket-ops.ini rerun=False,crash_warning=True,quota_percent=95 -t 
      failover.AutoFailoverTests.AutoFailoverTests. test_autofailover_during_rebalance,timeout=5,num_node_failures=1,nodes_in=0,nodes_out=1,auto_reprovision=True,failover_action=restart_server,nodes_init=4,can_abort_rebalance=False,override_spec_params=durability,durability=MAJORITY,bucket_spec=multi_bucket.buckets_all_ephemeral,GROUP=auto_reprovision_durability
      

      ./testrunner -i /tmp/win10-bucket-ops.ini rerun=False,crash_warning=True,quota_percent=95 -t failover.AutoFailoverTests.AutoFailoverTests.test_autofailover_during_rebalance,timeout=5,num_node_failures=1,nodes_in=0,nodes_out=1,failover_action=restart_server,nodes_init=4,can_abort_rebalance=False,override_spec_params=durability,durability=MAJORITY,bucket_spec=single_bucket.def_scope_fifty_collections,GROUP=auto_failover_durability
      

      Memcached Log on 172.23.104.186:

      memcached.log.000207.txt:2020-07-04T16:54:30.772315-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      memcached.log.000207.txt:2020-07-04T16:54:30.772390-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): ActiveDurabilityMonitor::commit vb:924 failed with status:1
      memcached.log.000207.txt:2020-07-04T16:54:30.975240-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-2524). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/ada15f48-bf3d-4625-91f97780-782054e8.dmp before terminating.
      memcached.log.000207.txt:2020-07-04T16:54:30.975269-07:00 CRITICAL Stack backtrace of crashed thread:
      memcached.log.000207.txt:2020-07-04T16:54:30.975521-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x13cd8d]
      memcached.log.000207.txt:2020-07-04T16:54:30.975544-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x15206a]
      memcached.log.000207.txt:2020-07-04T16:54:30.975560-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x1523a8]
      memcached.log.000207.txt:2020-07-04T16:54:30.975574-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f90ab565000+0xf630]
      memcached.log.000207.txt:2020-07-04T16:54:30.975619-07:00 CRITICAL     /lib64/libc.so.6(gsignal+0x37) [0x7f90ab197000+0x36387]
      memcached.log.000207.txt:2020-07-04T16:54:30.975658-07:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7f90ab197000+0x37a78]
      memcached.log.000207.txt:2020-07-04T16:54:30.975704-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f90abc9a000+0x91195]
      memcached.log.000207.txt:2020-07-04T16:54:30.975721-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x14d562]
      memcached.log.000207.txt:2020-07-04T16:54:30.975757-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f90abc9a000+0x8ef86]
      memcached.log.000207.txt:2020-07-04T16:54:30.975794-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f90abc9a000+0x8efd1]
      memcached.log.000207.txt:2020-07-04T16:54:30.975825-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f90abc9a000+0x8f213]
      memcached.log.000207.txt:2020-07-04T16:54:30.975841-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f90af3fd000+0xdd260]
      memcached.log.000207.txt:2020-07-04T16:54:30.975854-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f90af3fd000+0xd79b2]
      memcached.log.000207.txt:2020-07-04T16:54:30.975863-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f90af3fd000+0xd7b13]
      memcached.log.000207.txt:2020-07-04T16:54:30.975877-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f90af3fd000+0x1a3a60]
      memcached.log.000207.txt:2020-07-04T16:54:30.975887-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f90af3fd000+0xdeefd]
      memcached.log.000207.txt:2020-07-04T16:54:30.975899-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f90af3fd000+0x148d83]
      memcached.log.000207.txt:2020-07-04T16:54:30.975909-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f90af3fd000+0x13f5af]
      memcached.log.000207.txt:2020-07-04T16:54:30.975921-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f90adf1f000+0x10777]
      memcached.log.000207.txt:2020-07-04T16:54:30.975933-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f90ab565000+0x7ea5]
      memcached.log.000207.txt:2020-07-04T16:54:30.976031-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7f90ab197000+0xfe8dd]
      

      bt full

      (gdb) bt full
      #0  0x00007f22205f2387 in raise () from /lib64/libc.so.6
      No symbol table info available.
      #1  0x00007f22205f3a78 in abort () from /lib64/libc.so.6
      No symbol table info available.
      #2  0x00007f2221150195 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95
              terminating = false
              t = <optimized out>
      #3  0x000000000054d562 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:86
      No locals.
      #4  0x00007f222114df86 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
      No locals.
      #5  0x00007f222114dfd1 in std::terminate () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57
      No locals.
      #6  0x00007f222114e213 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f21740dc910, tinfo=0x7ff910 <typeinfo for std::logic_error>, dest=0x418e50 <_ZNSt11logic_errorD1Ev@plt>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_throw.cc:93
              globals = <optimized out>
              header = 0x7f21740dc890
      #7  0x00007f2224879fd6 in ThrowExceptionPolicy<long>::nonMonotonic (this=<optimized out>, curValue=<optimized out>, newValue=<optimized out>)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/monotonic.h:53
      No locals.
      #8  0x00007f22248f9943 in operator= (other=..., this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/monotonic.h:136
      No locals.
      #9  updateHighCompletedSeqno (this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/durability/active_durability_monitor.cc:1518
      No locals.
      #10 ActiveDurabilityMonitor::commit (this=this@entry=0x7f21cb44f460, sw=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/durability/active_durability_monitor.cc:863
              s = {
                holder_ = {<folly::LockedPtrBase<folly::Synchronized<std::unique_ptr<ActiveDurabilityMonitor::State, std::default_delete<ActiveDurabilityMonitor::State> >, folly::SharedMutexImpl<false, void, std::atomic, false, false> >, folly::SharedMutexImpl<false, void, std::atomic, false, false>, folly::LockPolicyExclusive>> = {parent_ = 0x7f21cb44f498}, static AllowsConcurrentAccess = false}}
              key = <optimized out>
              cHandle = {<Collections::VB::Manifest::ReadHandle> = {readLock = {lock_ = 0x7f21cb304c64, token_ = {type_ = folly::SharedMutexToken::DEFERRED_SHARED, slot_ = 16}}, manifest = 0x7f21cb304be0}, 
                itr = {<folly::f14::detail::BaseIter<std::pair<CollectionID const, Collections::VB::ManifestEntry> const*, unsigned int>> = {<std::iterator<std::forward_iterator_tag, std::pair<CollectionID const, Collections::VB::ManifestEntry>, long, std::pair<CollectionID const, Collections::VB::ManifestEntry> const*, std::pair<CollectionID const, Collections::VB::ManifestEntry> const&>> = {<No data fields>}, <No data fields>}, current_ = 0x7f21c1602da0, 
                  lowest_ = 0x7f21c1602200}, key = {<DocKeyInterface<DocKey>> = {<No data fields>}, buffer = {static npos = <optimized out>, buf = 0x7f21d9601320 <Address 0x7f21d9601320 out of bounds>, len = 22}, encoding = Yes}}
              prepareEnd = {__d = {__r = 144042419775393}}
              cookie = <optimized out>
              result = <optimized out>
              __func__ = "commit"
              index = <optimized out>
              commitDuration = <optimized out>
      #11 0x00007f22248f9b13 in ActiveDurabilityMonitor::processCompletedSyncWriteQueue (this=0x7f21cb44f460) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/durability/active_durability_monitor.cc:448
              sw = {storage_ = {{emptyState = -48 '\320', value = {<DurabilityMonitor::SyncWrite> = {item = {value = 0x7f21be84a6d0}, startTime = {__d = {__r = 140602082705333}}, status = ToCommit}, cookie = 0x0, firstChain = {
                        ackCount = {<ThrowExceptionPolicy<unsigned char>> = {label = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, 
                                _M_p = 0x7f21d9fd8ed0 "unlabelled"}, _M_string_length = 10, {_M_local_buf = "unlabelled\000\313!\177\000", _M_allocated_capacity = 7812730926796664437}}}, val = 2 '\002'}, chainPtr = 0x7f21b18e9440}, 
                      secondChain = {ackCount = {<ThrowExceptionPolicy<unsigned char>> = {label = {static npos = 18446744073709551615, 
                              _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7f21d9fd8f00 "unlabelled"}, _M_string_length = 10, {
                                _M_local_buf = "unlabelled\000!\"\177\000", _M_allocated_capacity = 7812730926796664437}}}, val = 0 '\000'}, chainPtr = 0x0}, 
                      expiryTime = {<std::_Optional_base<std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1, 1000000000> > > >> = {_M_payload = {{_M_empty = {<No data fields>}, _M_payload = {
                                __d = {__r = 139783120942176}}}, 
                            _M_engaged = false}}, <std::_Enable_copy_move<true, true, true, true, std::optional<std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1, 1000000000> > > > >> = {<No data fields>}, <No data fields>}}}, hasValue = true}}
              lock = {_M_device = @0x7f21fc003028}
      #12 0x00007f22249c5a60 in VBucket::processResolvedSyncWrites (this=0x7f21cb441200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:700
              rlh = {lock_ = 0x7f21cb4413d8, token_ = {type_ = folly::SharedMutexToken::INLINE_SHARED, slot_ = 8805}}
      #13 0x00007f2224900efd in DurabilityCompletionTask::run (this=0x7f2215736510) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/durability/durability_completion_task.cc:53
              vb = {<std::__shared_ptr<VBucket, (__gnu_cxx::_Lock_policy)2>> = {<std::__shared_ptr_access<VBucket, (__gnu_cxx::_Lock_policy)2, false, false>> = {<No data fields>}, _M_ptr = 0x7f21cb441200, _M_refcount = {
                    _M_pi = 0x7f21cb458ea0}}, <No data fields>}
              runtime = <optimized out>
              startTime = {__d = {__r = 144042419764058}}
              pendingVb = {vbid = 557}
      #14 0x00007f222496ad83 in GlobalTask::execute (this=0x7f2215736510) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:73
              guard = {previous = 0x0}
      #15 0x00007f22249615af in ExecutorThread::run (this=0x7f221f1f6880) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:188
              curTaskDescr = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, 
                  _M_p = 0x7f221ef0ba40 <Address 0x7f221ef0ba40 out of bounds>}, _M_string_length = 24, {_M_local_buf = "\030\000\000\000\000\000\000\000pressor", _M_allocated_capacity = 24}}
              woketime = <optimized out>
      ---Type <return> to continue, or q <return> to quit---
              scheduleOverhead = <optimized out>
              again = <optimized out>
              runtime = <optimized out>
              q = <optimized out>
              tick = 128 '\200'
              guard = {engine = 0x0}
      #16 0x00007f2223354777 in run (this=0x7f221fe4a0a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58
      No locals.
      #17 platform_thread_wrap (arg=0x7f221fe4a0a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71
              context = {_M_t = {
                  _M_t = {<std::_Tuple_impl<0, CouchbaseThread*, std::default_delete<CouchbaseThread> >> = {<std::_Tuple_impl<1, std::default_delete<CouchbaseThread> >> = {<std::_Head_base<1, std::default_delete<CouchbaseThread>, true>> = {<std::default_delete<CouchbaseThread>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, CouchbaseThread*, false>> = {_M_head_impl = 0x7f221fe4a0a0}, <No data fields>}, <No data fields>}}}
      #18 0x00007f2220991ea5 in start_thread () from /lib64/libpthread.so.0
      No symbol table info available.
      #19 0x00007f22206ba8dd in clone () from /lib64/libc.so.6
      No symbol table info available.
      (gdb) 
      

      Attachments

        1. pcaps.zip
          14.04 MB
        2. consoleText.txt.zip
          2.65 MB
        3. Automation_logs.zip
          11 kB

        Issue Links

          Activity

            People

              james.harrison James Harrison (Inactive)
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty