Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41253

[Magma] Underflow error in EPVBucket::decrNumTotalItems during EPBucket::compactionCompletionCallback

    XMLWordPrintable

Details

    Description

      Node 172.23.120.206 - Core dump seen: 5
      172.23.120.206 : Stack Trace of first crash: c8d36619-5b8b-4b22-a359ea9d-a08ab09f.dmp
      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
       #0  0x00007efcc1f10387 in raise () from /lib64/libc.so.6
       #0  0x00007efcc1f10387 in raise () from /lib64/libc.so.6
       #1  0x00007efcc1f11a78 in abort () from /lib64/libc.so.6
       #2  0x00007efcc2a6e195 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95
       #3  0x000000000054ebb2 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:86
       #4  0x00007efcc2a6bf86 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
       #5  0x00007efcc2a6bfd1 in std::terminate () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57
       #6  0x00007efcc2a6c213 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7efc68041200, tinfo=0x7efcc2d51ad8 <typeinfo for std::underflow_error>, dest=0x7efcc2a814f0 <std::underflow_error::~underflow_error()>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_throw.cc:93
       #7  0x00007efcc61fcfb4 in cb::ThrowExceptionUnderflowPolicy<unsigned long>::underflow (this=<optimized out>, desired=@0x7efc80d66b58: 18446744073709551612, current=<optimized out>, arg=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:48
       #8  0x00007efcc629d1f1 in fetch_sub (arg=<optimized out>, this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:160
       #9  operator-= (rhs=<optimized out>, this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:192
       #10 EPVBucket::decrNumTotalItems (this=<optimized out>, numItemsRemoved=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_vb.cc:388
       #11 0x00007efcc628fdca in EPBucket::compactionCompletionCallback (this=<optimized out>, ctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:1138
       #12 0x00007efcc63bf238 in operator() (__args#0=..., this=<optimized out>) at /usr/local/include/c++/7.3.0/bits/std_function.h:706
       #13 MagmaKVStore::compactDBInternal (this=this@entry=0x7efca00f3a00, ctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:1910
       #14 0x00007efcc63c00f2 in MagmaKVStore::compactDB (this=0x7efca00f3a00, ctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:1761
       #15 0x00007efcc6291e9f in EPBucket::compactInternal (this=0x7efca00f8000, config=..., purgeSeqno=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:1146
       #16 0x00007efcc62921a4 in EPBucket::doCompact (this=0x7efca00f8000, config=..., purgeSeqno=0, cookie=0x0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:1196
       #17 0x00007efcc633fb08 in CompactTask::run (this=0x7efc2d7422d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/tasks.cc:67
       #18 0x00007efcc62eaed3 in GlobalTask::execute (this=0x7efc2d7422d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:73
       #19 0x00007efcc61f6a4f in CB3ExecutorThread::run (this=0x7efcc09057a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/cb3_executorthread.cc:188
       #20 0x00007efcc4c72777 in run (this=0x7efcc19e45a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58
       #21 platform_thread_wrap (arg=0x7efcc19e45a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71
       #22 0x00007efcc22afea5 in start_thread () from /lib64/libpthread.so.0
       #23 0x00007efcc1fd88dd in clone () from /lib64/libc.so.6
      

      172.23.120.206 : Looking for CRITICAL messages in log
      172.23.120.206 : Found message in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000078.txt
      2020-08-31T15:10:29.531538-07:00 CRITICAL (default) magma_2 MagmaKVStore::rollback Rollback vb:22 status:NotExists: KVStore ID:22 does not exist
      2020-08-31T15:10:29.531554-07:00 CRITICAL (default) magma_2 MagmaKVStore::prepareToDeleteImpl vb:22 GetKVStoreRevision failed. Status:NotExists: KVStore ID:22 does not exist
      2020-08-31T15:10:40.381889-07:00 CRITICAL (default) magma_2 MagmaKVStore::prepareToDeleteImpl vb:18 GetKVStoreRevision failed. Status:NotExists: KVStore ID:18 does not exist
      2020-08-31T15:11:34.407697-07:00 CRITICAL (default) magma_2 MagmaKVStore::prepareToDeleteImpl vb:38 GetKVStoreRevision failed. Status:NotExists: KVStore ID:38 does not exist
      2020-08-31T15:11:44.710663-07:00 CRITICAL (default) magma_2 MagmaKVStore::prepareToDeleteImpl vb:34 GetKVStoreRevision failed. Status:NotExists: KVStore ID:34 does not exist
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty