Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43816

ThrowExceptionUnderflowPolicy current:20524 arg:20525 during Rebalance In operation while bucket is in 1% DGM.

    XMLWordPrintable

Details

    Description

      Steps:

      1. Create a 3 node cluster and a magma bucket with 3GB total bucket quota.
      2. Load items so that bucket goes into 1% DGM.
      3. Rebalance IN one node.
      4. Rebalance failed due to memcached crash on node 31

      /opt/couchbase/var/lib/couchbase/logs/babysitter.log.4.gz:[ns_server:info,2021-01-22T00:49:13.992-08:00,babysitter_of_ns_1@cb.local:<0.128.0>:ns_port_server:log:224]memcached<0.128.0>: 2021-01-22T00:49:13.790005-08:00 CRITICAL *** Fatal error encountered during exception handling ***
      /opt/couchbase/var/lib/couchbase/logs/babysitter.log.4.gz:memcached<0.128.0>: 2021-01-22T00:49:13.790078-08:00 CRITICAL Caught unhandled std::exception-derived exception. what(): ThrowExceptionUnderflowPolicy current:20524 arg:20525
      /opt/couchbase/var/lib/couchbase/logs/babysitter.log.4.gz:[ns_server:info,2021-01-22T00:49:14.813-08:00,babysitter_of_ns_1@cb.local:<0.128.0>:ns_port_server:log:224]memcached<0.128.0>: 2021-01-22T00:49:14.609043-08:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-4281). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/d0d24bfc-4a33-4bd7-5efaef8d-b2ef85b5.dmp before terminating.
      

      Stack Trace of crash:

      172.23.107.31 : Stack Trace of d0d24bfc-4a33-4bd7-5efaef8d-b2ef85b5.dmp
      rm -rf /opt/couchbase/var/lib/couchbase/crash/0d24bfc-4a33-4bd7-5efaef8d-b2ef85b5.core
      //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/d0d24bfc-4a33-4bd7-5efaef8d-b2ef85b5.dmp > /opt/couchbase/var/lib/couchbase/crash/0d24bfc-4a33-4bd7-5efaef8d-b2ef85b5.core
      gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/0d24bfc-4a33-4bd7-5efaef8d-b2ef85b5.core -ex bt full -ex quit
      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
       #0  0x00007f2609170387 in raise () from /lib64/libc.so.6
       #0  0x00007f2609170387 in raise () from /lib64/libc.so.6
       #1  0x00007f2609171a78 in abort () from /lib64/libc.so.6
       #2  0x00007f2609cce195 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95
       #3  0x0000000000555862 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:86
       #4  0x00007f2609ccbf86 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
       #5  0x00007f2609ccbfd1 in std::terminate () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57
       #6  0x00007f260d6d0533 in GlobalTask::execute (this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:77
       #7  0x00007f260d6ca592 in operator() (__closure=0x7f25a97e10c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:167
       #8  folly::detail::function::FunctionTraits<void ()>::callSmall<FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387
       #9  0x00007f260d848626 in operator() (this=0x7f25a97e10c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416
       #10 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) (this=0x7f26056f1400, thread=..., task=<unknown type in /usr/lib/debug/opt/couchbase/lib/libep.so.debug, CU 0x30726e3, DIE 0x30b8ada>) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:98
       #11 0x00007f260d830b1a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f26056f1400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265
       #12 0x00007f260d84b749 in __invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__t=<optimized out>, __f=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:73
       #13 __invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__fn=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:95
       #14 __call<void, 0, 1> (__args=<optimized out>, this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:467
       #15 operator()<> (this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:551
       #16 folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401
       #17 0x00007f260d6c89d3 in operator() (this=0x7f2605604820) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416
       #18 operator() (__closure=0x7f2605604810) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54
       #19 folly::detail::function::FunctionTraits<void ()>::callBig<folly::PriorityThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401
       #20 0x00007f2609cf6dcf in std::execute_native_thread_routine (__p=0x7f260570e7a0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83
       #21 0x00007f260950fea5 in start_thread () from /lib64/libpthread.so.0
       #22 0x00007f26092388dd in clone () from /lib64/libc.so.6
      

      Test:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ritesh_temp_jython.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t volumetests.Magma.volume.test_long_rebalance,nodes_init=3,replicas=2,skip_cleanup=True,num_items=100000000,num_buckets=1,bucket_names=GleamBook,doc_size=512,bucket_type=membase,compression_mode=off,iterations=10,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=info,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,maxttl=60,num_collections=50,doc_ops=create:update:expiry,durability=None,ramQuota=1024,pc=1,dgm=2,cursor_dropping_checkpoint=100 -m rest'
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            owend Daniel Owen added a comment -

            Added a depends on MB-43818 - as we need additional debug information to be able to track down the issue.

            owend Daniel Owen added a comment - Added a depends on MB-43818 - as we need additional debug information to be able to track down the issue.
            drigby Dave Rigby added a comment -

            Ritesh Agarwal We have improved logging patches included in 7.0.0-4413 - please can you re-run with that build or newer - thanks.

            drigby Dave Rigby added a comment - Ritesh Agarwal We have improved logging patches included in 7.0.0-4413 - please can you re-run with that build or newer - thanks.

            Not abe to reproduce it on 5247 CC build. Closing the defect unit we see it again.

            ritesh.agarwal Ritesh Agarwal added a comment - Not abe to reproduce it on 5247 CC build. Closing the defect unit we see it again.

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty