Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44469

CouchStore: Crash observed in ~VBucketMemoryDeletionTask () at ../kv_engine/engines/ep/src/vbucketdeletiontask.cc:45

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • None
    • Cheshire-Cat
    • couchbase-bucket
    • 7.0.0-4498
    • Untriaged
    • 1
    • No

    Description

      Steps:

      1. Create a 2 node cluster
      2. Rebalance completed with progress: 100% in 10.0649998188 sec
      3. Create required buckets and collections. num_buckets=10,num_scopes=5,num_collections=5
      4. Create 1000000 items sequentially
      5. Rebalance in with Loading of docs
      6. Rebalance reached >20.1857347513% in 90.381000042 seconds. Abort and Restart Rebalance
      7. Rebalance reached >41.2707476168% in 10.0380001068 seconds. Abort and Restart Rebalance
      8. Rebalance reached >61.4239494142% in 30.0769999027 seconds. Abort and Restart Rebalance
      9. Rebalance reached >80.7363839779% in 30.0840001106 seconds. Abort and Restart Rebalance
      10. Rebalance completed with progress: 100% in 207.252000093 sec
      11. Sleep 61 seconds. Reason: Iteration:0 waiting to kill memc on all nodes
      12. Rebalance Out with Loading of docs
      13. Rebalance reached >20.0% in 240.77699995 seconds. Abort and Restart Rebalance
      14. Rebalance reached >40.0% in 190.606999874 seconds. Abort and Restart Rebalance
      15. Rebalance reached >60.0% in 150.573999882 seconds. Abort and Restart Rebalance
      16. Rebalance reached >80.0% in 100.386000156 seconds. Abort and Restart Rebalance
      17. Rebalance completed with progress: 100% in 298.411999941 sec
      18. Sleep 111 seconds. Reason: Iteration:0 waiting to kill memc on all nodes
      19. Rebalance In_Out with Loading of docs
      20. Rebalance reached >20.0% in 401.085999966 seconds. Abort and Restart Rebalance
      21. Rebalance reached >40.0% in 100.285999775 seconds. Abort and Restart Rebalance
      22. Rebalance reached >60.0% in 230.694000006 seconds. Abort and Restart Rebalance
      23. Rebalance reached >80.0% in 260.7099998 seconds. Abort and Restart Rebalance
      24. Rebalance completed with progress: 100% in 435.161000013 sec
      25. Sleep 99 seconds. Reason: Iteration:0 waiting to kill memc on all nodes

      node-68

      		 
      172.23.110.68 : Stack Trace of 6416790e-4e6b-4af6-93a4a18b-9301eab5.dmp
      		 
      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
      		 
       #0  0x00007f0b492843d7 in raise () from /lib64/libc.so.6
      		 
       #0  0x00007f0b492843d7 in raise () from /lib64/libc.so.6
      		 
       #1  0x00007f0b49285ac8 in abort () from /lib64/libc.so.6
      		 
       #2  0x00007f0b49de2195 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95
      		 
       #3  0x000000000055c6f2 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:93
      		 
       #4  0x00007f0b49ddff86 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
      		 
       #5  0x00007f0b49ddef49 in __cxa_call_terminate (ue_header=ue_header@entry=0x7f0890000920) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_call.cc:54
      		 
       #6  0x00007f0b49ddf8b8 in __cxxabiv1::__gxx_personality_v0 (version=<optimized out>, actions=6, exception_class=5138137972254386944, ue_header=0x7f0890000920, context=<optimized out>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_personality.cc:676
      		 
       #7  0x00007f0b49847eb3 in _Unwind_RaiseException_Phase2 (exc=exc@entry=0x7f0890000920, context=context@entry=0x7f08a27e2b50) at /tmp/deploy/objdir/../gcc-7.3.0/libgcc/unwind.inc:62
      		 
       #8  0x00007f0b498486de in _Unwind_Resume (exc=exc@entry=0x7f0890000920) at /tmp/deploy/objdir/../gcc-7.3.0/libgcc/unwind.inc:230
      		 
       #9  0x00007f0b4d6ea6f9 in unlock (mutex=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/LockTraits.h:131
      		 
       #10 unlock<folly::SharedMutexImpl<false> > (mutex=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/LockTraits.h:377
      		 
       #11 ~LockedPtrBase (this=<optimized out>, __in_chrg=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Synchronized.h:1075
      		 
       #12 ~LockedPtr (this=<optimized out>, __in_chrg=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Synchronized.h:1450
      		 
       #13 Collections::Manager::dereferenceMeta (this=0x7f0735f5c190, cid=cid@entry=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/manager.cc:373
      		 
       #14 0x00007f0b4d7080f4 in Collections::VB::Manifest::~Manifest (this=0x7f07361aee40, __in_chrg=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest.cc:69
      		 
       #15 0x00007f0b4d6577fb in operator() (this=<optimized out>, __ptr=0x7f07361aee40) at /usr/local/include/c++/7.3.0/bits/unique_ptr.h:78
      		 
       #16 ~unique_ptr (this=0x7f07361c55e8, __in_chrg=<optimized out>) at /usr/local/include/c++/7.3.0/bits/unique_ptr.h:268
      		 
       #17 VBucket::~VBucket (this=0x7f07361c5400, __in_chrg=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:258
      		 
       #18 0x00007f0b4d59f548 in EPVBucket::~EPVBucket (this=0x7f07361c5400, __in_chrg=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_vb.cc:97
      		 
       #19 0x00007f0b4d59f629 in EPVBucket::~EPVBucket (this=0x7f07361c5400, __in_chrg=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_vb.cc:102
      		 
       #20 0x00007f0b4d66784d in operator() (this=<optimized out>, __ptr=<optimized out>) at /usr/local/include/c++/7.3.0/bits/unique_ptr.h:78
      		 
       #21 reset (__p=<optimized out>, this=0x7f07347d8468) at /usr/local/include/c++/7.3.0/bits/unique_ptr.h:376
      		 
       #22 ~VBucketMemoryDeletionTask (this=0x7f07347d8410, __in_chrg=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucketdeletiontask.cc:45
      		 
       #23 VBucketMemoryAndDiskDeletionTask::~VBucketMemoryAndDiskDeletionTask (this=0x7f07347d8410, __in_chrg=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucketdeletiontask.h:77
      		 
       #24 0x00007f0b4d5e8c3a in _M_release (this=0x7f07347d8400) at /usr/local/include/c++/7.3.0/bits/shared_ptr_base.h:154
      		 
       #25 ~__shared_count (this=<optimized out>, __in_chrg=<optimized out>) at /usr/local/include/c++/7.3.0/bits/shared_ptr_base.h:684
      		 
       #26 ~__shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/local/include/c++/7.3.0/bits/shared_ptr_base.h:1123
      		 
       #27 reset (this=0x7f08a27e3040) at /usr/local/include/c++/7.3.0/bits/shared_ptr_base.h:1235
      		 
       #28 operator() (__closure=0x7f08a27e3040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:249
      		 
       #29 folly::detail::function::FunctionTraits<void ()>::callSmall<FollyExecutorPool::TaskProxy::resetTaskPtrViaCpuPool()::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387
      		 
       #30 0x00007f0b4d75fa60 in operator() (this=0x7f08a27e3040) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416
      		 
       #31 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) (this=0x7f0af5bb8800, thread=..., task=<unknown type in /usr/lib/debug/opt/couchbase/lib/libep.so.debug, CU 0x354d664, DIE 0x3593b0d>) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97
      		 
       #32 0x00007f0b4d747f3a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f0af5bb8800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265
      		 
       #33 0x00007f0b4d762a19 in __invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__t=<optimized out>, __f=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:73
      		 
       #34 __invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__fn=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:95
      		 
       #35 __call<void, 0, 1> (__args=<optimized out>, this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:467
      		 
       #36 operator()<> (this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:551
      		 
       #37 folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401
      		 
       #38 0x00007f0b4d5e78e8 in operator() (this=0x7f0af6f91dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416
      		 
       #39 operator() (__closure=0x7f0af6f91dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54
      		 
       #40 folly::detail::function::FunctionTraits<void ()>::callBig<CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401
      		 
       #41 0x00007f0b4d5e77c3 in operator() (this=0x7f0af6f7e260) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416
      		 
       #42 operator() (__closure=0x7f0af6f7e250) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54
      		 
       #43 folly::detail::function::FunctionTraits<void ()>::callBig<folly::PriorityThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401
      		 
       #44 0x00007f0b49e0adcf in std::execute_native_thread_routine (__p=0x7f0af58e8280) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83
      		 
       #45 0x00007f0b49623ea5 in start_thread () from /lib64/libpthread.so.0
      		 
       #46 0x00007f0b4934c9fd in clone () from /lib64/libc.so.6

      Note: File collectinfo-2021-02-19T125728-ns_1@172.23.110.64.zip > 100MB hence couldn't upload it here.

      Links to cbcollect done at a later stage:
      http://supportal.couchbase.com/snapshot/334c5101c7675a996461e3dddb4fe934::0
      s3://cb-customers-secure/vbucketmemoryanddiskdeletiontask/2021-02-19/collectinfo-2021-02-19t164641-ns_1@172.23.110.65.zip
      s3://cb-customers-secure/vbucketmemoryanddiskdeletiontask/2021-02-19/collectinfo-2021-02-19t164641-ns_1@172.23.110.64.zip
      s3://cb-customers-secure/vbucketmemoryanddiskdeletiontask/2021-02-19/collectinfo-2021-02-19t164641-ns_1@172.23.110.68.zip
      s3://cb-customers-secure/vbucketmemoryanddiskdeletiontask/2021-02-19/collectinfo-2021-02-19t164641-ns_1@172.23.110.69.zip

      consoleText.txt

      QE Test

      		 
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P "args=-i /tmp/magma_temp_job2.ini -p bucket_storage=couchstore,bucket_eviction_policy=fullEviction -t volumetests.Magma.volume.SystemTestMagma,nodes_init=2,replicas=1,skip_cleanup=True,num_items=1000000,doc_size=256,bucket_type=membase,compression_mode=off,iterations=10,crashes=0,batch_size=10,sdk_timeout=60,log_level=debug,infra_log_level=info,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,maxttl=1200,num_buckets=10,num_scopes=5,num_collections=5,doc_ops=create:update:delete:expiry,durability=MAJORITY,sdk_client_pool=True,pc=1 -m rest"

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-44417 [Collections] - Collection CRUD + Multi node graceful failover + dgm + rebalance out fails with wait_seqno_persisted_failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty