Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46765

Magma crashed while bucket flush and disk is full.

    XMLWordPrintable

Details

    Description

      QE Test

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.95899.ini bucket_storage=magma,rerun=false,bucket_eviction_policy=fullEviction,randomize_value=True,doc_size=4096,enable_dp=True,log_level=debug,infra_log_level=debug,upgrade_version=7.0.0-5247 -t magma.magma_disk_full.MagmaDiskFull.test_bucket_flush_disk_full,nodes_init=4,num_items=5000000,doc_size=4096,sdk_timeout=60,replicas=1,GROUP=P0'
      

      Snapshot from thread dump

      Thread 29 (Thread 0x7f81797d8700 (LWP 22942)):
       #0  0x00000000009b9912 in magma::Memtable::Add(magma::Operation, long, magma::Slice const&, magma::Slice const&) ()
       #1  0x0000000000961a67 in magma::LSMTree::add(magma::Operation, magma::Slice const&, magma::Slice const&, bool) ()
       #2  0x000000000096253e in magma::LSMTree::Write(magma::Operation, magma::Slice const&, magma::Slice const&) ()
       #3  0x0000000000939fcd in magma::KVStore::writeDocument(magma::Magma::WriteOperation const&, magma::LookupBuffer*, bool, std::function<void (magma::Magma::WriteOperation const&, bool, magma::Slice)>, magma::WAL*) ()
       #4  0x000000000093a37c in magma::KVStore::Write(magma::Magma::WriteOperation const&, magma::LookupBuffer*, bool, std::function<void (magma::Magma::WriteOperation const&, bool, magma::Slice)>, magma::WAL*) ()
       #5  0x00000000009287e7 in magma::Magma::Impl::WriteDocs(unsigned short, std::vector<magma::Magma::WriteOperation, std::allocator<magma::Magma::WriteOperation> > const&, unsigned int, std::function<void (magma::Magma::WriteOperation const&, bool, magma::Slice)>, std::function<magma::Status (std::vector<magma::Magma::WriteOperation, std::allocator<magma::Magma::WriteOperation> >&)>) ()
       #6  0x0000000000928c72 in magma::Magma::WriteDocs(unsigned short, std::vector<magma::Magma::WriteOperation, std::allocator<magma::Magma::WriteOperation> > const&, unsigned int, std::function<void (magma::Magma::WriteOperation const&, bool, magma::Slice)>, std::function<magma::Status (std::vector<magma::Magma::WriteOperation, std::allocator<magma::Magma::WriteOperation> >&)>) ()
       #7  0x000000000084f1a7 in MagmaKVStore::saveDocs(VB::Commit&, kvstats_ctx&) ()
       #8  0x0000000000854c30 in MagmaKVStore::commit(VB::Commit&) ()
       #9  0x00000000007ee44e in EPBucket::commit(Vbid, KVStore&, VB::Commit&) ()
       #10 0x00000000007f379d in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) ()
       #11 0x00000000007f3b4f in EPBucket::flushVBucket(Vbid) ()
       #12 0x000000000069d860 in Flusher::flushVB() ()
       #13 0x000000000069e329 in Flusher::step(GlobalTask*) ()
       #14 0x00000000006a0567 in GlobalTask::execute() ()
       #15 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const ()
       #16 0x0000000000b0a3b0 in folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) ()
       #17 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
       #18 0x0000000000b0d369 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
       #19 0x0000000000699d4d in void folly::detail::function::FunctionTraits<void ()>::callBig<CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) ()
       #20 0x00007f81e451cd40 in execute_native_thread_routine ()
           at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80
       #21 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0
       #22 0x00007f81e3c6588d in clone () from /lib64/libc.so.6
      

      Full thread dump attached.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            #0  0x00007f100f86e337 in ssignal () from /lib64/libc.so.6
            #1  0x00007f100f86fa28 in abort () from /lib64/libc.so.6
            #2  0x00007f10101b963c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
            #3  0x0000000000a7c65f in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:87
            #4  0x00007f10101c48f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
            #5  0x00007f10101c4961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
            #6  0x00007f10101c4bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f0fb4000940, tinfo=0xf676e0 <typeinfo for std::runtime_error>, dest=0x4435c0 <_ZNSt13runtime_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95
            #7  0x00000000004c7874 in MagmaKVStore::makeCompactionContext (this=0x7f100029a400, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:2623
            #8  0x0000000000859585 in MagmaKVStore::compactionCallBack(MagmaKVStore::MagmaCompactionCB&, magma::Slice const&, magma::Slice const&, magma::Slice const&) ()
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:278
            #9  0x000000000094a6bd in magma::DocSeqGCContext::Transform (this=0x7f0f886b71c0, key=..., value=..., buffer=..., outputValue=0x7f0f5f7dbd50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/format.h:44
            #10 0x00000000009cf755 in magma::ItemGCFilter::Process(std::vector<std::pair<magma::Slice, magma::Slice>, std::allocator<std::pair<magma::Slice, magma::Slice> > >&, std::vector<std::pair<magma::Slice, magma::Slice>, std::allocator<std::pair<magma::Slice, magma::Slice> > >*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/mvcc.cc:106
            

            This is duplicate of MB-46811.

            sarath Sarath Lakshman added a comment - #0 0x00007f100f86e337 in ssignal () from /lib64/libc.so.6 #1 0x00007f100f86fa28 in abort () from /lib64/libc.so.6 #2 0x00007f10101b963c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x0000000000a7c65f in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:87 #4 0x00007f10101c48f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 #5 0x00007f10101c4961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 #6 0x00007f10101c4bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f0fb4000940, tinfo=0xf676e0 <typeinfo for std::runtime_error>, dest=0x4435c0 <_ZNSt13runtime_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 #7 0x00000000004c7874 in MagmaKVStore::makeCompactionContext (this=0x7f100029a400, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:2623 #8 0x0000000000859585 in MagmaKVStore::compactionCallBack(MagmaKVStore::MagmaCompactionCB&, magma::Slice const&, magma::Slice const&, magma::Slice const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:278 #9 0x000000000094a6bd in magma::DocSeqGCContext::Transform (this=0x7f0f886b71c0, key=..., value=..., buffer=..., outputValue=0x7f0f5f7dbd50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/format.h:44 #10 0x00000000009cf755 in magma::ItemGCFilter::Process(std::vector<std::pair<magma::Slice, magma::Slice>, std::allocator<std::pair<magma::Slice, magma::Slice> > >&, std::vector<std::pair<magma::Slice, magma::Slice>, std::allocator<std::pair<magma::Slice, magma::Slice> > >*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/mvcc.cc:106 This is duplicate of MB-46811 .

            Ritesh Agarwal Can we rerun this test on master?

            sarath Sarath Lakshman added a comment - Ritesh Agarwal Can we rerun this test on master?

            People

              sarath Sarath Lakshman
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty