Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38961

Memcached couldn't recover after killing purposely: EPVBucket::queueBGFetchItem(DocKey const&, std::unique_ptr<VBucketBGFetchItem, std::default_delete<VBucketBGFetchItem> >, BgFetcher*) () from /opt/couchbase/lib/ep.so

    XMLWordPrintable

Details

    Description

      Steps:
      1. Create a 10 node cluster
      2. Load 1000000 items to the bucket.
      3. Load random num_items(1-1000000)
      4. Kill memcached on all nodes.
      5. Repeat step 3,4 multiple times
      6. Memc couldn't recover at 10th iteration.

      Crash in Logs:

      memcached.log.000021.txt:2020-04-23T13:53:49.080411-07:00 CRITICAL Breakpad caught a crash (Couchbase version 1006.5.1-1121). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/4d49bd10-0bf9-fd28-59565089-242ab996.dmp before terminating.
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080440-07:00 CRITICAL Stack backtrace of crashed thread:
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080762-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x1324cd]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080780-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14a50e]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080789-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14a824]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080797-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f21dc9d9000+0xf5f0]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080809-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f21d78df000+0xf2047]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080818-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f21d78df000+0xef83d]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080833-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f21d78df000+0xeff2e]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080839-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f21d78df000+0xed92f]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080847-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f21d78df000+0x19674a]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080855-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f21d78df000+0x15f604]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080862-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f21d78df000+0xf77d3]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080868-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f21d78df000+0xf7955]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080876-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x55f59]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080884-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0xc0145]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080892-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0xc0568]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080901-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0xc9fc0]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080909-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0xa9012]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080915-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0xac5c4]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080922-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x9895c]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080931-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0xcd9a8]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080938-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x88930]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080944-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x8ab5b]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080952-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x9771c]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080991-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x6bbec]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.080997-07:00 CRITICAL     /opt/couchbase/bin/../lib/libevent_core.so.2.1.8() [0x7f21de285000+0x17107]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.081028-07:00 CRITICAL     /opt/couchbase/bin/../lib/libevent_core.so.2.1.8(event_base_loop+0x39f) [0x7f21de285000+0x1767f]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.081035-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x69d29]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.081040-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f21defb7000+0x8ee7]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.081046-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f21dc9d9000+0x7e65]
      		 
      memcached.log.000021.txt:2020-04-23T13:53:49.081084-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7f21dc60b000+0xfe88d]

      Stack Trace:

      172.23.120.61 : Stack Trace of 4d49bd10-0bf9-fd28-59565089-242ab996.dmp
      		 
      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
      		 
       #0  0x00007f21d79d1047 in std::__detail::_Map_base<DiskDocKey, std::pair<DiskDocKey const, vb_bgfetch_item_ctx_t>, std::allocator<std::pair<DiskDocKey const, vb_bgfetch_item_ctx_t> >, std::__detail::_Select1st, std::equal_to<DiskDocKey>, std::hash<DiskDocKey>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](DiskDocKey const&) () from /opt/couchbase/lib/ep.so
      		 
       #0  0x00007f21d79d1047 in std::__detail::_Map_base<DiskDocKey, std::pair<DiskDocKey const, vb_bgfetch_item_ctx_t>, std::allocator<std::pair<DiskDocKey const, vb_bgfetch_item_ctx_t> >, std::__detail::_Select1st, std::equal_to<DiskDocKey>, std::hash<DiskDocKey>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](DiskDocKey const&) () from /opt/couchbase/lib/ep.so
      		 
       #1  0x00007f21d79ce83d in EPVBucket::queueBGFetchItem(DocKey const&, std::unique_ptr<VBucketBGFetchItem, std::default_delete<VBucketBGFetchItem> >, BgFetcher*) () from /opt/couchbase/lib/ep.so
      		 
       #2  0x00007f21d79cef2e in EPVBucket::bgFetch(DocKey const&, void const*, EventuallyPersistentEngine&, bool) () from /opt/couchbase/lib/ep.so
      		 
       #3  0x00007f21d79cc92f in EPVBucket::addTempItemAndBGFetch(HashTable::HashBucketLock&, DocKey const&, void const*, EventuallyPersistentEngine&, bool) () from /opt/couchbase/lib/ep.so
      		 
       #4  0x00007f21d7a7574a in VBucket::add(Item&, void const*, EventuallyPersistentEngine&, Collections::VB::Manifest::CachingReadHandle const&) () from /opt/couchbase/lib/ep.so
      		 
       #5  0x00007f21d7a3e604 in KVBucket::add(Item&, void const*) () from /opt/couchbase/lib/ep.so
      		 
       #6  0x00007f21d79d67d3 in EventuallyPersistentEngine::storeIfInner(void const*, Item&, unsigned long, ENGINE_STORE_OPERATION, std::function<cb::StoreIfStatus (boost::optional<item_info> const&, cb::vbucket_info)> const&) () from /opt/couchbase/lib/ep.so
      		 
       #7  0x00007f21d79d6955 in EventuallyPersistentEngine::store_if(gsl::not_null<void const*>, gsl::not_null<void*>, unsigned long, ENGINE_STORE_OPERATION, std::function<cb::StoreIfStatus (boost::optional<item_info> const&, cb::vbucket_info)> const&, boost::optional<cb::durability::Requirements> const&, DocumentState) () from /opt/couchbase/lib/ep.so
      		 
       #8  0x0000000000455f59 in bucket_store_if(Cookie&, gsl::not_null<void*>, unsigned long, ENGINE_STORE_OPERATION, std::function<cb::StoreIfStatus (boost::optional<item_info> const&, cb::vbucket_info)>, boost::optional<cb::durability::Requirements>, DocumentState) ()
      		 
       #9  0x00000000004c0145 in MutationCommandContext::storeItem() ()
      		 
       #10 0x00000000004c0568 in MutationCommandContext::step() ()
      		 
       #11 0x00000000004c9fc0 in SteppableCommandContext::drive() ()
      		 
       #12 0x00000000004a9012 in add_executor(Cookie&) ()
      		 
       #13 0x00000000004ac5c4 in execute_client_request_packet(Cookie&, cb::mcbp::Request const&) ()
      		 
       #14 0x000000000049895c in Cookie::execute() ()
      		 
       #15 0x00000000004cd9a8 in StateMachine::conn_execute() ()
      		 
       #16 0x0000000000488930 in Connection::runStateMachinery() ()
      		 
       #17 0x000000000048ab5b in Connection::runEventLoop(short) ()
      		 
       #18 0x000000000049771c in run_event_loop(Connection*, short) ()
      		 
       #19 0x000000000046bbec in thread_libevent_process(int, short, void*) ()
      		 
       #20 0x00007f21de29c107 in event_process_active_single_queue.isra.26 () from /opt/couchbase/lib/libevent_core.so.2.1.8
      		 
       #21 0x00007f21de29c67f in event_base_loop () from /opt/couchbase/lib/libevent_core.so.2.1.8
      		 
       #22 0x0000000000469d29 in worker_libevent(void*) ()
      		 
       #23 0x00007f21defbfee7 in platform_thread_wrap(void*) () from /opt/couchbase/lib/libplatform_so.so.0.1.0
      		 
       #24 0x00007f21dc9e0e65 in start_thread () from /usr/lib64/libpthread-2.17.so
      		 
       #25 0x00007f21dc70988d in clone () from /usr/lib64/libc-2.17.so

      Test:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ritesh_magma.ini -t magma.magma_failure_conditions.MagmaCrashTests.test_crash_magma_n_times,num_items=1000000,infra_log_level=debug,skip_cleanup=true,rerun=False,doc_size=1024,doc_size_random=True,nodes_init=10,num_crashes=100,sdk_timeout=60,bucket_storage=magma,replicas=1,vbuvkets=128 -m rest'

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ben.huddleston Ben Huddleston
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty