Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51608

Memcached crashes in 20 bucket throughput test due to exception ThreadLocalData::getTCacheID: tcache.create failed rv:14

    XMLWordPrintable

Details

    Description

      Several crashes are observed in a magma 20 bucket throughput test. A 10 bucket variant of the test doesn't crash.

      The test creates 20 buckets, loads 50M docs each, then overwrite 50M docs and finally run an access phase of 50:50 read-writes. The crashes are observed during the access phase.

      je_mallctl("tcache.create",..) is returning 14 (EFAULT).

      Test run:
      http://perf.jenkins.couchbase.com/job/rhea-dev2/130/console

      BackTrace:

      (gdb) bt
      #0  0x00007fe73722b3d7 in raise () from /lib64/libc.so.6
      #1  0x00007fe73722cac8 in abort () from /lib64/libc.so.6
      #2  0x00007fe737b7663c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
      #3  0x0000000000b3255b in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88
      #4  0x00007fe737b818f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      #5  0x00007fe737b81961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      #6  0x00007fe737b81bf4 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x107f4e0 <typeinfo for std::runtime_error>, dest=0x444ef0 <_ZNSt13runtime_errorD1Ev@plt>)
          at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95
      #7  0x000000000053c64f in cb::ThreadLocalData::getTCacheID (this=0x7fe4aefedac0, client=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/je_arena_malloc.cc:373
      #8  0x0000000000bab3cb in cb::_JEArenaMalloc<cb::JEArenaCoreLocalTracker>::switchToClient(cb::ArenaMallocClient const&, cb::MemoryDomain, bool) ()
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/je_arena_malloc.cc:225
      #9  0x000000000074f2b0 in switchToClient (tcache=true, domain=cb::Primary, client=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.h:855
      #10 switchToEngine (domain=cb::Primary, want_old_thread_local=true, engine=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/objectregistry.cc:123
      #11 onSwitchThread (domain=cb::Primary, want_old_thread_local=true, engine=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/objectregistry.cc:136
      #12 BucketAllocationGuard::BucketAllocationGuard (this=0x7fe4aefeb3a0, engine=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/objectregistry.cc:148
      #13 0x0000000000aa0f85 in GlobalTask::execute(std::basic_string_view<char, std::char_traits<char> >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:72
      #14 0x0000000000a9a6aa in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7fe4aefeb650)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309
      #15 0x0000000000aa238e in operator() (this=0x7fe4aefeb650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42
      #16 CancellableCPUExecutor::add(GlobalTask*, folly::Function<void ()>)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42
      #17 0x0000000000bf9a90 in operator() (this=0x7fe4aefeb840) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416
      #18 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7fe734ddf900, thread=...,
          task=task@entry=<unknown type in /usr/lib/debug/opt/couchbase/bin/memcached-7.1.0-2506.x86_64.debug, CU 0xa40c0ed, DIE 0xa490022>)
          at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97
      #19 0x0000000000be452a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fe734ddf900, thread=...)
          at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265
      #20 0x0000000000bfca49 in __invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (
          __t=<optimized out>, __f=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:73
      #21 __invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__fn=<optimized out>)
          at /usr/local/include/c++/7.3.0/bits/invoke.h:95
      #22 __call<void, 0, 1> (__args=<optimized out>, this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:467
      #23 operator()<> (this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:551
      #24 folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401
      #25 0x0000000000a9a3a4 in operator() (this=0x7fe730144b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49
      #26 operator() (__closure=0x7fe730144b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49
      #27 folly::detail::function::FunctionTraits<void ()>::callBig<CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401
      #28 0x00007fe737baad40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80
      #29 0x00007fe7399b2ea5 in start_thread () from /lib64/libpthread.so.0
      #30 0x00007fe7372f39fd in clone () from /lib64/libc.so.6
      

      Attachments

        Issue Links

          Activity

            People

              bo-chun.wang Bo-Chun Wang
              sarath Sarath Lakshman
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty