Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41844

Data race in FollyExecutorPool::doTaskQStat

    XMLWordPrintable

Details

    • Triaged
    • 1
    • No

    Description

      As seen in TSan testing with FollyExecutorPool when running ep_perfsuite:

      /home/couchbase/couchbase/build/kv_engine/ep_perfsuite "-E" "ep" "-v" "-e" "dbname=./ep_perfsuite.value_eviction.db" -C 13
      

      Running [0013/0017]: Stat latency with 100 vbuckets. Also sets & DCP traffic on separate thread...==================
      WARNING: ThreadSanitizer: data race (pid=16435)
        Read of size 8 at 0x7b24000053f8 by main thread (mutexes: write M386882167168307840):
          #0 folly::HHWheelTimerBase<std::chrono::duration<long, std::ratio<1l, 1000l> > >::Callback::isScheduled() const follytsan/folly/io/async/HHWheelTimer.h:121:14 (libep.so+0x5c63b6)
          #1 FollyExecutorPool::doTaskQStat(Taskable&, void const*, std::function<void (std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, gsl::not_null<void const*>)> const&) kv_engine/engines/ep/src/folly_executorpool.cc:839:30 (libep.so+0x34ff85)
          #2 EventuallyPersistentEngine::doWorkloadStats(void const*, std::function<void (std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, gsl::not_null<void const*>)> const&) kv_engine/engines/ep/src/ep_engine.cc:4115:17 (libep.so+0x2e67d6)
          #3 EventuallyPersistentEngine::getStats(void const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::function<void (std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, gsl::not_null<void const*>)> const&) kv_engine/engines/ep/src/ep_engine.cc:4670:16 (libep.so+0x2d689a)
      ...
       
        Previous write of size 8 at 0x7b24000053f8 by thread T1:
          #0 memset <null> (ep_perfsuite+0x4b3a42)
          #1 folly::HHWheelTimerBase<std::chrono::duration<long, std::ratio<1l, 1000l> > >::Callback::cancelTimeoutImpl() follytsan/folly/io/async/HHWheelTimer.cpp:86:15 (libep.so+0x5c6507)
          #2 folly::HHWheelTimerBase<std::chrono::duration<long, std::ratio<1l, 1000l> > >::Callback::cancelTimeout() follytsan/folly/io/async/HHWheelTimer.h:114:7 (libep.so+0x5c6507)
          #3 FollyExecutorPool::TaskProxy::wake() kv_engine/engines/ep/src/folly_executorpool.cc:239:9 (libep.so+0x35e3e9)
          #4 FollyExecutorPool::State::wakeTask(unsigned long) kv_engine/engines/ep/src/folly_executorpool.cc:352:29 (libep.so+0x3625cf)
          #5 FollyExecutorPool::wake(unsigned long)::$_9::operator()() const kv_engine/engines/ep/src/folly_executorpool.cc:731:32 (libep.so+0x3515d1)
          #6 void folly::detail::function::FunctionTraits<void ()>::callSmall<FollyExecutorPool::wake(unsigned long)::$_9>(folly::detail::function::Data&) folly/Function.h:387:7 (libep.so+0x351545)
          #7 folly::detail::function::FunctionTraits<void ()>::operator()() follytsan/folly/Function.h:416:12 (libep.so+0x5bd22b)
          #8 folly::EventBase::runInEventBaseThreadAndWait(folly::Function<void ()>)::$_9::operator()() follytsan/folly/io/async/EventBase.cpp:671:5 (libep.so+0x5bd22b)
          #9 void folly::detail::function::FunctionTraits<void ()>::callBig<folly::EventBase::runInEventBaseThreadAndWait(folly::Function<void ()>)::$_9>(folly::detail::function::Data&) follytsan/folly/Function.h:401:7 (libep.so+0x5bd22b)
          #10 folly::detail::function::FunctionTraits<void ()>::operator()() follytsan/folly/Function.h:416:12 (libep.so+0x5c3dbc)
          #11 folly::EventBase::FunctionRunner::messageAvailable(folly::Function<void ()>&&) follytsan/folly/io/async/EventBase.cpp:137:5 (libep.so+0x5c3dbc)
          #12 folly::NotificationQueue<folly::Function<void ()> >::Consumer::consumeMessages(bool, unsigned long*) follytsan/folly/io/async/NotificationQueue.h:729:7 (libep.so+0x5c090a)
          #13 folly::NotificationQueue<folly::Function<void ()> >::Consumer::handlerReady(unsigned short) follytsan/folly/io/async/NotificationQueue.h:663:3 (libep.so+0x5c3e7d)
          #14 non-virtual thunk to folly::NotificationQueue<folly::Function<void ()> >::Consumer::handlerReady(unsigned short) follytsan/folly/io/async/NotificationQueue.h (libep.so+0x5c3e7d)
          #15 folly::EventHandler::libeventCallback(int, short, void*) follytsan/folly/io/async/EventHandler.cpp:158:12 (libep.so+0x5c577c)
          #16 event_persist_closure /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1629 (libevent_core-2.1.so.7+0x17f39)
          #17 event_process_active_single_queue.isra.31 /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1688 (libevent_core-2.1.so.7+0x17f39)
          #18 folly::EventBase::loopBody(int, bool) follytsan/folly/io/async/EventBase.cpp:397:19 (libep.so+0x5b8c72)
          #19 folly::EventBase::loop() follytsan/folly/io/async/EventBase.cpp:315:10 (libep.so+0x5ba1db)
          #20 folly::EventBase::loopForever() follytsan/folly/io/async/EventBase.cpp:538:11 (libep.so+0x5ba1db)
      ...
       
      SUMMARY: ThreadSanitizer: data race follytsan/folly/io/async/HHWheelTimer.h:121:14 in folly::HHWheelTimerBase<std::chrono::duration<long, std::ratio<1l, 1000l> > >::Callback::isScheduled() const
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-41844
          # Subject Branch Project Status CR V

          Activity

            People

              drigby Dave Rigby (Inactive)
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty