Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41309

TSan: false-positive data race in ExecutorPoolTest/1.increase_workers (FollyExecutorPool)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • Cheshire-Cat
    • couchbase-bucket
    • Triaged
    • 1
    • No

    Description

      As seen during CV, TSan intermittently reports the following data race inside Folly's SaturatingSemaphore:

      [ RUN      ] ExecutorPoolTest/1.increase_workers
       ==================
       WARNING: ThreadSanitizer: data race (pid=123526)
         Write of size 4 at 0x7f863e05e0a8 by thread T19:
           #0 std::__atomic_base<unsigned int>::__atomic_base(unsigned int) /usr/local/include/c++/7.3.0/bits/atomic_base.h:256 (ep-engine_ep_unit_tests+0x000001721429)
           #1 std::atomic<unsigned int>::atomic(unsigned int) /usr/local/include/c++/7.3.0/atomic:695 (ep-engine_ep_unit_tests+0x000001721429)
           #2 void folly::detail::LifoSemNode<...>::init<>() folly/synchronization/SaturatingSemaphore.h:135 (ep-engine_ep_unit_tests+0x000001721429)
           #3 std::unique_ptr<...>::allocateNode<>() folly/synchronization/LifoSem.h:575 (ep-engine_ep_unit_tests+0x000001721429)
           #4 bool folly::detail::LifoSemBase<...>::try_wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) folly/synchronization/LifoSem.h:505 (ep-engine_ep_unit_tests+0x000001721429)
           #5 bool folly::detail::LifoSemBase<...>::try_wait_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) folly/synchronization/LifoSem.h:491 (ep-engine_ep_unit_tests+0x000001726f13)
           #6 folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask>::try_take_for(std::chrono::duration<long, std::ratio<1l, 1000l> >) folly/executors/task_queue/UnboundedBlockingQueue.h:41 (ep-engine_ep_unit_tests+0x000001726f13)
           #7 folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) folly/executors/CPUThreadPoolExecutor.cpp:178 (ep-engine_ep_unit_tests+0x000001713bc5)
           #8 void std::__invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>(std::__invoke_memfun_deref, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&) /usr/local/include/c++/7.3.0/bits/invoke.h:73 (ep-engine_ep_unit_tests+0x00000173fe1b)
           #9 std::__invoke_result<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>::type std::__invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>(void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&) /usr/local/include/c++/7.3.0/bits/invoke.h:95 (ep-engine_ep_unit_tests+0x00000173fe1b)
           #10 void std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /usr/local/include/c++/7.3.0/functional:467 (ep-engine_ep_unit_tests+0x00000173fe1b)
           #11 void std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>::operator()<, void>() /usr/local/include/c++/7.3.0/functional:551 (ep-engine_ep_unit_tests+0x00000173fe1b)
           #12 void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) folly/Function.h:367 (ep-engine_ep_unit_tests+0x00000173fe1b)
           #13 std::thread::_State_impl<std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}> > >::_M_run() <null> (ep-engine_ep_unit_tests+0x00000062d56a)
           #14 execute_native_thread_routine /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 (libstdc++.so.6+0x0000000b995e)
       
         Previous atomic write of size 4 at 0x7f863e05e0a8 by thread T17:
           #0 __tsan_atomic32_compare_exchange_strong <null> (libtsan.so.0+0x000000064842)
           #1 std::__atomic_base<unsigned int>::compare_exchange_strong(unsigned int&, unsigned int, std::memory_order, std::memory_order) /usr/local/include/c++/7.3.0/bits/atomic_base.h:477 (ep-engine_ep_unit_tests+0x00000171fde2)
           #2 folly::SaturatingSemaphore<true, std::atomic>::postFastWaiterMayBlock() folly/synchronization/SaturatingSemaphore.h:196 (ep-engine_ep_unit_tests+0x00000171fde2)
           #3 folly::SaturatingSemaphore<true, std::atomic>::post() folly/synchronization/SaturatingSemaphore.h:155 (ep-engine_ep_unit_tests+0x00000171fde2)
           #4 folly::detail::LifoSemBase<...>::post() folly/synchronization/LifoSem.h:372 (ep-engine_ep_unit_tests+0x00000171fde2)
           #5 folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask>::add(folly::CPUThreadPoolExecutor::CPUTask) folly/executors/task_queue/UnboundedBlockingQueue.h:32 (ep-engine_ep_unit_tests+0x000001725cec)
           #6 folly::CPUThreadPoolExecutor::add(folly::Function<void ()>, std::chrono::duration<long, std::ratio<1l, 1000l> >, folly::Function<void ()>) folly/executors/CPUThreadPoolExecutor.cpp:114 (ep-engine_ep_unit_tests+0x0000017111fe)
           #7 folly::CPUThreadPoolExecutor::add(folly::Function<void ()>) folly/executors/CPUThreadPoolExecutor.cpp:106 (ep-engine_ep_unit_tests+0x000001710a89)
           #8 FollyExecutorPool::TaskProxy::~TaskProxy() ../kv_engine/engines/ep/src/folly_executorpool.cc:56 (ep-engine_ep_unit_tests+0x00000062da90)
           #9 void __gnu_cxx::new_allocator<FollyExecutorPool::TaskProxy>::destroy<FollyExecutorPool::TaskProxy>(FollyExecutorPool::TaskProxy*) /usr/local/include/c++/7.3.0/ext/new_allocator.h:140 (ep-engine_ep_unit_tests+0x00000062d333)
           #10 void std::allocator_traits<std::allocator<FollyExecutorPool::TaskProxy> >::destroy<FollyExecutorPool::TaskProxy>(std::allocator<FollyExecutorPool::TaskProxy>&, FollyExecutorPool::TaskProxy*) /usr/local/include/c++/7.3.0/bits/alloc_traits.h:487 (ep-engine_ep_unit_tests+0x00000062d333)
           #11 std::_Sp_counted_ptr_inplace<FollyExecutorPool::TaskProxy, std::allocator<FollyExecutorPool::TaskProxy>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /usr/local/include/c++/7.3.0/bits/shared_ptr_base.h:535 (ep-engine_ep_unit_tests+0x00000062d333)
           #12 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/local/include/c++/7.3.0/bits/shared_ptr_base.h:154 (ep-engine_ep_unit_tests+0x00000047458b)
           #13 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/local/include/c++/7.3.0/bits/shared_ptr_base.h:684 (ep-engine_ep_unit_tests+0x00000062dcc2)
           #14 std::__shared_ptr<FollyExecutorPool::TaskProxy, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/local/include/c++/7.3.0/bits/shared_ptr_base.h:1123 (ep-engine_ep_unit_tests+0x00000062dcc2)
           #15 std::shared_ptr<FollyExecutorPool::TaskProxy>::~shared_ptr() /usr/local/include/c++/7.3.0/bits/shared_ptr.h:93 (ep-engine_ep_unit_tests+0x00000062dcc2)
           #16 FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const::{lambda()#1}::~scheduleViaCPUPool() ../kv_engine/engines/ep/src/folly_executorpool.cc:149 (ep-engine_ep_unit_tests+0x00000062dcc2)
           #17 unsigned long folly::detail::function::execSmall<FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const::{lambda()#1}>(folly::detail::function::Op, folly::detail::function::Data*, folly::detail::function::Op) tlm/deps/folly.exploded/include/folly/Function.h:504 (ep-engine_ep_unit_tests+0x00000062dcc2)
           #18 folly::Function<void ()>::exec(folly::detail::function::Op, folly::detail::function::Data*, folly::detail::function::Data*) const folly/Function.h:561 (ep-engine_ep_unit_tests+0x000001752c51)
           #19 folly::Function<void ()>::~Function() folly/Function.h:693 (ep-engine_ep_unit_tests+0x000001752c51)
           #20 folly::NotificationQueue<folly::Function<void ()> >::Node::~Node() folly/io/async/NotificationQueue.h:70 (ep-engine_ep_unit_tests+0x000001752c51)
           #21 std::default_delete<folly::NotificationQueue<folly::Function<void ()> >::Node>::operator()(folly::NotificationQueue<folly::Function<void ()> >::Node*) const /usr/local/include/c++/7.3.0/bits/unique_ptr.h:78 (ep-engine_ep_unit_tests+0x000001752c51)
           #22 std::unique_ptr<folly::NotificationQueue<folly::Function<void ()> >::Node, std::default_delete<folly::NotificationQueue<folly::Function<void ()> >::Node> >::~unique_ptr() /usr/local/include/c++/7.3.0/bits/unique_ptr.h:268 (ep-engine_ep_unit_tests+0x000001752c51)
           #23 folly::NotificationQueue<folly::Function<void ()> >::Consumer::consumeMessages(bool, unsigned long*) folly/io/async/NotificationQueue.h:707 (ep-engine_ep_unit_tests+0x000001752c51)
           #24 folly::NotificationQueue<folly::Function<void ()> >::Consumer::handlerReady(unsigned short) folly/io/async/NotificationQueue.h:663 (ep-engine_ep_unit_tests+0x000001752ef9)
           #25 folly::EventHandler::libeventCallback(int, short, void*) folly/io/async/EventHandler.cpp:161 (ep-engine_ep_unit_tests+0x0000017572f0)
           #26 event_persist_closure /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1629 (libevent_core-2.1.so.7+0x000000017f39)
           #27 event_process_active_single_queue /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1688 (libevent_core-2.1.so.7+0x000000017f39)
      

      Full log: http://cv.jenkins.couchbase.com/job/kv_engine.threadsanitizer/job/master/5660

      From looking at Folly changes in this area, I suspect this is fixed by the following commit: https://github.com/facebook/folly/commit/b77dcf05d526d472c2ae1f38c9d37792c0666b72#diff-723339b1bdd178d1e36ed835a7f85481

      That is included in Folly v2020.06.29.00 upwards.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              drigby Dave Rigby (Inactive)
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty