Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42027

Data race on ThreadLocal<EventuallyPersistentEngine*> when using FollyExecutorPool

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • Cheshire-Cat
    • couchbase-bucket
    • Triaged
    • 1
    • No

    Description

      When enabling FollyExecutorPool by default (http://review.couchbase.org/c/kv_engine/+/135709), TSan reports the following race when running ./ep-engine_ep_unit_tests "--gtest_filter=DcpConnMapTest.*":

      WARNING: ThreadSanitizer: data race (pid=69324)
        Write of size 8 at 0x7b0400007b80 by main thread:
          #0 free <null> (libtsan.so.0+0x000000027806)
          #1 cb::SystemArenaMalloc::free(void*) platform/src/system_arena_malloc.cc:121 (libplatform_so.so.0.1.0+0x000000026719)
          #2 cb::SystemArenaMalloc::sized_free(void*, unsigned long) platform/src/system_arena_malloc.cc:125 (libplatform_so.so.0.1.0+0x000000026719)
          #3 cb::_ArenaMalloc<cb::SystemArenaMalloc>::sized_free(void*, unsigned long) platform/include/platform/cb_arena_malloc.h:169 (ep-engine_ep_unit_tests+0x0000014aad6b)
          #4 cb_sized_free platform/src/cb_malloc_arena.cc:71 (ep-engine_ep_unit_tests+0x0000014aad6b)
          #5 operator delete(void*, unsigned long) platform/src/global_new_replacement.cc:92 (ep-engine_ep_unit_tests+0x0000014aaa5e)
          #6 ~installer kv_engine/engines/ep/src/objectregistry.cc:42 (ep-engine_ep_unit_tests+0x0000012ff247)
          #7 <null> <null> (libtsan.so.0+0x000000024a83)
       
        Previous read of size 4 at 0x7b0400007b80 by thread T4:
          #0 ThreadLocalPosix<EventuallyPersistentEngine*>::set(EventuallyPersistentEngine* const&) kv_engine/engines/ep/src/threadlocal_posix.h:66 (ep-engine_ep_unit_tests+0x0000012ff275)
          #1 ObjectRegistry::onSwitchThread(EventuallyPersistentEngine*, bool) kv_engine/engines/ep/src/objectregistry.cc:148 (ep-engine_ep_unit_tests+0x0000012ff1be)
          #2 ~BucketAllocationGuard kv_engine/engines/ep/src/objectregistry.cc:173 (ep-engine_ep_unit_tests+0x0000012ff1be)
          #3 FollyExecutorPool::TaskProxy::~TaskProxy()::{lambda()#1}::operator()() kv_engine/engines/ep/src/folly_executorpool.cc:81 (ep-engine_ep_unit_tests+0x000001221ef8)
          #4 void folly::detail::function::FunctionTraits<void ()>::callSmall<FollyExecutorPool::TaskProxy::~TaskProxy()::{lambda()#1}>(folly::detail::function::Data&) build/tlm/deps/folly.exploded/include/folly/Function.h:387 (ep-engine_ep_unit_tests+0x000001221ef8)
          #5 folly::detail::function::FunctionTraits<void ()>::operator()() folly/Function.h:416 (ep-engine_ep_unit_tests+0x000001503b84)
          #6 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) folly/executors/ThreadPoolExecutor.cpp:98 (ep-engine_ep_unit_tests+0x000001503b84)
          #7 folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) folly/executors/CPUThreadPoolExecutor.cpp:265 (ep-engine_ep_unit_tests+0x0000014db748)
          #8 void std::__invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>(std::__invoke_memfun_deref, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&) /usr/local/include/c++/7.3.0/bits/invoke.h:73 (ep-engine_ep_unit_tests+0x00000150b69b)
      ...
       
        As if synchronized via sleep:
          #0 nanosleep <null> (libtsan.so.0+0x0000000473b1)
          #1 void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:376 (ep-engine_ep_unit_tests+0x000001215dba)
          #2 FollyExecutorPool::unregisterTaskable(Taskable&, bool) kv_engine/engines/ep/src/folly_executorpool.cc:682 (ep-engine_ep_unit_tests+0x000001215dba)
          #3 KVBucket::deinitialize() kv_engine/engines/ep/src/kv_bucket.cc:488 (ep-engine_ep_unit_tests+0x00000125227f)
          #4 EPBucket::deinitialize() kv_engine/engines/ep/src/ep_bucket.cc:330 (ep-engine_ep_unit_tests+0x0000011989a8)
          #5 ~EventuallyPersistentEngine kv_engine/engines/ep/src/ep_engine.cc:6569 (ep-engine_ep_unit_tests+0x0000011e3795)
          #6 ~SynchronousEPEngine kv_engine/engines/ep/tests/mock/mock_synchronous_ep_engine.h:43 (ep-engine_ep_unit_tests+0x0000013e8ed5)
          #7 SynchronousEPEngineDeleter::operator()(SynchronousEPEngine*) kv_engine/engines/ep/tests/mock/mock_synchronous_ep_engine.cc:135 (ep-engine_ep_unit_tests+0x0000013e8b1d)
          #8 ~unique_ptr /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/unique_ptr.h:268 (ep-engine_ep_unit_tests+0x0000005964a7)
          #9 ~DcpConnMapTest kv_engine/engines/ep/tests/module_tests/dcp_test.cc:1595 (ep-engine_ep_unit_tests+0x000000adab3a)
          #10 ~DcpConnMapTest_AvoidLockInversionInSetVBucketStateAndNewProducerExistingName_Test kv_engine/engines/ep/tests/module_tests/dcp_test.cc:2196 (ep-engine_ep_unit_tests+0x000000adad15)
          #11 testing::Test::DeleteSelf_() third_party/googletest/googletest/include/gtest/gtest.h:492 (ep-engine_ep_unit_tests+0x0000014730e3)
          #12 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) third_party/googletest/googletest/src/gtest.cc:2433 (ep-engine_ep_unit_tests+0x000001472a3c)
          #13 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) third_party/googletest/googletest/src/gtest.cc:2469 (ep-engine_ep_unit_tests+0x000001472a3c)
          #14 testing::TestInfo::Run() third_party/googletest/googletest/src/gtest.cc:2690 (ep-engine_ep_unit_tests+0x00000143fc26)
          #15 testing::TestSuite::Run() third_party/googletest/googletest/src/gtest.cc:2816 (ep-engine_ep_unit_tests+0x000001440986)
          #16 testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/googletest/src/gtest.cc:5338 (ep-engine_ep_unit_tests+0x000001456ba6)
          #17 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) third_party/googletest/googletest/src/gtest.cc:2433 (ep-engine_ep_unit_tests+0x000001473f1c)
          #18 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) third_party/googletest/googletest/src/gtest.cc:2469 (ep-engine_ep_unit_tests+0x000001473f1c)
          #19 testing::UnitTest::Run() third_party/googletest/googletest/src/gtest.cc:4925 (ep-engine_ep_unit_tests+0x0000014562b0)
          #20 RUN_ALL_TESTS() third_party/googletest/googletest/include/gtest/gtest.h:2473 (ep-engine_ep_unit_tests+0x000000b82c97)
          #21 main kv_engine/engines/ep/tests/module_tests/ep_unit_tests_main.cc:175 (ep-engine_ep_unit_tests+0x000000b82af9)
       
      SUMMARY: ThreadSanitizer: data race (install/lib/libtsan.so.0+0x27806) in free
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              drigby Dave Rigby (Inactive)
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty