Details
-
Bug
-
Resolution: Fixed
-
Major
-
4.5.0
-
None
-
Ubuntu 12.04 (commit validation)
-
Untriaged
-
Unknown
-
KV: April 13 - April 26
Description
The following ThreadSanitizer warning was initially seen during commit-validation for a patch adding an atomic unordered map (http://cv.jenkins.couchbase.com/job/ep-engine-threadsanitizer-master/394/consoleFull)
SUMMARY: ThreadSanitizer: data race ??:0 operator delete(void*)
|
However I have subsequently reproduced it without that patch, using current tip of couchbase/master (1b03555):
(first build with TSan)...
TSAN_OPTIONS=halt_on_error=true ./ep-engine_dcp_test --gtest_repeat=-1
|
...
|
WARNING: ThreadSanitizer: data race (pid=13347)
|
Write of size 8 at 0x7d100000c5d0 by thread T21 (mutexes: write M7104781):
|
#0 operator delete(void*) <null> (ep-engine_dcp_test+0x00000047319b)
|
#1 std::string::_Rep::_M_dispose(std::allocator<char> const&) /build/buildd/gcc-5-5.1.0/build/x86_64-linux-gnu/libstdc++-v3/include/bits/basic_string.h:2642 (libstdc++.so.6+0x0000000ceba7)
|
#2 CouchKVStore::snapshotVBucket(unsigned short, vbucket_state&, VBStatePersist) /home/couchbase/couchbase/ep-engine/src/couch-kvstore/couch-kvstore.cc:1003 (ep-engine_dcp_test+0x000000662b5e)
|
#3 EventuallyPersistentStore::persistVBState(Priority const&, unsigned short) /home/couchbase/couchbase/ep-engine/src/ep.cc:1358 (ep-engine_dcp_test+0x000000554a8f)
|
#4 VBStatePersistTask::run() /home/couchbase/couchbase/ep-engine/src/tasks.cc:88 (ep-engine_dcp_test+0x000000605a16)
|
#5 ExecutorThread::run() /home/couchbase/couchbase/ep-engine/src/executorthread.cc:115 (ep-engine_dcp_test+0x0000005bf281)
|
#6 launch_executor_thread(void*) /home/couchbase/couchbase/ep-engine/src/executorthread.cc:33 (ep-engine_dcp_test+0x0000005bee35)
|
#7 platform_thread_wrap(void*) /home/couchbase/couchbase/platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000558b)
|
|
|
Previous atomic write of size 4 at 0x7d100000c5d0 by thread T22:
|
#0 __tsan_atomic32_fetch_add <null> (ep-engine_dcp_test+0x0000004bcf86)
|
#1 std::string::_Rep::_M_dispose(std::allocator<char> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/5.1.0/../../../../include/c++/5.1.0/ext/atomicity.h:81 (ep-engine_dcp_test+0x00000057b6de)
|
#2 ~VBucketStateVisitor /usr/bin/../lib/gcc/x86_64-linux-gnu/5.1.0/../../../../include/c++/5.1.0/bits/stl_tree.h:859 (ep-engine_dcp_test+0x000000553ee2)
|
#3 VBSnapshotTask::run() /home/couchbase/couchbase/ep-engine/src/tasks.cc:69 (ep-engine_dcp_test+0x000000605756)
|
#4 ExecutorThread::run() /home/couchbase/couchbase/ep-engine/src/executorthread.cc:115 (ep-engine_dcp_test+0x0000005bf281)
|
#5 launch_executor_thread(void*) /home/couchbase/couchbase/ep-engine/src/executorthread.cc:33 (ep-engine_dcp_test+0x0000005bee35)
|
#6 platform_thread_wrap(void*) /home/couchbase/couchbase/platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000558b)
|
|
|
Mutex M7104781 (0x7db40000a000) created at:
|
#0 pthread_mutex_trylock <null> (ep-engine_dcp_test+0x000000477840)
|
#1 LockHolder /usr/bin/../lib/gcc/x86_64-linux-gnu/5.1.0/../../../../include/x86_64-linux-gnu/c++/5.1.0/bits/gthr-default.h:757 (ep-engine_dcp_test+0x000000553bc0)
|
#2 VBSnapshotTask::run() /home/couchbase/couchbase/ep-engine/src/tasks.cc:69 (ep-engine_dcp_test+0x000000605756)
|
#3 ExecutorThread::run() /home/couchbase/couchbase/ep-engine/src/executorthread.cc:115 (ep-engine_dcp_test+0x0000005bf281)
|
#4 launch_executor_thread(void*) /home/couchbase/couchbase/ep-engine/src/executorthread.cc:33 (ep-engine_dcp_test+0x0000005bee35)
|
#5 platform_thread_wrap(void*) /home/couchbase/couchbase/platform/src/cb_pthreads.cc:54 (libplatform.so.0.1.0+0x00000000558b)
|
|
|
Thread T21 'mc:writer_4' (tid=25161, running) created by main thread at:
|
#0 pthread_create <null> (ep-engine_dcp_test+0x000000476091)
|
#1 cb_create_named_thread /home/couchbase/couchbase/platform/src/cb_pthreads.cc:104 (libplatform.so.0.1.0+0x0000000053e5)
|
#2 ExecutorThread::start() /home/couchbase/couchbase/ep-engine/src/executorthread.cc:46 (ep-engine_dcp_test+0x0000005be8e8)
|
#3 ExecutorPool::_startWorkers() /home/couchbase/couchbase/ep-engine/src/executorpool.cc:523 (ep-engine_dcp_test+0x0000005b8363)
|
#4 ExecutorPool::_registerTaskable(Taskable&) /home/couchbase/couchbase/ep-engine/src/executorpool.cc:487 (ep-engine_dcp_test+0x0000005b787f)
|
#5 ExecutorPool::registerTaskable(Taskable&) /home/couchbase/couchbase/ep-engine/src/executorpool.cc:492 (ep-engine_dcp_test+0x0000005b9f6e)
|
#6 EventuallyPersistentStore::EventuallyPersistentStore(EventuallyPersistentEngine&) /home/couchbase/couchbase/ep-engine/src/ep.cc:393 (ep-engine_dcp_test+0x00000054a552)
|
#7 EventuallyPersistentEngine::initialize(char const*) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:2117 (ep-engine_dcp_test+0x00000057fce4)
|
#8 DCPTest::SetUp() /home/couchbase/couchbase/ep-engine/tests/module_tests/dcp_test.cc:105 (ep-engine_dcp_test+0x0000004e42ea)
|
#9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:2402 (ep-engine_dcp_test+0x0000006d4be2)
|
#10 testing::Test::Run() /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:2470 (ep-engine_dcp_test+0x0000006a1b2f)
|
#11 testing::TestInfo::Run() /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:2656 (ep-engine_dcp_test+0x0000006a3e2b)
|
#12 testing::TestCase::Run() /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:2774 (ep-engine_dcp_test+0x0000006a4afa)
|
#13 testing::internal::UnitTestImpl::RunAllTests() /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:4649 (ep-engine_dcp_test+0x0000006b5823)
|
#14 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:2402 (ep-engine_dcp_test+0x0000006d5c72)
|
#15 testing::UnitTest::Run() /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:4257 (ep-engine_dcp_test+0x0000006b4e00)
|
#16 main /home/couchbase/couchbase/googletest/googletest/include/gtest/gtest.h:2237 (ep-engine_dcp_test+0x0000004e2900)
|
|
|
Thread T22 'mc:writer_5' (tid=25162, running) created by main thread at:
|
#0 pthread_create <null> (ep-engine_dcp_test+0x000000476091)
|
#1 cb_create_named_thread /home/couchbase/couchbase/platform/src/cb_pthreads.cc:104 (libplatform.so.0.1.0+0x0000000053e5)
|
#2 ExecutorThread::start() /home/couchbase/couchbase/ep-engine/src/executorthread.cc:46 (ep-engine_dcp_test+0x0000005be8e8)
|
#3 ExecutorPool::_startWorkers() /home/couchbase/couchbase/ep-engine/src/executorpool.cc:523 (ep-engine_dcp_test+0x0000005b8363)
|
#4 ExecutorPool::_registerTaskable(Taskable&) /home/couchbase/couchbase/ep-engine/src/executorpool.cc:487 (ep-engine_dcp_test+0x0000005b787f)
|
#5 ExecutorPool::registerTaskable(Taskable&) /home/couchbase/couchbase/ep-engine/src/executorpool.cc:492 (ep-engine_dcp_test+0x0000005b9f6e)
|
#6 EventuallyPersistentStore::EventuallyPersistentStore(EventuallyPersistentEngine&) /home/couchbase/couchbase/ep-engine/src/ep.cc:393 (ep-engine_dcp_test+0x00000054a552)
|
#7 EventuallyPersistentEngine::initialize(char const*) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:2117 (ep-engine_dcp_test+0x00000057fce4)
|
#8 DCPTest::SetUp() /home/couchbase/couchbase/ep-engine/tests/module_tests/dcp_test.cc:105 (ep-engine_dcp_test+0x0000004e42ea)
|
#9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:2402 (ep-engine_dcp_test+0x0000006d4be2)
|
#10 testing::Test::Run() /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:2470 (ep-engine_dcp_test+0x0000006a1b2f)
|
#11 testing::TestInfo::Run() /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:2656 (ep-engine_dcp_test+0x0000006a3e2b)
|
#12 testing::TestCase::Run() /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:2774 (ep-engine_dcp_test+0x0000006a4afa)
|
#13 testing::internal::UnitTestImpl::RunAllTests() /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:4649 (ep-engine_dcp_test+0x0000006b5823)
|
#14 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:2402 (ep-engine_dcp_test+0x0000006d5c72)
|
#15 testing::UnitTest::Run() /home/couchbase/couchbase/googletest/googletest/src/gtest.cc:4257 (ep-engine_dcp_test+0x0000006b4e00)
|
#16 main /home/couchbase/couchbase/googletest/googletest/include/gtest/gtest.h:2237 (ep-engine_dcp_test+0x0000004e2900)
|
|
|
SUMMARY: ThreadSanitizer: data race ??:0 operator delete(void*)
|
It took a variable number of iterations to hit the issue - least I needed was 1, most I needed was 34 - but it should show up within a couple of minutes.
Assigning to Dan as he's been working in this area (and authored the test which exposes it).
Attachments
| For Gerrit Dashboard: MB-19183 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 62897,3 | MB-19183: Clean-up connection handlers in DCP tests | master | ep-engine | Status: ABANDONED | +2 | +1 |
| 62952,2 | MB-19183: Clean-up connection handlers in DCP tests | watson | ep-engine | Status: MERGED | +2 | +1 |
| 63170,2 | Merge remote-tracking branch 'couchbase/watson' into master | master | ep-engine | Status: ABANDONED | +2 | -1 |
| 63427,1 | Merge remote-tracking branch 'couchbase/watson' into master | master | ep-engine | Status: MERGED | +2 | +1 |