Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Major
Fix Version/s: 3.1.6
Affects Version/s: 3.1.4
Component/s: couchbase-bucket
Labels:
- threadsanitizer

Triage:
Untriaged
Is this a Regression?:
Unknown
Sprint:
KV: Mar 30 - April 12, KV: April 13 - April 26

Description

[1m[31mWARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=5899)

[1m[0m  Cycle in lock order graph: [1m[35mM21372 (0x7d780000f510)[1m[0m => [1m[35mM21408 (0x7d640000f920)[1m[0m => [1m[35mM21372[1m[0m

  Mutex [1m[35mM21408[1m[0m acquired here while holding mutex [1m[35mM21372[1m[0m in [1m[36mmain thread:

[1m[0m    #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e970)

    #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x000000003870)

    #2 Mutex::acquire() /home/couchbase/couchbase/ep-engine/src/mutex.cc:31 (ep.so+0x0000001e287e)

    #3 LockHolder::lock() /home/couchbase/couchbase/ep-engine/src/locks.h:71 (ep.so+0x000000082543)

    #4 LockHolder::LockHolder(Mutex&, bool) /home/couchbase/couchbase/ep-engine/src/locks.h:48 (ep.so+0x0000000821b2)

    #5 VBucket::getSnapshotLock() /home/couchbase/couchbase/ep-engine/src/vbucket.h:212 (ep.so+0x000000104c72)

    #6 EventuallyPersistentStore::queueDirty(RCPtr<VBucket>&, StoredValue*, LockHolder*, bool, bool, bool) /home/couchbase/couchbase/ep-engine/src/ep.cc:2863 (ep.so+0x0000000d7123)

    #7 EventuallyPersistentStore::set(Item const&, void const*, bool, unsigned char) /home/couchbase/couchbase/ep-engine/src/ep.cc:683 (ep.so+0x0000000d9dfa)

    #8 EventuallyPersistentEngine::store(void const*, void*, unsigned long*, ENGINE_STORE_OPERATION, unsigned short) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:2128 (ep.so+0x00000013d538)

    #9 EvpStore(engine_interface*, void const*, void*, unsigned long*, ENGINE_STORE_OPERATION, unsigned short) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:229 (ep.so+0x00000013712d)

    #10 mock_store /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c7304)

    #11 storeCasVb11(engine_interface*, engine_interface_v1*, void const*, ENGINE_STORE_OPERATION, char const*, char const*, unsigned long, unsigned int, void**, unsigned long, unsigned short, unsigned int, unsigned char) /home/couchbase/couchbase/ep-engine/tests/ep_test_apis.cc:659 (ep_testsuite.so+0x0000000e8d17)

    #12 store(engine_interface*, engine_interface_v1*, void const*, ENGINE_STORE_OPERATION, char const*, char const*, void**, unsigned long, unsigned short, unsigned int, unsigned char) /home/couchbase/couchbase/ep-engine/tests/ep_test_apis.cc:631 (ep_testsuite.so+0x0000000e654a)

    #13 test_mb16357(engine_interface*, engine_interface_v1*) /home/couchbase/couchbase/ep-engine/tests/ep_testsuite.cc:11713 (ep_testsuite.so+0x0000000afc36)

    #14 execute_test /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c4e2f)

    #15 main crtstuff.c (engine_testapp+0x0000004c2d91)

  Mutex [1m[35mM21372[1m[0m previously acquired by the same thread here:

    #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e970)

    #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x000000003870)

    #2 Mutex::acquire() /home/couchbase/couchbase/ep-engine/src/mutex.cc:31 (ep.so+0x0000001e287e)

    #3 LockHolder::lock() /home/couchbase/couchbase/ep-engine/src/locks.h:71 (ep.so+0x000000082543)

    #4 LockHolder::LockHolder(Mutex&, bool) /home/couchbase/couchbase/ep-engine/src/locks.h:48 (ep.so+0x0000000821b2)

    #5 HashTable::getLockedBucket(int, int*) /home/couchbase/couchbase/ep-engine/src/stored-value.h:1266 (ep.so+0x00000008418a)

    #6 HashTable::getLockedBucket(std::string const&, int*) /home/couchbase/couchbase/ep-engine/src/stored-value.h:1295 (ep.so+0x00000007df9b)

    #7 EventuallyPersistentStore::set(Item const&, void const*, bool, unsigned char) /home/couchbase/couchbase/ep-engine/src/ep.cc:652 (ep.so+0x0000000d9a18)

    #8 EventuallyPersistentEngine::store(void const*, void*, unsigned long*, ENGINE_STORE_OPERATION, unsigned short) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:2128 (ep.so+0x00000013d538)

    #9 EvpStore(engine_interface*, void const*, void*, unsigned long*, ENGINE_STORE_OPERATION, unsigned short) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:229 (ep.so+0x00000013712d)

    #10 mock_store /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c7304)

    #11 storeCasVb11(engine_interface*, engine_interface_v1*, void const*, ENGINE_STORE_OPERATION, char const*, char const*, unsigned long, unsigned int, void**, unsigned long, unsigned short, unsigned int, unsigned char) /home/couchbase/couchbase/ep-engine/tests/ep_test_apis.cc:659 (ep_testsuite.so+0x0000000e8d17)

    #12 store(engine_interface*, engine_interface_v1*, void const*, ENGINE_STORE_OPERATION, char const*, char const*, void**, unsigned long, unsigned short, unsigned int, unsigned char) /home/couchbase/couchbase/ep-engine/tests/ep_test_apis.cc:631 (ep_testsuite.so+0x0000000e654a)

    #13 test_mb16357(engine_interface*, engine_interface_v1*) /home/couchbase/couchbase/ep-engine/tests/ep_testsuite.cc:11713 (ep_testsuite.so+0x0000000afc36)

    #14 execute_test /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c4e2f)

    #15 main crtstuff.c (engine_testapp+0x0000004c2d91)

  Mutex [1m[35mM21372[1m[0m acquired here while holding mutex [1m[35mM21408[1m[0m in [1m[36mthread T10:

[1m[0m    #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e970)

    #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x000000003870)

    #2 Mutex::acquire() /home/couchbase/couchbase/ep-engine/src/mutex.cc:31 (ep.so+0x0000001e287e)

    #3 LockHolder::lock() /home/couchbase/couchbase/ep-engine/src/locks.h:71 (ep.so+0x000000082543)

    #4 LockHolder::LockHolder(Mutex&, bool) /home/couchbase/couchbase/ep-engine/src/locks.h:48 (ep.so+0x0000000821b2)

    #5 HashTable::getLockedBucket(int, int*) /home/couchbase/couchbase/ep-engine/src/stored-value.h:1266 (ep.so+0x00000008418a)

    #6 HashTable::getLockedBucket(std::string const&, int*) /home/couchbase/couchbase/ep-engine/src/stored-value.h:1295 (ep.so+0x00000007df9b)

    #7 EventuallyPersistentStore::setWithMeta(Item const&, unsigned long, void const*, bool, bool, unsigned char, bool, bool) /home/couchbase/couchbase/ep-engine/src/ep.cc:1827 (ep.so+0x0000000e6b4f)

    #8 PassiveStream::commitMutation(MutationResponse*, bool) /home/couchbase/couchbase/ep-engine/src/dcp-stream.cc:1369 (ep.so+0x00000029ba21)

    #9 PassiveStream::processMutation(MutationResponse*) /home/couchbase/couchbase/ep-engine/src/dcp-stream.cc:1341 (ep.so+0x00000029a7a0)

    #10 PassiveStream::processBufferedMessages(unsigned int&) /home/couchbase/couchbase/ep-engine/src/dcp-stream.cc:1281 (ep.so+0x00000029a0f2)

    #11 DcpConsumer::processBufferedItems() /home/couchbase/couchbase/ep-engine/src/dcp-consumer.cc:599 (ep.so+0x000000262a23)

    #12 Processer::run() /home/couchbase/couchbase/ep-engine/src/dcp-consumer.cc:48 (ep.so+0x0000002625ff)

    #13 ExecutorThread::run() /home/couchbase/couchbase/ep-engine/src/executorthread.cc:110 (ep.so+0x0000001e3dd9)

    #14 launch_executor_thread(void*) /home/couchbase/couchbase/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e32ea)

    #15 platform_thread_wrap /home/couchbase/couchbase/platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000362c)

  Mutex [1m[35mM21408[1m[0m previously acquired by the same thread here:

    #0 pthread_mutex_lock <null> (engine_testapp+0x00000047e970)

    #1 cb_mutex_enter <null> (libplatform.so.0.1.0+0x000000003870)

    #2 Mutex::acquire() /home/couchbase/couchbase/ep-engine/src/mutex.cc:31 (ep.so+0x0000001e287e)

    #3 LockHolder::lock() /home/couchbase/couchbase/ep-engine/src/locks.h:71 (ep.so+0x000000082543)

    #4 LockHolder::LockHolder(Mutex&, bool) /home/couchbase/couchbase/ep-engine/src/locks.h:48 (ep.so+0x0000000821b2)

    #5 VBucket::getSnapshotLock() /home/couchbase/couchbase/ep-engine/src/vbucket.h:212 (ep.so+0x000000104c72)

    #6 PassiveStream::processMutation(MutationResponse*) /home/couchbase/couchbase/ep-engine/src/dcp-stream.cc:1340 (ep.so+0x00000029a73d)

    #7 PassiveStream::processBufferedMessages(unsigned int&) /home/couchbase/couchbase/ep-engine/src/dcp-stream.cc:1281 (ep.so+0x00000029a0f2)

    #8 DcpConsumer::processBufferedItems() /home/couchbase/couchbase/ep-engine/src/dcp-consumer.cc:599 (ep.so+0x000000262a23)

    #9 Processer::run() /home/couchbase/couchbase/ep-engine/src/dcp-consumer.cc:48 (ep.so+0x0000002625ff)

    #10 ExecutorThread::run() /home/couchbase/couchbase/ep-engine/src/executorthread.cc:110 (ep.so+0x0000001e3dd9)

    #11 launch_executor_thread(void*) /home/couchbase/couchbase/ep-engine/src/executorthread.cc:34 (ep.so+0x0000001e32ea)

    #12 platform_thread_wrap /home/couchbase/couchbase/platform/src/cb_pthreads.c (libplatform.so.0.1.0+0x00000000362c)

[1m[36m  Thread T10 (tid=5910, running) created by main thread at:

[1m[0m    #0 pthread_create <null> (engine_testapp+0x00000045fb21)

    #1 cb_create_thread <null> (libplatform.so.0.1.0+0x00000000349e)

    #2 ExecutorThread::start() /home/couchbase/couchbase/ep-engine/src/executorthread.cc:47 (ep.so+0x0000001e2fd3)

    #3 ExecutorPool::_startWorkers() /home/couchbase/couchbase/ep-engine/src/executorpool.cc:488 (ep.so+0x0000001afa04)

    #4 ExecutorPool::_registerBucket(EventuallyPersistentEngine*) /home/couchbase/couchbase/ep-engine/src/executorpool.cc:438 (ep.so+0x0000001ae966)

    #5 ExecutorPool::registerBucket(EventuallyPersistentEngine*) /home/couchbase/couchbase/ep-engine/src/executorpool.cc:443 (ep.so+0x0000001afe73)

    #6 EventuallyPersistentStore::EventuallyPersistentStore(EventuallyPersistentEngine&) /home/couchbase/couchbase/ep-engine/src/ep.cc:220 (ep.so+0x0000000d0b94)

    #7 EventuallyPersistentEngine::initialize(char const*) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:2018 (ep.so+0x00000013c675)

    #8 EvpInitialize(engine_interface*, char const*) /home/couchbase/couchbase/ep-engine/src/ep_engine.cc:135 (ep.so+0x0000001365e5)

    #9 init_engine <null> (libmcd_util.so.1.0.0+0x000000003dbb)

    #10 start_your_engines /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c3fef)

    #11 execute_test /home/couchbase/couchbase/memcached/programs/engine_testapp/engine_testapp.c (engine_testapp+0x0000004c4cd2)

    #12 main crtstuff.c (engine_testapp+0x0000004c2d91)

Attachments

Issue Links

blocks

MB-19323 3.1.6 release

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Abhi Dangeti

Reporter:: Abhi Dangeti

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 07/Apr/16 11:00 AM

Updated:: 01/Jun/16 11:53 AM

Resolved:: 25/Apr/16 10:32 AM

Gerrit Reviews

There are no open Gerrit changes

Show There are 10 closed Gerrit changes

Hide There are 10 closed Gerrit changes

MB-19113: Address false positive lock inversion seen with test_mb16357: Gerrit Review:

MB-19113: Suppress test_mb16357 when on thread sanitizer: Gerrit Review:

Merge tag 'v3.1.5' into sherlock: Gerrit Review:

Merge remote-tracking branch 'couchbase/3.0.x' into sherlock: Gerrit Review:

Merge remote-tracking branch 'couchbase/3.0.x' into sherlock: Gerrit Review:

Merge commit 'couchbase/sherlock' into 'couchbase/watson': Gerrit Review:

Merge commit 'couchbase/sherlock' into 'couchbase/watson': Gerrit Review:

Merge remote-tracking branch 'couchbase/sherlock' into 'couchbase/watson': Gerrit Review:

Merge remote-tracking branch 'couchbase/sherlock' into 'couchbase/watson': Gerrit Review:

Add unmerged-commits.py: Gerrit Review:

[ThreadSanitizer] False positive lock inversion pointed out between hash table lock and vbucket snapshot lock

Details

Description

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty