Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-21325

ThreadSanitizer: data race BloomFilter::addKey(char const*, unsigned long)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 4.6.0, 5.0.0
    • 4.5.1
    • None
    • None
    • Untriaged
    • Unknown

    Description

      Thread sanitiser reports a data race during `del_with_meta and drift stats` test:

      SUMMARY: ThreadSanitizer: data race /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/bloomfilter.cc:119 BloomFilter::addKey(char const*, unsigned long)
      

      This doesn't always get reported, so would appear to be some intermittent event triggering it. See below for full TSAN details:

      WARNING: ThreadSanitizer: data race (pid=341)
        Write of size 8 at 0x7d140000df80 by thread T11 (mutexes: write M20582, write M20817, write M19369):
          #0 BloomFilter::addKey(char const*, unsigned long) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/bloomfilter.cc:119 (ep.so+0x00000002453d)
          #1 VBucket::addToFilter(std::string const&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/vbucket.cc:482 (ep.so+0x000000149704)
          #2 PersistenceCallback::callback(int&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep.cc:3211 (ep.so+0x0000000b27c1)
          #3 non-virtual thunk to PersistenceCallback::callback(int&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep.cc:3167 (ep.so+0x0000000b36c2)
          #4 CouchKVStore::commitCallback(std::vector<CouchRequest*, std::allocator<CouchRequest*> >&, KVStatsCtx&, couchstore_error_t) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/couch-kvstore/couch-kvstore.cc:1898 (ep.so+0x00000017aa51)
          #5 _ZN12CouchKVStore17commit2couchstoreEP8CallbackIJ10KVStatsCtxEE /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/couch-kvstore/couch-kvstore.cc:1712 (ep.so+0x00000017208b)
          #6 _ZN12CouchKVStore6commitEP8CallbackIJ10KVStatsCtxEE /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/couch-kvstore/couch-kvstore.cc:954 (ep.so+0x000000171c92)
          #7 EventuallyPersistentStore::commit(unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep.cc:3451 (ep.so+0x0000000a29e6)
          #8 EventuallyPersistentStore::flushVBucket(unsigned short) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep.cc:3399 (ep.so+0x0000000a1935)
          #9 Flusher::flushVB() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/flusher.cc:293 (ep.so+0x0000001038ae)
          #10 Flusher::step(GlobalTask*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/flusher.cc:183 (ep.so+0x000000101f7e)
          #11 FlusherTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/tasks.cc:138 (ep.so+0x00000013ac42)
          #12 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000fd53f)
          #13 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000fd095)
          #14 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/platform/src/cb_pthreads.cc:55 (libplatform_so.so.0.1.0+0x000000005deb)
       
        Previous read of size 8 at 0x7d140000df80 by main thread (mutexes: write M15538, write M1828880534441356728):
          #0 BloomFilter::getNumOfKeysInFilter() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/bloomfilter.cc:142 (ep.so+0x0000000246a0)
          #1 VBucket::addStats(bool, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), void const*, item_eviction_policy_t) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/vbucket.cc:597 (ep.so+0x00000014a527)
          #2 EventuallyPersistentEngine::doVBucketStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), char const*, int, bool, bool)::StatVBucketVisitor::addVBStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), RCPtr<VBucket>&, EventuallyPersistentStore*, bool, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.cc:3837 (ep.so+0x0000000c69f6)
          #3 EventuallyPersistentEngine::doVBucketStats(void const*, void (*)(char const*, unsigned short, char const*, unsigned int, void const*), char const*, int, bool, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.cc:3861 (ep.so+0x0000000c664a)
          #4 EventuallyPersistentEngine::getStats(void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.cc:4774 (ep.so+0x0000000cbbb3)
          #5 EvpGetStats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.cc:231 (ep.so+0x0000000baafe)
          #6 mock_get_stats(engine_interface*, void const*, char const*, int, void (*)(char const*, unsigned short, char const*, unsigned int, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/memcached/programs/engine_testapp/engine_testapp.cc:215 (engine_testapp+0x0000004ce40d)
          #7 std::string get_stat<std::string>(engine_interface*, engine_interface_v1*, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_test_apis.cc:1140 (ep_testsuite_xdcr.so+0x000000020c34)
          #8 unsigned long get_stat<unsigned long>(engine_interface*, engine_interface_v1*, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_test_apis.cc:1179 (ep_testsuite_xdcr.so+0x00000002089b)
          #9 get_ull_stat(engine_interface*, engine_interface_v1*, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_test_apis.cc:1174 (ep_testsuite_xdcr.so+0x00000001e73e)
          #10 test_del_with_meta_and_check_drift_stats(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_testsuite_xdcr.cc:1711 (ep_testsuite_xdcr.so+0x0000000143f1)
          #11 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/memcached/programs/engine_testapp/engine_testapp.cc:1118 (engine_testapp+0x0000004cc997)
          #12 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
       
        Location is heap block of size 72 at 0x7d140000df70 allocated by main thread:
          #0 operator new(unsigned long) <null> (engine_testapp+0x00000046450d)
          #1 VBucket::createFilter(unsigned long, double) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/vbucket.cc:458 (ep.so+0x00000014950f)
          #2 EventuallyPersistentStore::setVBucketState(unsigned short, vbucket_state_t, bool, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep.cc:1337 (ep.so+0x000000092f8c)
          #3 setVBucket(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.h:709 (ep.so+0x0000000da7e4)
          #4 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.cc:1448 (ep.so+0x0000000bad98)
          #5 mock_unknown_command(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/memcached/programs/engine_testapp/engine_testapp.cc:356 (engine_testapp+0x0000004ce589)
          #6 set_vbucket_state(engine_interface*, engine_interface_v1*, unsigned short, vbucket_state_t) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_test_apis.cc:714 (ep_testsuite_xdcr.so+0x00000001d6db)
          #7 test_del_with_meta_and_check_drift_stats(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_testsuite_xdcr.cc:1627 (ep_testsuite_xdcr.so+0x00000001399c)
          #8 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/memcached/programs/engine_testapp/engine_testapp.cc:1118 (engine_testapp+0x0000004cc997)
          #9 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
       
        Mutex M20582 (0x7db40000a0c8) created at:
          #0 pthread_mutex_trylock <null> (engine_testapp+0x000000469160)
          #1 EventuallyPersistentStore::persistVBState(unsigned short) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:757 (ep.so+0x000000091dd8)
          #2 VBStatePersistTask::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/tasks.cc:160 (ep.so+0x00000013af35)
          #3 ExecutorThread::run() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/executorthread.cc:115 (ep.so+0x0000000fd53f)
          #4 launch_executor_thread(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/executorthread.cc:33 (ep.so+0x0000000fd095)
          #5 platform_thread_wrap(void*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/platform/src/cb_pthreads.cc:55 (libplatform_so.so.0.1.0+0x000000005deb)
       
        Mutex M20817 (0x7d700001af30) created at:
          #0 pthread_mutex_lock <null> (engine_testapp+0x000000486800)
          #1 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:748 (ep.so+0x000000021fd7)
          #2 EventuallyPersistentStore::setWithMeta(Item const&, unsigned long, unsigned long*, void const*, bool, bool, unsigned char, bool, ExtendedMetaData*, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/stored-value.h:1410 (ep.so+0x00000009b5bd)
          #3 EventuallyPersistentEngine::setWithMeta(void const*, protocol_binary_request_set_with_meta*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.cc:5573 (ep.so+0x0000000d4851)
          #4 processUnknownCommand(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.cc:1328 (ep.so+0x0000000da5da)
          #5 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.cc:1448 (ep.so+0x0000000bad98)
          #6 mock_unknown_command(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/memcached/programs/engine_testapp/engine_testapp.cc:356 (engine_testapp+0x0000004ce589)
          #7 set_with_meta(engine_interface*, engine_interface_v1*, char const*, unsigned long, char const*, unsigned long, unsigned int, ItemMetaData*, unsigned long, bool, unsigned char, bool, void const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_test_apis.cc:811 (ep_testsuite_xdcr.so+0x00000001ea5e)
          #8 test_del_with_meta_and_check_drift_stats(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_testsuite_xdcr.cc:1656 (ep_testsuite_xdcr.so+0x000000013c68)
          #9 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/memcached/programs/engine_testapp/engine_testapp.cc:1118 (engine_testapp+0x0000004cc997)
          #10 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
       
        Mutex M19369 (0x7d640002f728) created at:
          #0 pthread_mutex_lock <null> (engine_testapp+0x000000486800)
          #1 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:748 (ep.so+0x000000149485)
          #2 EventuallyPersistentStore::setVBucketState(unsigned short, vbucket_state_t, bool, bool) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep.cc:1337 (ep.so+0x000000092f8c)
          #3 setVBucket(EventuallyPersistentEngine*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.h:709 (ep.so+0x0000000da7e4)
          #4 EvpUnknownCommand(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.cc:1448 (ep.so+0x0000000bad98)
          #5 mock_unknown_command(engine_interface*, void const*, protocol_binary_request_header*, bool (*)(void const*, unsigned short, void const*, unsigned char, void const*, unsigned int, unsigned char, unsigned short, unsigned long, void const*)) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/memcached/programs/engine_testapp/engine_testapp.cc:356 (engine_testapp+0x0000004ce589)
          #6 set_vbucket_state(engine_interface*, engine_interface_v1*, unsigned short, vbucket_state_t) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_test_apis.cc:714 (ep_testsuite_xdcr.so+0x00000001d6db)
          #7 test_del_with_meta_and_check_drift_stats(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_testsuite_xdcr.cc:1627 (ep_testsuite_xdcr.so+0x00000001399c)
          #8 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/memcached/programs/engine_testapp/engine_testapp.cc:1118 (engine_testapp+0x0000004cc997)
          #9 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
       
        Mutex M15538 (0x7f3dbc5ff260) created at:
          #0 pthread_mutex_lock <null> (engine_testapp+0x000000486800)
          #1 std::mutex::lock() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/x86_64-linux-gnu/c++/4.9/bits/gthr-default.h:748 (ep_testsuite_xdcr.so+0x000000020bac)
          #2 wait_for_warmup_complete(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_test_apis.cc:1179 (ep_testsuite_xdcr.so+0x000000022254)
          #3 test_setup(engine_interface*, engine_interface_v1*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/tests/ep_testsuite_common.cc:148 (ep_testsuite_xdcr.so+0x00000000595b)
          #4 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/memcached/programs/engine_testapp/engine_testapp.cc:1113 (engine_testapp+0x0000004cc950)
          #5 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
       
        Mutex M1828880534441356728 is already destroyed.
       
        Thread T11 'mc:writer_10' (tid=357, running) created by main thread at:
          #0 pthread_create <null> (engine_testapp+0x0000004679b1)
          #1 cb_create_named_thread /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/platform/src/cb_pthreads.cc:105 (libplatform_so.so.0.1.0+0x000000005c45)
          #2 ExecutorThread::start() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/executorthread.cc:46 (ep.so+0x0000000fca37)
          #3 ExecutorPool::_startWorkers() /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/executorpool.cc:512 (ep.so+0x0000000f622a)
          #4 ExecutorPool::_registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/executorpool.cc:476 (ep.so+0x0000000f580f)
          #5 ExecutorPool::registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/executorpool.cc:481 (ep.so+0x0000000f58be)
          #6 EventuallyPersistentStore::EventuallyPersistentStore(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep.cc:301 (ep.so+0x000000087c02)
          #7 EventuallyPersistentEngine::initialize(char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.cc:2208 (ep.so+0x0000000bce7e)
          #8 EvpInitialize(engine_interface*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/ep_engine.cc:143 (ep.so+0x0000000ba7b8)
          #9 init_engine_instance <null> (libmcd_util.so.1.0.0+0x000000005acb)
          #10 create_bucket(bool, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/memcached/programs/engine_testapp/engine_testapp.cc:961 (engine_testapp+0x0000004cd836)
          #11 execute_test(test, char const*, char const*) /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/memcached/programs/engine_testapp/engine_testapp.cc:1111 (engine_testapp+0x0000004cc8fa)
          #12 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
       
      SUMMARY: ThreadSanitizer: data race /home/couchbase/jenkins/workspace/ep-engine-threadsanitizer-watson/ep-engine/src/bloomfilter.cc:119 BloomFilter::addKey(char const*, unsigned long)
      ==================
      Running [0030/0032]: test del_with_meta and drift stats (couchstore)...(5 sec) UNKNOWN
      

      Log file: http://cv.jenkins.couchbase.com/job/ep-engine-threadsanitizer-watson/1326/consoleFull#155605765861882284-c5b1-40af-8076-4f8cb2d12fb1

      Attachments

        For Gerrit Dashboard: MB-21325
        # Subject Branch Project Status CR V

        Activity

          People

            drigby Dave Rigby (Inactive)
            drigby Dave Rigby (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty