24/180 Test #25: ep-engine-persistence-unit-tests.rocksdb .....................................***Failed 51.24 sec 08:58:44 Note: Google Test filter = *PersistToTest* 08:58:44 [==========] Running 4 tests from 1 test case. 08:58:44 [----------] Global test environment set-up. 08:58:44 [----------] 4 tests from CleanOrUnclean/PersistToTest 08:58:44 [ RUN ] CleanOrUnclean/PersistToTest.PersistedAfterShutdown/Clean 08:58:44 2019-07-23T00:57:57.902534-07:00 INFO This version of Couchbase is built without allocator hooks for accurate memory tracking 08:58:44 2019-07-23T00:57:57.903099-07:00 INFO Changing logging level to 0 08:58:44 2019-07-23T00:57:57.903345-07:00 INFO Setting SSL minimum protocol to: tlsv1 08:58:44 2019-07-23T00:57:58.007394-07:00 INFO Breakpad disabled 08:58:44 2019-07-23T00:57:59.298210-07:00 INFO This version of Couchbase is built without allocator hooks for accurate memory tracking 08:58:44 2019-07-23T00:57:59.298848-07:00 INFO Changing logging level to 0 08:58:44 2019-07-23T00:57:59.299093-07:00 INFO Setting SSL minimum protocol to: tlsv1 08:58:44 2019-07-23T00:57:59.309166-07:00 INFO Breakpad disabled 08:58:44 [ OK ] CleanOrUnclean/PersistToTest.PersistedAfterShutdown/Clean (2517 ms) 08:58:44 [ RUN ] CleanOrUnclean/PersistToTest.PersistedAfterShutdown/Unclean 08:58:44 2019-07-23T00:58:00.089327-07:00 INFO This version of Couchbase is built without allocator hooks for accurate memory tracking 08:58:44 2019-07-23T00:58:00.089913-07:00 INFO Changing logging level to 0 08:58:44 2019-07-23T00:58:00.090156-07:00 INFO Setting SSL minimum protocol to: tlsv1 08:58:44 2019-07-23T00:58:00.100015-07:00 INFO Breakpad disabled 08:58:44 unknown file: Failure 08:58:44 C++ exception with description "MemcachedConnection::backoff_executor: Timed out after waiting more than 30 seconds" thrown in SetUp(). 08:58:44 ================== 08:58:44 WARNING: ThreadSanitizer: data race (pid=3621) 08:58:44 Write of size 8 at 0x7b2c00006268 by thread T32 (mutexes: write M170146898957606912, read M1073681247781781760): 08:58:44 #0 KVStore::updateCachedVBState(Vbid, vbucket_state const&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kvstore.cc:157 (ep.so+0x000000208a22) 08:58:44 #1 RocksDBKVStore::snapshotVBucket(Vbid, vbucket_state const&, VBStatePersist) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:826 (ep.so+0x00000031b172) 08:58:44 #2 EPBucket::flushVBucket(Vbid) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_bucket.cc:518 (ep.so+0x0000001688bd) 08:58:44 #3 Flusher::flushVB() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/flusher.cc:303 (ep.so+0x0000001eb21e) 08:58:44 #4 Flusher::step(GlobalTask*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/flusher.cc:210 (ep.so+0x0000001eb8df) 08:58:44 #5 FlusherTask::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/tasks.cc:32 (ep.so+0x00000025c69c) 08:58:44 #6 ExecutorThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:153 (ep.so+0x0000001e0785) 08:58:44 #7 launch_executor_thread /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:34 (ep.so+0x0000001e0e06) 08:58:44 #8 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #9 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #10 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 Previous read of size 8 at 0x7b2c00006268 by thread T28: 08:58:44 #0 RocksDBKVStore::initScanContext(std::shared_ptr >, std::shared_ptr >, Vbid, unsigned long, DocumentFilter, ValueFilter) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:1444 (ep.so+0x00000030df40) 08:58:44 #1 Warmup::keyDumpforShard(unsigned short) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/warmup.cc:1392 (ep.so+0x0000002a386c) 08:58:44 #2 WarmupKeyDump::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/warmup.cc:375 (ep.so+0x0000002b4599) 08:58:44 #3 ExecutorThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:153 (ep.so+0x0000001e0785) 08:58:44 #4 launch_executor_thread /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:34 (ep.so+0x0000001e0e06) 08:58:44 #5 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #6 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #7 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 Location is heap block of size 168 at 0x7b2c00006250 allocated by thread T13: 08:58:44 #0 operator new(unsigned long) (libtsan.so.0+0x00000006a4d6) 08:58:44 #1 RocksDBKVStore::readVBState(VBHandle const&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:1148 (ep.so+0x000000317496) 08:58:44 #2 RocksDBKVStore::RocksDBKVStore(RocksDBKVStoreConfig&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:396 (ep.so+0x00000031af22) 08:58:44 #3 std::_MakeUniq::__single_object std::make_unique(RocksDBKVStoreConfig&) /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825 (ep.so+0x0000002065fa) 08:58:44 #4 KVStoreFactory::create(KVStoreConfig&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kvstore.cc:106 (ep.so+0x0000002065fa) 08:58:44 #5 KVShard::KVShard(unsigned short, Configuration&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kvshard.cc:56 (ep.so+0x000000234482) 08:58:44 #6 std::_MakeUniq::__single_object std::make_unique(unsigned long&, Configuration&) /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825 (ep.so+0x000000296687) 08:58:44 #7 VBucketMap::VBucketMap(Configuration&, KVBucket&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/vbucketmap.cc:29 (ep.so+0x000000296687) 08:58:44 #8 KVBucket::KVBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kv_bucket.cc:294 (ep.so+0x0000002163b1) 08:58:44 #9 EPBucket::EPBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_bucket.cc:231 (ep.so+0x0000001626b2) 08:58:44 #10 std::_MakeUniq::__single_object std::make_unique(EventuallyPersistentEngine&) /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825 (ep.so+0x000000188c7c) 08:58:44 #11 EventuallyPersistentEngine::makeBucket(Configuration&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:6059 (ep.so+0x000000188c7c) 08:58:44 #12 EventuallyPersistentEngine::initialize(char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:2060 (ep.so+0x00000019a849) 08:58:44 #13 EWB_Engine::initialize(char const*) (ewouldblock_engine.so+0x00000002a87e) 08:58:44 #14 CreateBucketThread::create() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1867 (memcached+0x0000004352d0) 08:58:44 #15 CreateBucketThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1914 (memcached+0x00000043589e) 08:58:44 #16 Couchbase::Thread::thread_entry() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:45 (libplatform_so.so.0.1.0+0x00000001d0a6) 08:58:44 #17 Couchbase::StartThreadDelegator::run(Couchbase::Thread&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:59 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #18 task_thread_main /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:65 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #19 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #20 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #21 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 Mutex M170146898957606912 is already destroyed. 08:58:44 08:58:44 Mutex M1073681247781781760 is already destroyed. 08:58:44 08:58:44 Thread T32 'mc:writer_3' (tid=3752, running) created by thread T13 at: 08:58:44 #0 pthread_create (libtsan.so.0+0x0000000282a0) 08:58:44 #1 cb_create_named_thread(unsigned long*, void (*)(void*), void*, int, char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:109 (libplatform_so.so.0.1.0+0x000000009885) 08:58:44 #2 ExecutorThread::start() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:51 (ep.so+0x0000001dff3d) 08:58:44 #3 ExecutorPool::_adjustWorkers(task_type_t, unsigned long) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:527 (ep.so+0x0000001d53a0) 08:58:44 #4 ExecutorPool::_startWorkers() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:596 (ep.so+0x0000001d58a9) 08:58:44 #5 ExecutorPool::_registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:483 (ep.so+0x0000001d32a7) 08:58:44 #6 ExecutorPool::registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:488 (ep.so+0x0000001d331c) 08:58:44 #7 KVBucket::KVBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kv_bucket.cc:315 (ep.so+0x000000216f54) 08:58:44 #8 EPBucket::EPBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_bucket.cc:231 (ep.so+0x0000001626b2) 08:58:44 #9 std::_MakeUniq::__single_object std::make_unique(EventuallyPersistentEngine&) /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825 (ep.so+0x000000188c7c) 08:58:44 #10 EventuallyPersistentEngine::makeBucket(Configuration&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:6059 (ep.so+0x000000188c7c) 08:58:44 #11 EventuallyPersistentEngine::initialize(char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:2060 (ep.so+0x00000019a849) 08:58:44 #12 EWB_Engine::initialize(char const*) (ewouldblock_engine.so+0x00000002a87e) 08:58:44 #13 CreateBucketThread::create() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1867 (memcached+0x0000004352d0) 08:58:44 #14 CreateBucketThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1914 (memcached+0x00000043589e) 08:58:44 #15 Couchbase::Thread::thread_entry() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:45 (libplatform_so.so.0.1.0+0x00000001d0a6) 08:58:44 #16 Couchbase::StartThreadDelegator::run(Couchbase::Thread&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:59 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #17 task_thread_main /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:65 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #18 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #19 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #20 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 Thread T28 'mc:reader_3' (tid=3748, running) created by thread T13 at: 08:58:44 #0 pthread_create (libtsan.so.0+0x0000000282a0) 08:58:44 #1 cb_create_named_thread(unsigned long*, void (*)(void*), void*, int, char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:109 (libplatform_so.so.0.1.0+0x000000009885) 08:58:44 #2 ExecutorThread::start() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:51 (ep.so+0x0000001dff3d) 08:58:44 #3 ExecutorPool::_adjustWorkers(task_type_t, unsigned long) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:527 (ep.so+0x0000001d53a0) 08:58:44 #4 ExecutorPool::_startWorkers() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:595 (ep.so+0x0000001d5899) 08:58:44 #5 ExecutorPool::_registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:483 (ep.so+0x0000001d32a7) 08:58:44 #6 ExecutorPool::registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:488 (ep.so+0x0000001d331c) 08:58:44 #7 KVBucket::KVBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kv_bucket.cc:315 (ep.so+0x000000216f54) 08:58:44 #8 EPBucket::EPBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_bucket.cc:231 (ep.so+0x0000001626b2) 08:58:44 #9 std::_MakeUniq::__single_object std::make_unique(EventuallyPersistentEngine&) /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825 (ep.so+0x000000188c7c) 08:58:44 #10 EventuallyPersistentEngine::makeBucket(Configuration&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:6059 (ep.so+0x000000188c7c) 08:58:44 #11 EventuallyPersistentEngine::initialize(char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:2060 (ep.so+0x00000019a849) 08:58:44 #12 EWB_Engine::initialize(char const*) (ewouldblock_engine.so+0x00000002a87e) 08:58:44 #13 CreateBucketThread::create() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1867 (memcached+0x0000004352d0) 08:58:44 #14 CreateBucketThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1914 (memcached+0x00000043589e) 08:58:44 #15 Couchbase::Thread::thread_entry() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:45 (libplatform_so.so.0.1.0+0x00000001d0a6) 08:58:44 #16 Couchbase::StartThreadDelegator::run(Couchbase::Thread&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:59 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #17 task_thread_main /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:65 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #18 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #19 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #20 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 Thread T13 'mc:bucket_add' (tid=3636, finished) created by thread T5 at: 08:58:44 #0 pthread_create (libtsan.so.0+0x0000000282a0) 08:58:44 #1 cb_create_named_thread(unsigned long*, void (*)(void*), void*, int, char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:109 (libplatform_so.so.0.1.0+0x000000009885) 08:58:44 #2 Couchbase::Thread::start() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:72 (libplatform_so.so.0.1.0+0x00000001d251) 08:58:44 #3 McbpCreateBucketTask::start() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/protocol/mcbp/create_remove_bucket_command_context.cc:41 (memcached+0x00000052c3ff) 08:58:44 #4 CreateRemoveBucketCommandContext::create() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/protocol/mcbp/create_remove_bucket_command_context.cc:85 (memcached+0x00000052c3ff) 08:58:44 #5 CreateRemoveBucketCommandContext::step() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/protocol/mcbp/create_remove_bucket_command_context.cc:128 (memcached+0x00000052d46a) 08:58:44 #6 SteppableCommandContext::drive() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/protocol/mcbp/steppable_command_context.cc:37 (memcached+0x00000054b358) 08:58:44 #7 create_remove_bucket_executor /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/mcbp_executors.cc:483 (memcached+0x000000516897) 08:58:44 #8 std::_Function_handler::_M_invoke(std::_Any_data const&, Cookie&) /usr/local/include/c++/7.3.0/bits/std_function.h:316 (memcached+0x00000051da7d) 08:58:44 #9 std::function::operator()(Cookie&) const /usr/local/include/c++/7.3.0/bits/std_function.h:706 (memcached+0x00000051de23) 08:58:44 #10 execute_client_request_packet(Cookie&, cb::mcbp::Request const&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/mcbp_executors.cc:850 (memcached+0x00000051d342) 08:58:44 #11 execute_request_packet(Cookie&, cb::mcbp::Request const&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/mcbp_executors.cc:874 (memcached+0x00000051d46a) 08:58:44 #12 Cookie::execute() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/cookie.cc:124 (memcached+0x000000501eb8) 08:58:44 #13 StateMachine::conn_execute() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/statemachine.cc:410 (memcached+0x000000550b45) 08:58:44 #14 StateMachine::execute() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/statemachine.cc:137 (memcached+0x00000055296a) 08:58:44 #15 Connection::runStateMachinery() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/connection.cc:1340 (memcached+0x0000004e9c9b) 08:58:44 #16 Connection::runEventLoop(short) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/connection.cc:1414 (memcached+0x0000004e9d2a) 08:58:44 #17 run_event_loop(Connection*, short) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/connections.cc:148 (memcached+0x0000004ffdc0) 08:58:44 #18 event_handler(int, short, void*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:848 (memcached+0x00000043b328) 08:58:44 #19 event_persist_closure /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1580 (libevent_core.so.2.1.8+0x000000017086) 08:58:44 #20 event_process_active_single_queue /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1639 (libevent_core.so.2.1.8+0x000000017086) 08:58:44 #21 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #22 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #23 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 SUMMARY: ThreadSanitizer: data race /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kvstore.cc:157 in KVStore::updateCachedVBState(Vbid, vbucket_state const&) 08:58:44 ================== 08:58:44 ================== 08:58:44 WARNING: ThreadSanitizer: data race (pid=3621) 08:58:44 Read of size 8 at 0x7b2c00006268 by thread T27: 08:58:44 #0 RocksDBKVStore::initScanContext(std::shared_ptr >, std::shared_ptr >, Vbid, unsigned long, DocumentFilter, ValueFilter) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:1444 (ep.so+0x00000030df40) 08:58:44 #1 Warmup::loadDataforShard(unsigned short) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/warmup.cc:1649 (ep.so+0x0000002a45ec) 08:58:44 #2 WarmupLoadingData::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/warmup.cc:516 (ep.so+0x0000002b490d) 08:58:44 #3 ExecutorThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:153 (ep.so+0x0000001e0785) 08:58:44 #4 launch_executor_thread /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:34 (ep.so+0x0000001e0e06) 08:58:44 #5 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #6 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #7 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 Previous write of size 8 at 0x7b2c00006268 by thread T32 (mutexes: write M170146898957606912, read M1073681247781781760): 08:58:44 #0 KVStore::updateCachedVBState(Vbid, vbucket_state const&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kvstore.cc:157 (ep.so+0x000000208a22) 08:58:44 #1 RocksDBKVStore::snapshotVBucket(Vbid, vbucket_state const&, VBStatePersist) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:826 (ep.so+0x00000031b172) 08:58:44 #2 EPBucket::flushVBucket(Vbid) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_bucket.cc:518 (ep.so+0x0000001688bd) 08:58:44 #3 Flusher::flushVB() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/flusher.cc:303 (ep.so+0x0000001eb21e) 08:58:44 #4 Flusher::step(GlobalTask*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/flusher.cc:210 (ep.so+0x0000001eb8df) 08:58:44 #5 FlusherTask::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/tasks.cc:32 (ep.so+0x00000025c69c) 08:58:44 #6 ExecutorThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:153 (ep.so+0x0000001e0785) 08:58:44 #7 launch_executor_thread /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:34 (ep.so+0x0000001e0e06) 08:58:44 #8 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #9 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #10 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 Location is heap block of size 168 at 0x7b2c00006250 allocated by thread T13: 08:58:44 #0 operator new(unsigned long) (libtsan.so.0+0x00000006a4d6) 08:58:44 #1 RocksDBKVStore::readVBState(VBHandle const&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:1148 (ep.so+0x000000317496) 08:58:44 #2 RocksDBKVStore::RocksDBKVStore(RocksDBKVStoreConfig&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:396 (ep.so+0x00000031af22) 08:58:44 #3 std::_MakeUniq::__single_object std::make_unique(RocksDBKVStoreConfig&) /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825 (ep.so+0x0000002065fa) 08:58:44 #4 KVStoreFactory::create(KVStoreConfig&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kvstore.cc:106 (ep.so+0x0000002065fa) 08:58:44 #5 KVShard::KVShard(unsigned short, Configuration&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kvshard.cc:56 (ep.so+0x000000234482) 08:58:44 #6 std::_MakeUniq::__single_object std::make_unique(unsigned long&, Configuration&) /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825 (ep.so+0x000000296687) 08:58:44 #7 VBucketMap::VBucketMap(Configuration&, KVBucket&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/vbucketmap.cc:29 (ep.so+0x000000296687) 08:58:44 #8 KVBucket::KVBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kv_bucket.cc:294 (ep.so+0x0000002163b1) 08:58:44 #9 EPBucket::EPBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_bucket.cc:231 (ep.so+0x0000001626b2) 08:58:44 #10 std::_MakeUniq::__single_object std::make_unique(EventuallyPersistentEngine&) /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825 (ep.so+0x000000188c7c) 08:58:44 #11 EventuallyPersistentEngine::makeBucket(Configuration&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:6059 (ep.so+0x000000188c7c) 08:58:44 #12 EventuallyPersistentEngine::initialize(char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:2060 (ep.so+0x00000019a849) 08:58:44 #13 EWB_Engine::initialize(char const*) (ewouldblock_engine.so+0x00000002a87e) 08:58:44 #14 CreateBucketThread::create() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1867 (memcached+0x0000004352d0) 08:58:44 #15 CreateBucketThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1914 (memcached+0x00000043589e) 08:58:44 #16 Couchbase::Thread::thread_entry() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:45 (libplatform_so.so.0.1.0+0x00000001d0a6) 08:58:44 #17 Couchbase::StartThreadDelegator::run(Couchbase::Thread&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:59 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #18 task_thread_main /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:65 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #19 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #20 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #21 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 Mutex M170146898957606912 is already destroyed. 08:58:44 08:58:44 Mutex M1073681247781781760 is already destroyed. 08:58:44 08:58:44 Thread T27 'mc:reader_2' (tid=3747, running) created by thread T13 at: 08:58:44 #0 pthread_create (libtsan.so.0+0x0000000282a0) 08:58:44 #1 cb_create_named_thread(unsigned long*, void (*)(void*), void*, int, char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:109 (libplatform_so.so.0.1.0+0x000000009885) 08:58:44 #2 ExecutorThread::start() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:51 (ep.so+0x0000001dff3d) 08:58:44 #3 ExecutorPool::_adjustWorkers(task_type_t, unsigned long) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:527 (ep.so+0x0000001d53a0) 08:58:44 #4 ExecutorPool::_startWorkers() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:595 (ep.so+0x0000001d5899) 08:58:44 #5 ExecutorPool::_registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:483 (ep.so+0x0000001d32a7) 08:58:44 #6 ExecutorPool::registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:488 (ep.so+0x0000001d331c) 08:58:44 #7 KVBucket::KVBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kv_bucket.cc:315 (ep.so+0x000000216f54) 08:58:44 #8 EPBucket::EPBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_bucket.cc:231 (ep.so+0x0000001626b2) 08:58:44 #9 std::_MakeUniq::__single_object std::make_unique(EventuallyPersistentEngine&) /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825 (ep.so+0x000000188c7c) 08:58:44 #10 EventuallyPersistentEngine::makeBucket(Configuration&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:6059 (ep.so+0x000000188c7c) 08:58:44 #11 EventuallyPersistentEngine::initialize(char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:2060 (ep.so+0x00000019a849) 08:58:44 #12 EWB_Engine::initialize(char const*) (ewouldblock_engine.so+0x00000002a87e) 08:58:44 #13 CreateBucketThread::create() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1867 (memcached+0x0000004352d0) 08:58:44 #14 CreateBucketThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1914 (memcached+0x00000043589e) 08:58:44 #15 Couchbase::Thread::thread_entry() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:45 (libplatform_so.so.0.1.0+0x00000001d0a6) 08:58:44 #16 Couchbase::StartThreadDelegator::run(Couchbase::Thread&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:59 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #17 task_thread_main /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:65 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #18 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #19 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #20 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 Thread T32 'mc:writer_3' (tid=3752, running) created by thread T13 at: 08:58:44 #0 pthread_create (libtsan.so.0+0x0000000282a0) 08:58:44 #1 cb_create_named_thread(unsigned long*, void (*)(void*), void*, int, char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:109 (libplatform_so.so.0.1.0+0x000000009885) 08:58:44 #2 ExecutorThread::start() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorthread.cc:51 (ep.so+0x0000001dff3d) 08:58:44 #3 ExecutorPool::_adjustWorkers(task_type_t, unsigned long) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:527 (ep.so+0x0000001d53a0) 08:58:44 #4 ExecutorPool::_startWorkers() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:596 (ep.so+0x0000001d58a9) 08:58:44 #5 ExecutorPool::_registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:483 (ep.so+0x0000001d32a7) 08:58:44 #6 ExecutorPool::registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/executorpool.cc:488 (ep.so+0x0000001d331c) 08:58:44 #7 KVBucket::KVBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/kv_bucket.cc:315 (ep.so+0x000000216f54) 08:58:44 #8 EPBucket::EPBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_bucket.cc:231 (ep.so+0x0000001626b2) 08:58:44 #9 std::_MakeUniq::__single_object std::make_unique(EventuallyPersistentEngine&) /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825 (ep.so+0x000000188c7c) 08:58:44 #10 EventuallyPersistentEngine::makeBucket(Configuration&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:6059 (ep.so+0x000000188c7c) 08:58:44 #11 EventuallyPersistentEngine::initialize(char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/ep_engine.cc:2060 (ep.so+0x00000019a849) 08:58:44 #12 EWB_Engine::initialize(char const*) (ewouldblock_engine.so+0x00000002a87e) 08:58:44 #13 CreateBucketThread::create() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1867 (memcached+0x0000004352d0) 08:58:44 #14 CreateBucketThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:1914 (memcached+0x00000043589e) 08:58:44 #15 Couchbase::Thread::thread_entry() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:45 (libplatform_so.so.0.1.0+0x00000001d0a6) 08:58:44 #16 Couchbase::StartThreadDelegator::run(Couchbase::Thread&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:59 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #17 task_thread_main /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:65 (libplatform_so.so.0.1.0+0x00000001d17f) 08:58:44 #18 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #19 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #20 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 Thread T13 'mc:bucket_add' (tid=3636, finished) created by thread T5 at: 08:58:44 #0 pthread_create (libtsan.so.0+0x0000000282a0) 08:58:44 #1 cb_create_named_thread(unsigned long*, void (*)(void*), void*, int, char const*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:109 (libplatform_so.so.0.1.0+0x000000009885) 08:58:44 #2 Couchbase::Thread::start() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/thread.cc:72 (libplatform_so.so.0.1.0+0x00000001d251) 08:58:44 #3 McbpCreateBucketTask::start() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/protocol/mcbp/create_remove_bucket_command_context.cc:41 (memcached+0x00000052c3ff) 08:58:44 #4 CreateRemoveBucketCommandContext::create() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/protocol/mcbp/create_remove_bucket_command_context.cc:85 (memcached+0x00000052c3ff) 08:58:44 #5 CreateRemoveBucketCommandContext::step() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/protocol/mcbp/create_remove_bucket_command_context.cc:128 (memcached+0x00000052d46a) 08:58:44 #6 SteppableCommandContext::drive() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/protocol/mcbp/steppable_command_context.cc:37 (memcached+0x00000054b358) 08:58:44 #7 create_remove_bucket_executor /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/mcbp_executors.cc:483 (memcached+0x000000516897) 08:58:44 #8 std::_Function_handler::_M_invoke(std::_Any_data const&, Cookie&) /usr/local/include/c++/7.3.0/bits/std_function.h:316 (memcached+0x00000051da7d) 08:58:44 #9 std::function::operator()(Cookie&) const /usr/local/include/c++/7.3.0/bits/std_function.h:706 (memcached+0x00000051de23) 08:58:44 #10 execute_client_request_packet(Cookie&, cb::mcbp::Request const&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/mcbp_executors.cc:850 (memcached+0x00000051d342) 08:58:44 #11 execute_request_packet(Cookie&, cb::mcbp::Request const&) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/mcbp_executors.cc:874 (memcached+0x00000051d46a) 08:58:44 #12 Cookie::execute() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/cookie.cc:124 (memcached+0x000000501eb8) 08:58:44 #13 StateMachine::conn_execute() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/statemachine.cc:410 (memcached+0x000000550b45) 08:58:44 #14 StateMachine::execute() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/statemachine.cc:137 (memcached+0x00000055296a) 08:58:44 #15 Connection::runStateMachinery() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/connection.cc:1340 (memcached+0x0000004e9c9b) 08:58:44 #16 Connection::runEventLoop(short) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/connection.cc:1414 (memcached+0x0000004e9d2a) 08:58:44 #17 run_event_loop(Connection*, short) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/connections.cc:148 (memcached+0x0000004ffdc0) 08:58:44 #18 event_handler(int, short, void*) /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/daemon/memcached.cc:848 (memcached+0x00000043b328) 08:58:44 #19 event_persist_closure /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1580 (libevent_core.so.2.1.8+0x000000017086) 08:58:44 #20 event_process_active_single_queue /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1639 (libevent_core.so.2.1.8+0x000000017086) 08:58:44 #21 CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #22 platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x000000009ad7) 08:58:44 #23 (libtsan.so.0+0x000000024feb) 08:58:44 08:58:44 SUMMARY: ThreadSanitizer: data race /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:1444 in RocksDBKVStore::initScanContext(std::shared_ptr >, std::shared_ptr >, Vbid, unsigned long, DocumentFilter, ValueFilter) 08:58:44 ================== 08:58:44 ThreadSanitizer: reported 2 warnings 08:58:44 /home/couchbase/jenkins/workspace/kv_engine-threadsanitizer-master/kv_engine/tests/testapp/testapp.cc:1004: Failure 08:58:44 Value of: ((((*(int *) &(status))) & 0xff00) >> 8) 08:58:44 Actual: 66 08:58:44 Expected: 0 08:58:44 [ FAILED ] CleanOrUnclean/PersistToTest.PersistedAfterShutdown/Unclean, where GetParam() = Unclean (37550 ms) 08:58:44 [ RUN ] CleanOrUnclean/PersistToTest.ConsistentStateAfterShutdown/Clean 08:58:44 2019-07-23T00:58:37.673861-07:00 INFO This version of Couchbase is built without allocator hooks for accurate memory tracking 08:58:44 2019-07-23T00:58:37.674448-07:00 INFO Changing logging level to 0 08:58:44 2019-07-23T00:58:37.674693-07:00 INFO Setting SSL minimum protocol to: tlsv1 08:58:44 2019-07-23T00:58:37.684654-07:00 INFO Breakpad disabled 08:58:44 2019-07-23T00:58:39.636797-07:00 INFO This version of Couchbase is built without allocator hooks for accurate memory tracking 08:58:44 2019-07-23T00:58:39.637416-07:00 INFO Changing logging level to 0 08:58:44 2019-07-23T00:58:39.637679-07:00 INFO Setting SSL minimum protocol to: tlsv1 08:58:44 2019-07-23T00:58:39.661768-07:00 INFO Breakpad disabled 08:58:44 [ OK ] CleanOrUnclean/PersistToTest.ConsistentStateAfterShutdown/Clean (3969 ms) 08:58:44 [ RUN ] CleanOrUnclean/PersistToTest.ConsistentStateAfterShutdown/Unclean 08:58:44 2019-07-23T00:58:41.621445-07:00 INFO This version of Couchbase is built without allocator hooks for accurate memory tracking 08:58:44 2019-07-23T00:58:41.622033-07:00 INFO Changing logging level to 0 08:58:44 2019-07-23T00:58:41.622287-07:00 INFO Setting SSL minimum protocol to: tlsv1 08:58:44 2019-07-23T00:58:41.632274-07:00 INFO Breakpad disabled 08:58:44 2019-07-23T00:58:42.972957-07:00 INFO This version of Couchbase is built without allocator hooks for accurate memory tracking 08:58:44 2019-07-23T00:58:42.973503-07:00 INFO Changing logging level to 0 08:58:44 2019-07-23T00:58:42.973761-07:00 INFO Setting SSL minimum protocol to: tlsv1 08:58:44 2019-07-23T00:58:42.983861-07:00 INFO Breakpad disabled 08:58:44 [ OK ] CleanOrUnclean/PersistToTest.ConsistentStateAfterShutdown/Unclean (2507 ms) 08:58:44 [----------] 4 tests from CleanOrUnclean/PersistToTest (46544 ms total) 08:58:44 08:58:44 [----------] Global test environment tear-down 08:58:44 [==========] 4 tests from 1 test case ran. (46647 ms total) 08:58:44 [ PASSED ] 3 tests. 08:58:44 [ FAILED ] 1 test, listed below: 08:58:44 [ FAILED ] CleanOrUnclean/PersistToTest.PersistedAfterShutdown/Unclean, where GetParam() = Unclean 08:58:44 08:58:44 1 FAILED TEST