Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35415

TSan: Data race in Connection::enableDatatype

    XMLWordPrintable

Details

    • Triaged
    • Unknown
    • KV-Engine MH 2nd Beta

    Description

      As reported by ThreadSanitizer when running a 4-node cluster-run and performing an initial rebalance (2 replicas):

      WARNING: ThreadSanitizer: data race (pid=63544)
        Write of size 8 at 0x7b5c000706e0 by thread T13 (mutexes: write M2451):
          #0 std::_Base_bitset<1ul>::_M_do_or(std::_Base_bitset<1ul> const&) /usr/local/include/c++/7.3.0/bitset:433 (memcached+0x00000050a7e8)
          #1 std::bitset<8ul>::operator|=(std::bitset<8ul> const&) /usr/local/include/c++/7.3.0/bitset:973 (memcached+0x00000050a7e8)
          #2 Datatype::enable(cb::mcbp::Feature) /home/couchbase/couchbase/kv_engine/daemon/datatype.cc:57 (memcached+0x00000050a7e8)
          #3 Connection::enableDatatype(cb::mcbp::Feature) /home/couchbase/couchbase/kv_engine/daemon/connection.h:653 (memcached+0x00000051ea8b)
          #4 process_bin_dcp_response /home/couchbase/couchbase/kv_engine/daemon/mcbp_executors.cc:573 (memcached+0x00000051ea8b)
          #5 std::_Function_handler<void (Cookie&), void (*)(Cookie&)>::_M_invoke(std::_Any_data const&, Cookie&) /usr/local/include/c++/7.3.0/bits/std_function.h:316 (memcached+0x00000051fffd)
          #6 std::function<void (Cookie&)>::operator()(Cookie&) const /usr/local/include/c++/7.3.0/bits/std_function.h:706 (memcached+0x0000005203a3)
          #7 execute_client_response_packet /home/couchbase/couchbase/kv_engine/daemon/mcbp_executors.cc:897 (memcached+0x00000051fb35)
          #8 execute_response_packet(Cookie&, cb::mcbp::Response const&) /home/couchbase/couchbase/kv_engine/daemon/mcbp_executors.cc:946 (memcached+0x00000051ff68)
          #9 Cookie::execute() /home/couchbase/couchbase/kv_engine/daemon/cookie.cc:120 (memcached+0x0000005043cb)
          #10 StateMachine::conn_execute() /home/couchbase/couchbase/kv_engine/daemon/statemachine.cc:410 (memcached+0x0000005530c5)
          #11 StateMachine::execute() /home/couchbase/couchbase/kv_engine/daemon/statemachine.cc:137 (memcached+0x000000554eea)
          #12 Connection::runStateMachinery() /home/couchbase/couchbase/kv_engine/daemon/connection.cc:1340 (memcached+0x0000004ec21b)
          #13 Connection::runEventLoop(short) /home/couchbase/couchbase/kv_engine/daemon/connection.cc:1414 (memcached+0x0000004ec2aa)
          #14 run_event_loop(Connection*, short) /home/couchbase/couchbase/kv_engine/daemon/connections.cc:148 (memcached+0x000000502340)
          #15 event_handler(int, short, void*) /home/couchbase/couchbase/kv_engine/daemon/memcached.cc:848 (memcached+0x00000043d8a8)
          #16 event_persist_closure /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1580 (libevent_core.so.2.1.8+0x000000017856)
          #17 event_process_active_single_queue /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1639 (libevent_core.so.2.1.8+0x000000017856)
          #18 CouchbaseThread::run() /home/couchbase/couchbase/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x00000000a217)
          #19 platform_thread_wrap /home/couchbase/couchbase/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x00000000a217)
          #20 <null> <null> (libtsan.so.0+0x00000002843b)
       
        Previous read of size 8 at 0x7b5c000706e0 by thread T36:
          [failed to restore the stack]
       
        Location is heap block of size 880 at 0x7b5c00070380 allocated by thread T13:
          #0 operator new(unsigned long) <null> (libtsan.so.0+0x00000006d926)
          #1 allocate_connection /home/couchbase/couchbase/kv_engine/daemon/connections.cc:256 (memcached+0x000000502b73)
          #2 conn_new(int, ListeningPort const&, event_base*, FrontEndThread&) /home/couchbase/couchbase/kv_engine/daemon/connections.cc:165 (memcached+0x000000502e27)
          #3 dispatch_new_connections /home/couchbase/couchbase/kv_engine/daemon/thread.cc:251 (memcached+0x0000004b9e3e)
          #4 thread_libevent_process /home/couchbase/couchbase/kv_engine/daemon/thread.cc:292 (memcached+0x0000004bb5f7)
          #5 event_persist_closure /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1580 (libevent_core.so.2.1.8+0x000000017856)
          #6 event_process_active_single_queue /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1639 (libevent_core.so.2.1.8+0x000000017856)
          #7 CouchbaseThread::run() /home/couchbase/couchbase/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x00000000a217)
          #8 platform_thread_wrap /home/couchbase/couchbase/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x00000000a217)
          #9 <null> <null> (libtsan.so.0+0x00000002843b)
       
        Mutex M2451 (0x7b8c000013a8) created at:
          #0 pthread_mutex_lock <null> (libtsan.so.0+0x00000003bbbf)
          #1 __gthread_mutex_lock /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 (memcached+0x0000004b9303)
          #2 std::mutex::lock() /usr/local/include/c++/7.3.0/bits/std_mutex.h:103 (memcached+0x0000004b9303)
          #3 phosphor::MutexEventGuard<std::mutex>::MutexEventGuard(phosphor::tracepoint_info const*, phosphor::tracepoint_info const*, bool, std::mutex&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) /home/couchbase/couchbase/phosphor/include/phosphor/scoped_event_guard.h:93 (memcached+0x0000004b9303)
          #4 iterate_all_connections(std::function<void (Connection&)>) /home/couchbase/couchbase/kv_engine/daemon/thread.cc:123 (memcached+0x0000004b9303)
          #5 CccpNotificationTask::execute() /home/couchbase/couchbase/kv_engine/daemon/cccp_notification_task.cc:126 (memcached+0x000000566f0a)
          #6 Executor::run() /home/couchbase/couchbase/kv_engine/daemon/executor.cc:60 (memcached+0x00000050d1b0)
          #7 Couchbase::Thread::thread_entry() /home/couchbase/couchbase/platform/src/thread.cc:45 (libplatform_so.so.0.1.0+0x00000001d7e6)
          #8 Couchbase::StartThreadDelegator::run(Couchbase::Thread&) /home/couchbase/couchbase/platform/src/thread.cc:59 (libplatform_so.so.0.1.0+0x00000001d8bf)
          #9 task_thread_main /home/couchbase/couchbase/platform/src/thread.cc:65 (libplatform_so.so.0.1.0+0x00000001d8bf)
          #10 CouchbaseThread::run() /home/couchbase/couchbase/platform/src/cb_pthreads.cc:58 (libplatform_so.so.0.1.0+0x00000000a217)
          #11 platform_thread_wrap /home/couchbase/couchbase/platform/src/cb_pthreads.cc:71 (libplatform_so.so.0.1.0+0x00000000a217)
          #12 <null> <null> (libtsan.so.0+0x00000002843b)
       
        Thread T13 'mc:worker_7' (tid=63603, running) created by main thread at:
          #0 pthread_create <null> (libtsan.so.0+0x00000002b6f0)
          #1 cb_create_named_thread(unsigned long*, void (*)(void*), void*, int, char const*) /home/couchbase/couchbase/platform/src/cb_pthreads.cc:109 (libplatform_so.so.0.1.0+0x000000009fc5)
          #2 create_worker /home/couchbase/couchbase/kv_engine/daemon/thread.cc:111 (memcached+0x0000004b9be8)
          #3 thread_init(unsigned long, event_base*, void (*)(int, short, void*)) /home/couchbase/couchbase/kv_engine/daemon/thread.cc:460 (memcached+0x0000004bad7b)
      

      (Some output dropped as this is from babysitter.log)

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            drigby Dave Rigby (Inactive)
            drigby Dave Rigby (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty