Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35552

TSan: data race on DurabilityMonitor::Position::lastWriteSeqno

    XMLWordPrintable

Details

    • Triaged
    • Unknown

    Description

      As seen during work on MB-35458, there is a race on DurabilityMonitor::Position::lastWriteSeqno as detailed below. Essentially it is written under the DM::State lock (exclusive), but is read without any lock by the DurabilityTimeoutVisitor task via ActiveDurabilityMonitor::processCompletedSyncWriteQueue:

      WARNING: ThreadSanitizer: data race (pid=12218)
        Write of size 8 at 0x7b240001e000 by thread T13 (mutexes: write M1073399927423374248, read M704104603360363264, write M873270703586071984):
          #0 Monotonic<...>::operator=(long const&) kv_engine/engines/ep/src/monotonic.h:115 (ep.so+0x0000002a4b2c)
          #1 ActiveDurabilityMonitor::State::advanceAndAckForPosition(...) kv_engine/engines/ep/src/durability/active_durability_monitor.cc:879 (ep.so+0x00000029a078)
          #2 ActiveDurabilityMonitor::State::advanceNodePosition(...) kv_engine/engines/ep/src/durability/active_durability_monitor.cc:850 (ep.so+0x000000299e6f)
          #3 ActiveDurabilityMonitor::State::updateHighPreparedSeqno(...) kv_engine/engines/ep/src/durability/active_durability_monitor.cc:1639 (ep.so+0x00000029e866)
          #4 ActiveDurabilityMonitor::checkForCommit() kv_engine/engines/ep/src/durability/active_durability_monitor.cc:1654 (ep.so+0x00000029e977)
          #5 VBucket::notifyActiveDMOfLocalSyncWrite() kv_engine/engines/ep/src/vbucket.cc:981 (ep.so+0x00000047cf54)
          #6 KVBucket::set(...)>) kv_engine/engines/ep/src/kv_bucket.cc:701 (ep.so+0x0000003ef1ad)
          #7 EventuallyPersistentEngine::storeIfInner(...) kv_engine/engines/ep/src/ep_engine.cc:2403 (ep.so+0x0000003195e2)
          #8 EventuallyPersistentEngine::store_if(...) kv_engine/engines/ep/src/ep_engine.cc:377 (ep.so+0x00000030f081)
          #9 bucket_store_if(...) kv_engine/daemon/protocol/mcbp/engine_wrapper.cc:139 (memcached+0x00000055d7bf)
          #10 MutationCommandContext::storeItem() kv_engine/daemon/protocol/mcbp/mutation_context.cc:296 (memcached+0x00000066524c)
          #11 MutationCommandContext::step() kv_engine/daemon/protocol/mcbp/mutation_context.cc:63 (memcached+0x000000663fc5)
          #12 SteppableCommandContext::drive() kv_engine/daemon/protocol/mcbp/steppable_command_context.cc:37 (memcached+0x00000067c19a)
          #13 add_set_replace_executor kv_engine/daemon/mcbp_executors.cc:170 (memcached+0x00000063173d)
          #14 set_executor kv_engine/daemon/mcbp_executors.cc:178 (memcached+0x0000006317d3)
          ...
       
        Previous read of size 8 at 0x7b240001e000 by thread T38 (mutexes: read M704104603360363264, write M877493051575179176):
          #0 Monotonic<...>::operator long() const kv_engine/engines/ep/src/monotonic.h:123 (ep.so+0x0000002a3922)
          #1 DurabilityMonitor::ReplicationChain::hasAcked(...) const kv_engine/engines/ep/src/durability/durability_monitor_impl.cc:348 (ep.so+0x0000002bab4d)
          #2 DurabilityMonitor::SyncWrite::isSatisfied() const kv_engine/engines/ep/src/durability/durability_monitor_impl.cc:141 (ep.so+0x0000002b960e)
          #3 ActiveDurabilityMonitor::processCompletedSyncWriteQueue() kv_engine/engines/ep/src/durability/active_durability_monitor.cc:743 (ep.so+0x000000299317)
          #4 ActiveDurabilityMonitor::processTimeout(...) kv_engine/engines/ep/src/durability/active_durability_monitor.cc:643 (ep.so+0x0000002988ae)
          #5 VBucket::processDurabilityTimeout(...) kv_engine/engines/ep/src/vbucket.cc:706 (ep.so+0x00000047b65f)
          #6 DurabilityTimeoutVisitor::visitBucket(...) kv_engine/engines/ep/src/durability_timeout_task.cc:55 (ep.so+0x0000002cc286)
          #7 VBCBAdaptor::run() kv_engine/engines/ep/src/kv_bucket.cc:2368 (ep.so+0x0000003fa98f)
          #8 ExecutorThread::run() kv_engine/engines/ep/src/executorthread.cc:153 (ep.so+0x0000003a7be7)
          ...
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ben.huddleston Ben Huddleston
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty