Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62120

ActiveStreamCheckpointProcessorTask::run keeps readLock on StreamContainer

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 7.2.6, 7.6.4
    • 7.6.0, 7.0.0-Beta1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.1.4, 7.0.5, 7.1.0, 7.1.1, 7.1.2, 7.2.0, 7.1.3, 7.2.1, 7.1.5, 7.2.4, 7.2.2, 7.1.6, 7.2.3, 7.2.5, 7.6.1
    • couchbase-bucket
    • Untriaged
    • 0
    • Unknown
    • March-June 24

    Description

      https://github.com/couchbase/kv_engine/blob/3071fdcc090aa6974c87c14d0340d9753eab0322/engines/ep/src/dcp/active_stream_checkpoint_processor_task.cc#L49-L53

      We've seen DCP_STREAM_REQUEST execute slowly and a suspect was the StreamContainer data structure which is used twice per request

      1) A look-up (does a stream already exist) - read-lock only
      2) An insert (new stream placed in map) - write-lock

      In a same set of logs we see a slow-runtime for ActiveStreamCheckpointProcessorTask just before a bunch of DCP_STREAM_REQUESTS are released (also logging as slow).

      The logs in question showed a CBAS workload reconnecting, which is a heavy user of the stream-id feature which means a stream-container can hold many objects.

      I suspect that alongside maybe some large checkpoints the read-lock is held for quite sometime - although in the logs we see a 17 second runtime for the stream processor which is worrying, that would have to be a huge checkpoint.

      However it's clear that the ActiveStreamCheckpointProcessorTask should not keep a read-lock whilst processing all of the streams, it should copy the ActiveStream (shared_ptr) and drop the lock then process.

      Suspect all collection releases affected, will check.

      Attachments

        Issue Links

          Activity

            People

              raghav.sk Raghav S K
              jwalker Jim Walker
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty