Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56891

Don't block updates to DcpConnmap while "dcp" and "dcpagg" stats are executing

    XMLWordPrintable

Details

    • 0
    • KV 2023-4

    Description

      As observed in MB-56770 - during a rebalance when a client (query) was disconnecting and reconnecting to KV-Engine ~30 times per second, front-end threads which are trying to remove a closed connection can be blocked by STAT commands running on other connections - specifically "dcp" and "dcpagg" stat commands - e.g:

      This is because both "dcp" and "dcpagg" call DcpConnMap::each to print stats for every connection, and DcpConnMap::each currently takes a write (exclusive) lock on the cookieToConnMap for the duration of the call.

      We can reduce this cost by not locking the cookieToConnMap for the entire call; instead just copy a reference to each element in cookieToConnMap into a temp container, then unlock the cookieToConnMap before actually invoking the callback.

      Attachments

        1. Screenshot 2023-05-11 at 15.20.12.png
          1013 kB
          Dave Rigby
        2. Screenshot 2023-07-19 at 14.43.10.png
          1.46 MB
          Dave Rigby
        3. Screenshot 2023-07-19 at 14.44.47.png
          360 kB
          Dave Rigby
        4. Screenshot 2023-07-19 at 14.50.49.png
          197 kB
          Dave Rigby

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MB-56982 N1QL queries are failing as KV lookup couldn't finish in ~75s.

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. MB-57328 Avoid exclusive lock on DcpConnMap for "dcp" and "dcpagg" stats

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-56770 Data node rebalance slow under load

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MB-57340 Allow const iteration for DcpConnMap::each()

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MB-57400 ActiveStream::getItemsRemaining when many items are held in checkpoints can cause KV gridlock.

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              drigby Dave Rigby (Inactive)
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty