Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60891

XDCR should not ask for all DCP stats from KV

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • Morpheus
    • 7.6.0
    • XDCR
    • None
    • Untriaged
    • 0
    • No

    Description

      I've noticed XDCR appears to collect the `"dcp"` stats group from KV. It appears the output is retained in memory.

      https://github.com/couchbase/goxdcr/blob/c82c4d94cc4c595b237c12db4d77cf73b42d949d/service_impl/bucketTopologyService.go#L227

      This stat group contains detailed stats for every DCP stream (multiple streams per producer) in KV, and can generate a lot of data and stall the connection. We've previously seen timeouts of 60s during cbcollects, and 100s MiB of output (see linked MBs).

      The stat group supports a filter, passed in as the value of the request, used to ignore streams not belonging to a specific user, something like `{"user": "@goxdcr"}`. Ns_server has started doing that recently: https://review.couchbase.org/c/ns_server/+/200841/7/src/dcp_replicator.erl#298

      Additionally, as of 7.6.0, XDCR can specify "stream_format" to ignore the per-vBucket stream stats, if not required, as they are the bulk of the output.

      https://github.com/couchbase/kv_engine/blob/master/docs/BinaryProtocol.md#statdcp

      My worry here is that if XDCR asks for that STAT group and there is a very large output to generate, XDCR will observe a very large payload on that connection, and potentially delay other operations in a non-obvious way (theory).

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              neil.huang Neil Huang
              vesko.karaganev Vesko Karaganev
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty