Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59480

Projector/DCP - buffer acknowledgment is happening at the limit leading to frequent logging and connection pauses.

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      Projector DCP is setting a 2MB flow-control window and then only acking when it has processed 2MB - from server side this will lead to frequent pausing and penalise healthy nodes who could be consuming faster. A secondary affect is that memcached.log will be filled with messages about unpausing.

      Notifying paused connection now that DcpProducer::BufferLog is no longer

      I think this behaviour is observable in perf tests, although there's no noticeable performance issue, e.g.

      https://showfast.sc.couchbase.com/#/runs/secondary_1M_multiple_moi_thr_nyx/7.2.2-6316

      All nodes are logging "Notify paused"

      > find . -name memcached.log | xargs grep -c "Notif"
      		 
      ./cbcollect_info_ns_1@172.23.97.7_20230916-102234/memcached.log:604
      		 
      ./cbcollect_info_ns_1@172.23.97.6_20230916-102234/memcached.log:606
      		 
      ./cbcollect_info_ns_1@172.23.97.4_20230916-102234/memcached.log:606
      		 
      ./cbcollect_info_ns_1@172.23.97.3_20230916-102235/memcached.log:606
      		 
      ./cbcollect_info_ns_1@172.23.97.5_20230916-102234/memcached.log:606

      Attachments

        1. DCP ack at limit.png
          337 kB
          Jim Walker
        2. Screenshot 2023-11-20 at 22.37.34.png
          50 kB
          Varun Velamuri

        Issue Links

          backports to

          Bug - A problem which impairs or prevents the functions of the product. MB-59762 [BP to 7.2.4]- Projector/DCP - buffer acknowledgment is happening at the limit leading to frequent logging and connection pauses.

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-59480
          # Subject Branch Project Status CR V
          201237,3 MB-59480 Change bufferAck scheme for mutation queue unstable indexing Status: MERGED +2 +1
          201541,3 MB-59762 Change bufferAck scheme for mutation queue neo indexing Status: MERGED +2 +1

          Activity

            People

              varun.velamuri Varun Velamuri
              jwalker Jim Walker
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty