Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11642

Intra-replication falling far behind under moderate-heavy workload when XDCR is enabled on small-scale hardwares (eg. AWS instances)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 4.1.1
    • 3.0-Beta, 3.0, 3.0.1, 3.0.2
    • couchbase-bucket
    • Security Level: Public
    • Platform = Physical
      OS = CentOS 6.5
      CPU = Intel Xeon E5-2630 (24 vCPU)
      Memory = 64 GB
      Disk = RAID 10 HDD
    • Triaged
    • Yes

    Description

      Running the "standard sales" demo that puts a 50/50 workload of about 80k ops/sec across 4 nodes of m1.xlarge, 1 bucket 1 replica.

      The "intra-cluster replication" value grows into the many k's.

      This is a value that our users look rather closely at to determine the "safety" of their replication status. A reasonable number on 2.x has always been below 1k but I think we need to reproduce and set appropriate baselines for ourselves with 3.0.

      Assigning to Pavel as it falls into the performance area and we would likely be best served if this behavior was reproduced and tracked.

      Attachments

        1. ep_dcp_replica_items_remaining.png
          78 kB
          Pavel Paulau
        2. ep_upr_replica_items_remaining.png
          45 kB
          Pavel Paulau
        3. latency_observe.png
          101 kB
          Pavel Paulau
        4. OPS_during_rebalance.png
          198 kB
          Thomas Anderson
        5. Repl_items_remaining_after_rebalance.png
          184 kB
          Thomas Anderson
        6. Repl_items_remaining_before_rebalance.png
          226 kB
          Thomas Anderson
        7. Repl_items_remaining_during_rebalance.png
          239 kB
          Thomas Anderson
        8. Repl_items_remaining_start_of_rebalance.png
          252 kB
          Thomas Anderson
        9. Screen Shot 2014-07-15 at 11.47.19 AM.png
          41 kB
          Perry Krug
        10. Screen Shot 2014-08-13 at 10.21.24 AM.png
          44 kB
          Perry Krug

        Issue Links

          blocks

          Task - A task that needs to be done. MB-17211 4.1.1 Minor Release

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-11643 Incoming workload suffers when XDCR enabled

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MB-11640 Provide ability to set the DCP stream priority (low/high)

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MB-11675 20-30% performance degradation on append-heavy workload compared to 2.5.1

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-11984 Intra-cluster replication slows down during intensive ejection

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          mentioned in

          Page Loading...

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ericcooper Eric Cooper (Inactive)
              perry Perry Krug
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 47h
                  47h

                  Gerrit Reviews

                    There are no open Gerrit changes

                    PagerDuty