Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10437

XDCR replication rate drops almost to zero in presence of light write workload on src side

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 3.0
    • 2.5.0, 3.0
    • XDCR
    • Security Level: Public
    • Platform = Physical
      OS = CentOS 6.5
      CPU = Intel Xeon E5-2680 v2
      Memory = 256 GB
      Disk = RAID 10 SSD
    • Untriaged
    • Centos 64-bit
    • On demand.
    • No

    Description

      Setup:
      1 -> 1, unidir, non-DGM, no compaction, xdcrOptimisticReplicationThreshold = 0.
      Both capi and xmem.

      Steps:
      – Load data, wait for persistence
      – Start replication to remote cluster
      – Monitor replication rate, it should be high, something like 10-30K ops/sec
      – Start front-end workload on source server (1000 updates/sec)

      Observation:
      – Replication rate drops to very low value (about 100 ops/sec)
      – There is no obvious resource saturation
      – "erlang:statistics(run_queues)." reports zero queues
      – Nothing suspicious reported by "etop:start([

      {output, text}

      ])." (though I'm not an expert).

      Originally spotted in local test with emulated WAN effects (using netem on both sides). Confirmed in EC2 setup (east -> west).

      Attachments

        1. drop_ec2.jpeg
          drop_ec2.jpeg
          601 kB
        2. drop.jpeg
          drop.jpeg
          628 kB
        3. xdcr_rate.jpeg
          xdcr_rate.jpeg
          571 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavelpaulau Pavel Paulau (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty