Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6643

Very slow replication rate( less than 100 on average) with 2 unidirectional replications between 2 clusters.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: 2.0-beta-2
    • Fix Version/s: 2.0-beta-2
    • Component/s: XDCR
    • Security Level: Public
    • Environment:
      2.0-1717
      1024 vbuckets

      2 unidirectional replications on 2 clusters.

      Centos

      6G, 4 core VMs

      Description

      • Setup two 3 node clusters.
      • Load 3M items on bucket1, cluster1 and 3M on bucket2, cluster2 [No expires on any load]
        Example
        -nohup lib/perf_engines/mcsoda.py localhost:23201 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=1 ratio-sets=1 min-value-size=256 max-items=1000000 exit-after-creates=1 prefix=a_one
        -nohup lib/perf_engines/mcsoda.py localhost:23202 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=1 ratio-sets=1 min-value-size=256 max-items=1000000 exit-after-creates=1 prefix=a_two
        -nohup lib/perf_engines/mcsoda.py localhost:23203 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=1 ratio-sets=1 min-value-size=256 max-items=1000000 exit-after-creates=1 prefix=a_three

      Start unidirectional from cluster1 bucket1 to cluster2 bucket1
      Start unidirectional from cluster2 bucket2 to cluster1 bucket2

      Keep load running on cluster1 -new load with deletes/expires
      lib/perf_engines/mcsoda.py localhost:23203 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=1 ratio-sets=1 ratio-deletes=0.02 ratio-expirations=0.05 expirations=1200 min-value-size=256 max-items=1000000 exit-after-creates=1 prefix=a_four&

      Observing very slow replication rate on cluster1 and close to 100 percent CPU used.

      Replication rate on cluster 1 is between 10 - 115
      Replication rate on cluster2 - is between 1-2k

      Adding screenshots from cluster1.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        ketaki Ketaki Gangal added a comment -

        Over a period of time, xdc gets is much higher but xdc sets is low.

        Seeing 9K/sec Xdc ops and gets 9K/sec and creates <100 per sec on the cluster.

        Show
        ketaki Ketaki Gangal added a comment - Over a period of time, xdc gets is much higher but xdc sets is low. Seeing 9K/sec Xdc ops and gets 9K/sec and creates <100 per sec on the cluster.
        Hide
        junyi Junyi Xie (Inactive) added a comment -

        Unable to tell a lot from the screen shot. Please post more information about two clusters, e.g, the complete screenshot with XDC sections (Replication and Destination).

        Also, you load data from your local machine from the commands above. Can you please load the data from another machine in your test?

        Show
        junyi Junyi Xie (Inactive) added a comment - Unable to tell a lot from the screen shot. Please post more information about two clusters, e.g, the complete screenshot with XDC sections (Replication and Destination). Also, you load data from your local machine from the commands above. Can you please load the data from another machine in your test?
        Hide
        ketaki Ketaki Gangal added a comment -

        The clusters are still running here

        http://10.3.121.32:8091/index.html#sec=analytics&statsBucket=%2Fpools%2Fdefault%2Fbuckets%2Fbucket1%3Fbucket_uuid%3Db0a2f01b5f787f2f85f7978a42d99a6b&zoom=zoom_minute&graph=ep_ops_create

        http://10.3.121.38:8091/index.html#sec=analytics&statsBucket=%2Fpools%2Fdefault%2Fbuckets%2Fbucket1%3Fbucket_uuid%3Db0a2f01b5f787f2f85f7978a42d99a6b&zoom=zoom_minute&graph=ep_ops_create

        Loading commands are put up on the bug description above. I am not loading data from my local machine, it is from different clients.

        The screenshot was from one of the relevant clusters with XDC ops/sec and XDC create/sec rate. I will add the other one as well.

        Show
        ketaki Ketaki Gangal added a comment - The clusters are still running here http://10.3.121.32:8091/index.html#sec=analytics&statsBucket=%2Fpools%2Fdefault%2Fbuckets%2Fbucket1%3Fbucket_uuid%3Db0a2f01b5f787f2f85f7978a42d99a6b&zoom=zoom_minute&graph=ep_ops_create http://10.3.121.38:8091/index.html#sec=analytics&statsBucket=%2Fpools%2Fdefault%2Fbuckets%2Fbucket1%3Fbucket_uuid%3Db0a2f01b5f787f2f85f7978a42d99a6b&zoom=zoom_minute&graph=ep_ops_create Loading commands are put up on the bug description above. I am not loading data from my local machine, it is from different clients. The screenshot was from one of the relevant clusters with XDC ops/sec and XDC create/sec rate. I will add the other one as well.
        Hide
        junyi Junyi Xie (Inactive) added a comment -

        This sounds the same issue as MB-6662 because you have expired items in workload, and you see extremely high getMeta ops but very low setWithMeta/DeleteWithMeta.

        MB-6662 has been fixed by recent commits.

        Show
        junyi Junyi Xie (Inactive) added a comment - This sounds the same issue as MB-6662 because you have expired items in workload, and you see extremely high getMeta ops but very low setWithMeta/DeleteWithMeta. MB-6662 has been fixed by recent commits.
        Hide
        junyi Junyi Xie (Inactive) added a comment -
        Show
        junyi Junyi Xie (Inactive) added a comment - MB-6662

          People

          • Assignee:
            junyi Junyi Xie (Inactive)
            Reporter:
            ketaki Ketaki Gangal
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes