Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta-2
    • Fix Version/s: 2.0
    • Security Level: Public
    • Labels:
      None
    • Environment:
      CentOS release 5.7 (Final)
      Couchbase Server 2.0 beta through build 1950

      Description

      A testcase I'm running starts XDCR between 2 clusters (each 1 node, 4 core, 4g).

      It then runs a mix of create/update/delete operations (items=10000).

      At some point, the rate of XDCR replication drops, and it takes a long time (varying, but over still over 20 minutes in build 1950) to finish replicating all the changes. Even though the cpu/disk are idle on both nodes the entire time.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        mschoch Marty Schoch added a comment -

        Confirmed Junyi's observations. I then went back to my original ElasticSearch tests and confirmed the same thing was happening. If I started XDCR too soon after creating the ElasticSearch index, it would return bucket_not found for many of the vbuckets initially. This then causes those vbuckets to not be retried for some period of time. By waiting a sufficient amount of time after creating bucket/index before starting the XDCR transfer this problem is avoided. Replication keeps up at the expected pace.

        Show
        mschoch Marty Schoch added a comment - Confirmed Junyi's observations. I then went back to my original ElasticSearch tests and confirmed the same thing was happening. If I started XDCR too soon after creating the ElasticSearch index, it would return bucket_not found for many of the vbuckets initially. This then causes those vbuckets to not be retried for some period of time. By waiting a sufficient amount of time after creating bucket/index before starting the XDCR transfer this problem is avoided. Replication keeps up at the expected pace.
        Hide
        mschoch Marty Schoch added a comment -

        Also, I believe this should be closed as not an issue (I don't have permission)

        Show
        mschoch Marty Schoch added a comment - Also, I believe this should be closed as not an issue (I don't have permission)
        Hide
        dipti Dipti Borkar added a comment -

        Bucket creation is much better than it used to be, but it is not 100% synchronous. The ns_server timeout is 30seconds. If disk is slow, creation of all vBuckets may not complete.

        We need an example of how to check if the bucket creation is done and add it to the "Create a bucket" section here : http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-restapi-creating-buckets.html

        With a pointer to this from the XDCR note saying "Make sure the bucket on the remote cluster is created"

        assigning to Karen for documenting.

        Show
        dipti Dipti Borkar added a comment - Bucket creation is much better than it used to be, but it is not 100% synchronous. The ns_server timeout is 30seconds. If disk is slow, creation of all vBuckets may not complete. We need an example of how to check if the bucket creation is done and add it to the "Create a bucket" section here : http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-restapi-creating-buckets.html With a pointer to this from the XDCR note saying "Make sure the bucket on the remote cluster is created" assigning to Karen for documenting.
        Hide
        kzeller kzeller added a comment -

        Ok, I added this to the XDCR section and will cross reference the REST API. Here is the XDCR warning/note:

        " Before you set up replication via XDCR,
        you should make sure that a destination bucket already exists.
        If this bucket does not exist, replication
        via XDCR may not find some shards on the destination cluster;
        this will result in replication of only some data from
        the source bucket. This would then require you to retry replication
        multiple times to get a source bucket
        to be fully replicated to a destination. Therefore, do make sure
        that you check that a destination bucket exists."

        Show
        kzeller kzeller added a comment - Ok, I added this to the XDCR section and will cross reference the REST API. Here is the XDCR warning/note: " Before you set up replication via XDCR, you should make sure that a destination bucket already exists. If this bucket does not exist, replication via XDCR may not find some shards on the destination cluster; this will result in replication of only some data from the source bucket. This would then require you to retry replication multiple times to get a source bucket to be fully replicated to a destination. Therefore, do make sure that you check that a destination bucket exists."
        Hide
        kzeller kzeller added a comment -

        Added warning about ensuring bucket exists prior to replicating. Added workarounds for determine if bucket successfully created across nodes via cbstats or SDK.

        Show
        kzeller kzeller added a comment - Added warning about ensuring bucket exists prior to replicating. Added workarounds for determine if bucket successfully created across nodes via cbstats or SDK.

          People

          • Assignee:
            kzeller kzeller
            Reporter:
            mschoch Marty Schoch
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes