Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10913

Mutations (1 deleted items) not replicated, caused items mismatch on destination cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Blocker
    • 3.0
    • 3.0
    • XDCR
    • Security Level: Public
    • None
    • Build 3.0.0 531, Centos 32 bit

    Description

      [Jenkins]
      http://qa.hq.northscale.net/job/centos_x64--31_01--uniXDCR-P1/29/consoleFull

      [Test]
      ./testrunner -i /tmp/ubuntu-64-2.0-uniXDCR.ini GROUP=CHAIN,num_items=50000,get-cbcollect-info=True -t xdcr.uniXDCR.unidirectional.load_with_async_ops,items=100000,rdirection=unidirection,ctopology=chain,expires=60,standard_buckets=1,sasl_buckets=2,default_bucket=False,doc-ops=delete,GROUP=CHAIN;P1

      [Test Logs]
      [2014-04-18 03:54:11,221] - [rest_client:790] INFO - adding remote cluster hostname:10.3.4.177:8091 with username:password Administrator:password name:cluster1
      [2014-04-18 03:54:11,295] - [rest_client:836] INFO - starting replication type:continuous from sasl_bucket_1 to sasl_bucket_1 in the remote cluster cluster1
      [2014-04-18 03:54:11,506] - [xdcrbasetests:355] INFO - sleep for 5 secs. ...
      [2014-04-18 03:54:16,513] - [rest_client:836] INFO - starting replication type:continuous from sasl_bucket_2 to sasl_bucket_2 in the remote cluster cluster1
      [2014-04-18 03:54:16,653] - [xdcrbasetests:355] INFO - sleep for 5 secs. ...
      [2014-04-18 03:54:21,659] - [rest_client:836] INFO - starting replication type:continuous from standard_bucket_1 to standard_bucket_1 in the remote cluster cluster1
      [2014-04-18 03:54:21,824] - [xdcrbasetests:355] INFO - sleep for 5 secs. ...
      ..
      ..
      ..
      2014-04-18 04:11:16,490] - [task:420] WARNING - Not Ready: curr_items 70001 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', sasl_bucket_1 bucket
      [2014-04-18 04:11:16,542] - [task:420] WARNING - Not Ready: vb_active_curr_items 70001 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', sasl_bucket_1 bucket
      [2014-04-18 04:11:21,585] - [task:420] WARNING - Not Ready: curr_items 70001 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', sasl_bucket_1 bucket
      [2014-04-18 04:11:21,622] - [task:420] WARNING - Not Ready: vb_active_curr_items 70001 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', sasl_bucket_1 bucket

      [Test Steps] - No UPR.
      1. Create 4-4 Nodes SRC and DEST.
      2. Create 1 sasl, 2 standard buckets.
      3. Configure XMEM Uni XDCR for both buckets.
      4. Load 1M Items each bucket.
      5. Delete 30% items on each bucket.
      6. expected items on Destination = 70000 while 70001 items on destination cluster (for sasl_bucket_1). Following key loadOne97310 has mismatch. Keys shows deleted from Source but not deleted from Destination, also it shows higher sequence number of Source, showing mutations were not replicated properly.

      2014-04-18 04:18:16,419] - [task:1169] ERROR - ===== Verifying rev_ids failed for key: loadOne97310 =====
      [2014-04-18 04:18:16,419] - [task:1170] ERROR - deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:1
      [2014-04-18 04:18:16,420] - [task:1170] ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:2
      [2014-04-18 04:18:16,421] - [task:1170] ERROR - cas mismatch: Source cas:17748657181694094, Destination cas:17748657181694093, Error Count:3
      [2014-04-18 04:18:16,423] - [task:1171] ERROR - Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 17748657181694094, 'flags': 0, 'expiration': 1397818797}

      [2014-04-18 04:18:16,424] - [task:1172] ERROR - Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 17748657181694093, 'flags': 0, 'expiration': 0}

      Items were properly replicated on other 2 buckets i.e. standard_bucekt_1, sasl_bucket_2. But not sasl_bucket_1.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Aliaksey Artamonau Aliaksey Artamonau (Inactive)
              sangharsh Sangharsh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty