Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10856

Persistence and internal replication(TAP/UPR) are broken , affects XDCR

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • Untriaged
    • Unknown
    • June 30 - July 18

    Description

      Seeing data mismatch consistently since build 588 with -

      ./testrunner -i unixdcr.ini -t xdcr.uniXDCR.unidirectional.load_with_async_ops,items=100000,rdirection=unidirection,ctopology=chain,doc-ops=update, delete,sasl_buckets=1,replication_type=xmem

      I'm also seeing some weird things.

      There are 2 buckets, both have mismatch, let us just take default.

      Source: 70403 (cbstats is consistent with this value)
      [root@cen-0401 ~]# /opt/couchbase/bin/cbstats localhost:11211 all|grep curr_items
      curr_items: 70403
      Query on source returns only 70236 (http://10.1.3.93:8092/default/_design/dev_new/_view/docs?full_set=true&descending=false&stale=false&connection_timeout=60000&limit=100000&skip=0)

      Destination : 7000, query returns 7000 items. (http://10.1.3.96:8092/default/_design/dev_new/_view/docs?full_set=true&descending=false&stale=false&connection_timeout=60000&limit=100000&skip=0)

      The diff I provide based on the query is not going to be reliable. The clusters are available for debugging.

      This Lost Deletes item count mismatch bug is seen ONLY on the source cluster's active vbuckets when:
      1. A cluster has both active and replica vbuckets.
      2. There is an active XDCR replication setup.
      3. There are at least 10000 items inserted or updated and 30% of them deleted.

      Attachments

        1. 10.1.2.12-4172014-146-diag.zip
          4.97 MB
        2. 10.1.3.93-4172014-1356-diag.zip
          30.58 MB
        3. 10.1.3.94-4172014-1359-diag.zip
          33.86 MB
        4. 10.1.3.95-4172014-143-diag.zip
          33.59 MB
        5. 10.1.3.96-4172014-148-diag.zip
          4.86 MB
        6. 10.1.3.97-4172014-1411-diag.zip
          5.03 MB
        7. 10.1.3.99-4172014-1413-diag.zip
          5.39 MB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              apiravi Aruna Piravi (Inactive)
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty