Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10975

[TAP xmem XDCR] seqno/cas mismatch between source and destination for 7 expired (deleted) items

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 3.0
    • 3.0
    • couchbase-bucket, XDCR
    • Security Level: Public
    • None
    • Build 3.0.0-614

    Description

      [Job]
      http://qa.hq.northscale.net/job/centos_x64--31_01--uniXDCR-P1/32/consoleFull

      [Test]
      ./testrunner -i /tmp/ubuntu-64-2.0-uniXDCR.ini GROUP=CHAIN,num_items=50000,get-cbcollect-info=True -t xdcr.uniXDCR.unidirectional.load_with_async_ops,items=100000,rdirection=unidirection,ctopology=chain,expires=60,doc-ops=update-delete,replication_type=xmem,GROUP=CHAIN;P0;xmem

      [Test Error] -> There are errors for 7 keys

      [2014-04-26 10:35:57,973] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne29446 =====
      [2014-04-26 10:35:57,974] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:1
      [2014-04-26 10:35:57,974] - [task:1183] ERROR - cas mismatch: Source cas:8592894078905619, Destination cas:8592894078905620, Error Count:2
      [2014-04-26 10:35:57,975] - [task:1184] ERROR - Source meta data:

      {'deleted': 1, 'seqno': 3, 'cas': 8592894078905619, 'flags': 0, 'expiration': 1398533140}

      [2014-04-26 10:35:57,975] - [task:1185] ERROR - Dest meta data:

      {'deleted': 1, 'seqno': 4, 'cas': 8592894078905620, 'flags': 0, 'expiration': 1398533141}

      [2014-04-26 10:36:09,680] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne29578 =====
      [2014-04-26 10:36:09,680] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:3
      [2014-04-26 10:36:09,681] - [task:1183] ERROR - cas mismatch: Source cas:18463067062267569, Destination cas:18463067062267570, Error Count:4
      [2014-04-26 10:36:09,682] - [task:1184] ERROR - Source meta data:

      {'deleted': 1, 'seqno': 3, 'cas': 18463067062267569, 'flags': 0, 'expiration': 1398533142}

      [2014-04-26 10:36:09,683] - [task:1185] ERROR - Dest meta data:

      {'deleted': 1, 'seqno': 4, 'cas': 18463067062267570, 'flags': 0, 'expiration': 1398533141}

      [2014-04-26 10:36:19,864] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne29451 =====
      [2014-04-26 10:36:19,864] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:5
      [2014-04-26 10:36:19,864] - [task:1183] ERROR - cas mismatch: Source cas:8950894458988185, Destination cas:8950894458988186, Error Count:6
      [2014-04-26 10:36:19,864] - [task:1184] ERROR - Source meta data:

      {'deleted': 1, 'seqno': 3, 'cas': 8950894458988185, 'flags': 0, 'expiration': 1398533145}

      [2014-04-26 10:36:19,865] - [task:1185] ERROR - Dest meta data:

      {'deleted': 1, 'seqno': 4, 'cas': 8950894458988186, 'flags': 0, 'expiration': 1398533140}

      [2014-04-26 10:36:41,956] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne412 =====
      [2014-04-26 10:36:41,956] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:7
      [2014-04-26 10:36:41,957] - [task:1183] ERROR - cas mismatch: Source cas:18462956912623569, Destination cas:18462956912623570, Error Count:8
      [2014-04-26 10:36:41,957] - [task:1184] ERROR - Source meta data:

      {'deleted': 1, 'seqno': 3, 'cas': 18462956912623569, 'flags': 0, 'expiration': 1398533025}

      [2014-04-26 10:36:41,958] - [task:1185] ERROR - Dest meta data:

      {'deleted': 1, 'seqno': 4, 'cas': 18462956912623570, 'flags': 0, 'expiration': 1398532988}

      [2014-04-26 10:37:27,692] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne29556 =====
      [2014-04-26 10:37:27,693] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:9
      [2014-04-26 10:37:27,693] - [task:1183] ERROR - cas mismatch: Source cas:8592894387719619, Destination cas:8592894387719620, Error Count:10
      [2014-04-26 10:37:27,694] - [task:1184] ERROR - Source meta data:

      {'deleted': 1, 'seqno': 3, 'cas': 8592894387719619, 'flags': 0, 'expiration': 1398533148}

      [2014-04-26 10:37:27,694] - [task:1185] ERROR - Dest meta data:

      {'deleted': 1, 'seqno': 4, 'cas': 8592894387719620, 'flags': 0, 'expiration': 1398533141}

      [2014-04-26 10:38:18,620] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne582 =====
      [2014-04-26 10:38:18,620] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:11
      [2014-04-26 10:38:18,621] - [task:1183] ERROR - cas mismatch: Source cas:18462957665894569, Destination cas:18462957665894570, Error Count:12
      [2014-04-26 10:38:18,621] - [task:1184] ERROR - Source meta data:

      {'deleted': 1, 'seqno': 3, 'cas': 18462957665894569, 'flags': 0, 'expiration': 1398533025}

      [2014-04-26 10:38:18,623] - [task:1185] ERROR - Dest meta data:

      {'deleted': 1, 'seqno': 4, 'cas': 18462957665894570, 'flags': 0, 'expiration': 1398532988}

      [2014-04-26 10:38:22,957] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne28739 =====
      [2014-04-26 10:38:22,957] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:13
      [2014-04-26 10:38:22,958] - [task:1183] ERROR - cas mismatch: Source cas:18463064250570569, Destination cas:18463064250570570, Error Count:14
      [2014-04-26 10:38:22,959] - [task:1184] ERROR - Source meta data:

      {'deleted': 1, 'seqno': 3, 'cas': 18463064250570569, 'flags': 0, 'expiration': 1398533140}

      [2014-04-26 10:38:22,960] - [task:1185] ERROR - Dest meta data:

      {'deleted': 1, 'seqno': 4, 'cas': 18463064250570570, 'flags': 0, 'expiration': 1398533141}

      [Test Steps]
      1. Setup 4-4 Nodes SRC and DEST clusters with xmem XDCR.
      2. Load 1M items on SRC nodes. (1 sasl buckets, 1 default bucket)
      3. Perform 30% deletes.
      4. Updates 30% items with expiration time to 60 seconds. Wait for expiration time.
      5. There are two errors
      a) Items count mismtach on destination side Expected 40000, Actual: 39993 (7 keys are missing as mentioned above).
      b) seq and cas mismatch for 7 keys between Source and Destination
      c) Why sequence number raised to 3 or 4, since updation is done only once? As per my understanding, it starts from 1 and increase by 1 for each update. Effectively it should be 2.

      Data files and logs are uploaded.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-10856 Persistence and internal replication(TAP/UPR) are broken , affects XDCR

          • Test Blocker - QE-related tests can't proceed
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sriram Sriram Ganesan (Inactive)
              sangharsh Sangharsh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty