Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35051

[Durability]: cbstats seqno, persisted snap_start-stop mismatch after CRUD failure

    XMLWordPrintable

Details

    Description

      Build: 6.5.0 - 3748

      Scenario:

      1. 2 nodes, couchbase-bucket (replica=1)
      2. Load 10K docs with durability=MAJORITY (succeeds)
      3. Stop Memcached on node-2 and wait for 10seconds (Just to make sure mc is stopped)
      4. Perform Create/delete/read/update ops. (each ops affecting 5 docs)
      5. Wait to receive ambiguous / timeout exception for all operation we do. (Seeing this as well)
      6. Resume Memcached on node-2 again
      7. Validate cbstats' vbucket-seq_no to make sure no mutation went-in due to the memcached stopped on node-2

      Observation:

      Following vbucket_seqno stats mismatch taken between  step-3 and at step-7

      Mismatch in vb #104
       - abs_high_seqno, Expected: 28,  Actual: 30
       - last_persisted_seqno, Expected: 28,  Actual: 30
       - last_persisted_snap_end, Expected: 28,  Actual: 30
       - last_persisted_snap_start, Expected: 21,  Actual: 29
       - high_prepared_seqno, Expected: 27,  Actual: 29
       - high_seqno, Expected: 28,  Actual: 30
      Mismatch in vb #330
       - abs_high_seqno, Expected: 30,  Actual: 32
       - last_persisted_seqno, Expected: 30,  Actual: 32
       - last_persisted_snap_end, Expected: 30,  Actual: 32
       - last_persisted_snap_start, Expected: 23,  Actual: 31
       - high_prepared_seqno, Expected: 29,  Actual: 31
       - high_seqno, Expected: 30,  Actual: 32
      Mismatch in vb #453
       - abs_high_seqno, Expected: 10,  Actual: 12
       - last_persisted_seqno, Expected: 10,  Actual: 12
       - last_persisted_snap_end, Expected: 10,  Actual: 12
       - last_persisted_snap_start, Expected: 9,  Actual: 11
       - high_prepared_seqno, Expected: 9,  Actual: 11
       - high_seqno, Expected: 10,  Actual: 12
      Mismatch in vb #204
       - abs_high_seqno, Expected: 12,  Actual: 14
       - last_persisted_seqno, Expected: 12,  Actual: 14
       - last_persisted_snap_end, Expected: 12,  Actual: 14
       - last_persisted_snap_start, Expected: 11,  Actual: 13
       - high_prepared_seqno, Expected: 11,  Actual: 13
       - high_seqno, Expected: 12,  Actual: 14
      Mismatch in vb #166
       - abs_high_seqno, Expected: 16,  Actual: 18
       - last_persisted_seqno, Expected: 16,  Actual: 18
       - last_persisted_snap_end, Expected: 16,  Actual: 18
       - last_persisted_snap_start, Expected: 12,  Actual: 17
       - high_prepared_seqno, Expected: 15,  Actual: 17
       - high_seqno, Expected: 16,  Actual: 18
      Mismatch in vb #353
       - abs_high_seqno, Expected: 28,  Actual: 30
       - last_persisted_seqno, Expected: 28,  Actual: 30
       - last_persisted_snap_end, Expected: 28,  Actual: 30
       - last_persisted_snap_start, Expected: 21,  Actual: 29
       - high_prepared_seqno, Expected: 27,  Actual: 29
       - high_seqno, Expected: 28,  Actual: 30
      Mismatch in vb #41
       - abs_high_seqno, Expected: 30,  Actual: 32
       - last_persisted_seqno, Expected: 30,  Actual: 32
       - last_persisted_snap_end, Expected: 30,  Actual: 32
       - last_persisted_snap_start, Expected: 23,  Actual: 31
       - high_prepared_seqno, Expected: 29,  Actual: 31
       - high_seqno, Expected: 30,  Actual: 32
      Mismatch in vb #288
       - abs_high_seqno, Expected: 30,  Actual: 32
       - last_persisted_seqno, Expected: 30,  Actual: 32
       - last_persisted_snap_end, Expected: 30,  Actual: 32
       - last_persisted_snap_start, Expected: 23,  Actual: 31
       - high_prepared_seqno, Expected: 29,  Actual: 31
       - high_seqno, Expected: 30,  Actual: 32

      Expected behavior:

      Since all CRUDs failed between this steps, expects seqno also should remain the same after roll backs

      TAF test case:

      epengine.durability_failures.TimeoutTests.test_timeout_with_crud_failures,nodes_init=2,replicas=1,num_items=10000,sdk_timeout=60,simulate_error=stop_memcached,durability=MAJORITY

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty