Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50427

MagmaKVStore::saveDocs vb:540 WriteDocs failed. Status:Invalid: kvstore-540/rev-000000001: seqno (2) should always increase monotonically between write batches (164) numCommits:0

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Steps to repro:

      1. Create a 4 node cluster(replicas=3)
      2. Created 5 million items(doc size =1024 bytes)
      3. Stopped persistence on one of the nodes
      4. Started doc ops(create:expiry:update) on node(where persistence is stopped)
      5. Killed memcached on node(where persistence is stopped) to trigger rollback on other nodes
      6. During rollback keep killing memcached on all the nodes(which are rolling back)
      7. Repeat above steps multiple times.

      QE Test

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.91973.ini bucket_storage=magma,rerun=false,GROUP=P2;crash,randomize_value=True,bucket_eviction_policy=fullEviction,get-cbcollect-info=True,infra_log_level=debug,log_level=debug,enable_dp=false,collect_pcaps=True,get-cbcollect-info=True,autoCompactionDefined=true,upgrade_version=7.1.0-2071 -t storage.magma.magma_rollback.MagmaRollbackTests.test_crash_during_rollback,num_items=5000000,doc_size=1024,nodes_init=4,num_rollbacks=5,vbuckets=1024,rollback_items=500,replicas=3,key_size=12,init_loading=False,doc_ops=expiry:create:update:delete,num_collections=0,process_concurrency=2,collections_for_rollback=1,threads_to_use=1000,target_active_nodes=3,GROUP=P2;crash'
      

      Note: 165 items couldn't be persisted in 1200s.
      Found these messages in the logs:
      2022-01-17T05:50:22.448585-08:00 CRITICAL (default) MagmaKVStore::saveDocs vb:540 WriteDocs failed. Status:Invalid: kvstore-540/rev-000000001: seqno (2) should always increase monotonically between write batches (164) numCommits:0

      Ankush Sharma, can you please update the test steps?
      Here is the test details: http://qe-jenkins.sc.couchbase.com/job/test_suite_executor-TAF/6951/artifact/job_logs/testrunner-22-Jan-17_04-37-09/test_2/

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty