Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44832

Prepare seqno of abort in backfill snapshot may be lower than snap start seqno

    XMLWordPrintable

Details

    Description

      On Node 172.23.121.115:

      ['2021-03-09T00:53:34.840274-08:00 ERROR 1267: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":0,"cas":0,"datatype":"raw","extlen":0,"keylen":0,"magic":"ClientResponse","opaque":7,"opcode":"DCP_ABORT","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default, vb:143, state:backfilling\n', '2021-03-09T00:53:34.840325-08:00 ERROR 1267: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":0,"cas":0,"datatype":"raw","extlen":0,"keylen":0,"magic":"ClientResponse","opaque":7,"opcode":"DCP_ABORT","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default, vb:143, state:backfilling\n', '2021-03-09T00:53:34.840353-08:00 ERROR 1267: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":0,"cas":0,"datatype":"raw","extlen":0,"keylen":0,"magic":"ClientResponse","opaque":7,"opcode":"DCP_ABORT","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default, vb:143, state:backfilling\n', '2021-03-09T00:53:34.840382-08:00 ERROR 1267: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":0,"cas":0,"datatype":"raw","extlen":0,"keylen":0,"magic":"ClientResponse","opaque":7,"opcode":"DCP_ABORT","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default, vb:143, state:backfilling\n', '2021-03-09T00:53:34.840409-08:00 ERROR 1267: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":0,"cas":0,"datatype":"raw","extlen":0,"keylen":0,"magic":"ClientResponse","opaque":7,"opcode":"DCP_ABORT","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default, vb:143, state:backfilling\n', '2021-03-09T00:53:34.840438-08:00 ERROR 1267: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":0,"cas":0,"datatype":"raw","extlen":0,"keylen":0,"magic":"ClientResponse","opaque":7,"opcode":"DCP_ABORT","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default, vb:143, state:backfilling\n', '2021-03-09T00:53:34.840464-08:00 ERROR 1267: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":0,"cas":0,"datatype":"raw","extlen":0,"keylen":0,"magic":"ClientResponse","opaque":7,"opcode":"DCP_ABORT","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.121.115->ns_1@172.23.121.124:default, vb:143, state:backfilling\n']
      

      QE Test:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/test_job_magma.ini  -t magma.magma_crash_recovery.MagmaCrashTests.test_crash_during_ops,num_items=5000000,infra_log_level=debug,log_level=debug,rerun=False,skip_cleanup=true,doc_size=1024,randomize_value=False,nodes_init=20,num_crashes=20,sdk_timeout=60,bucket_storage=magma,replicas=1,vbuckets=1024,graceful=False,doc_ops=create:update:delete:expiry,wait_warmup=False,maxttl=10,get-cbcollect-info=True,multiplier=100,process_concurrency=5,durability=MAJORITY,stop_server_on_crash=false'
      

      Note: Not seen this issue with couchstore

      Attachments

        For Gerrit Dashboard: MB-44832
        # Subject Branch Project Status CR V

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty