Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49453

[System Test] Seeing occurrences of "seq order violation" in projector logs for MAINT_STREAM

    XMLWordPrintable

Details

    Description

      Build : 7.1.0-1663
      Test : -test tests/2i/neo/test_neo_idx_clusterops_recovery.yml -scope tests/2i/neo/scope_neo_plasma_idx_dgm.yml (OSO enabled)
      Scale : 2
      Iteration : 1st

      On the following nodes, 172.23.97.215, 172.23.97.232, 172.23.97.235, 172.23.97.237, between 2021-11-09T00:22:10 and 2021-11-09T00:22:48, in the projector logs, msgs related to seq order violation were seen. 

      172.23.97.215 : projector
      		 
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2021-11-09T00:22:11.040-08:00 [Fatal] DCPT[secidx:proj-bucket3-MAINT_STREAM_TOPIC_799c07c811d6f05cef27c3dba8db6676-3190590407978436790/0] seq order violation for vb = 72, seq = 26655, opcode = DCP_SEQNO_ADVANCED, orderState = {snapStart: 26655, snapEnd 26655, snapStarted true, prevSeq: 26656, prevSeqValid: true, errCount: 1}, event = Opcode DCP_SEQNO_ADVANCED, Status SUCCESS, Datatype 0, VBucket 72, Opaque 1286, VBuuid 237329969027338, Key <ud>()</ud>, Cas 0, Seqno 26655, RevSeqno 0, Flags 0, Expiry 0, LockTime 0, Nru 0, SnapstartSeq 0, SnapendSeq 0, SnapshotType 0, FailoverLog <nil>, Error <nil>, Ctime 1636446131040248105
      		 
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2021-11-09T00:22:11.192-08:00 [Fatal] ENDP[<-(172.23.97.217:9105,f4eb)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_799c07c811d6f05cef27c3dba8db6676] seq order violation for vb = bucket3:72, seq = 26655, command = 16, orderState = {snapStart: 26655, snapEnd 26655, snapStarted true, prevSeq: 26656, prevSeqValid: true, errCount: 1}, mutation = Docidx <ud>()</ud>, Seqno 26655, Ctime 0, Uuids [0], Commands [16]
      		 
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2021-11-09T00:22:11.680-08:00 [Fatal] DCPT[secidx:proj-bucket3-MAINT_STREAM_TOPIC_799c07c811d6f05cef27c3dba8db6676-3190590407978436790/3] seq order violation for vb = 111, seq = 25450, opcode = DCP_SEQNO_ADVANCED, orderState = {snapStart: 25450, snapEnd 25450, snapStarted true, prevSeq: 25451, prevSeqValid: true, errCount: 1}, event = Opcode DCP_SEQNO_ADVANCED, Status SUCCESS, Datatype 0, VBucket 111, Opaque 1286, VBuuid 81113905331924, Key <ud>()</ud>, Cas 0, Seqno 25450, RevSeqno 0, Flags 0, Expiry 0, LockTime 0, Nru 0, SnapstartSeq 0, SnapendSeq 0, SnapshotType 0, FailoverLog <nil>, Error <nil>, Ctime 1636446131680590923
      		 
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2021-11-09T00:22:11.779-08:00 [Fatal] ENDP[<-(172.23.97.217:9105,f4eb)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_799c07c811d6f05cef27c3dba8db6676] seq order violation for vb = bucket3:111, seq = 25450, command = 16, orderState = {snapStart: 25450, snapEnd 25450, snapStarted true, prevSeq: 25451, prevSeqValid: true, errCount: 1}, mutation = Docidx <ud>()</ud>, Seqno 25450, Ctime 0, Uuids [0], Commands [16]

      This could be a regression since this issue was fixed in Cheshire Cat with MB-46466.

      Note : OSO is enabled for GSI in this test.

      Attachments

        For Gerrit Dashboard: MB-49453
        # Subject Branch Project Status CR V
        166077,5 MB-49453: Don't generate a new snapshot when ending on a new checkpoint master kv_engine Status: MERGED +2 +1

        Activity

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty