Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57958

[Backport MB-57399 to 7.2.1] [System Test] - Seeing occurrences of "seq order violation" in projector logs for #INIT_STREAM_TOPIC

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 0
    • No

    Description

      Steps to Repro
      Started 7.2 Longevity on latest 7.2.1 build

      ./sequoia -client 172.23.104.27:2375 -provider file:centos_pine.yml -test tests/integration/7.2/test_7.2.yml -scope tests/integration/7.2/scope_7.2_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.2.1-5788 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
      

      I saw following Fatal messages in projector logs.

      172.23.99.25 : projector

      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 3, command = 9, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 1}, mutation = Docidx <ud>()</ud>, Seqno 3, Ctime 1693965, Uuids [0], Commands [9]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124203, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 2}, mutation = Docidx <ud>(doc_10428)</ud>, Seqno 124203, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124283, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 3}, mutation = Docidx <ud>(doc_13609)</ud>, Seqno 124283, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124241, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 4}, mutation = Docidx <ud>(doc_13799)</ud>, Seqno 124241, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124229, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 5}, mutation = Docidx <ud>(doc_15539)</ud>, Seqno 124229, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124273, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 6}, mutation = Docidx <ud>(doc_16688)</ud>, Seqno 124273, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124278, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 7}, mutation = Docidx <ud>(doc_16718)</ud>, Seqno 124278, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 123993, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 8}, mutation = Docidx <ud>(doc_1961)</ud>, Seqno 123993, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124302, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 9}, mutation = Docidx <ud>(doc_20218)</ud>, Seqno 124302, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124309, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 10}, mutation = Docidx <ud>(doc_20388)</ud>, Seqno 124309, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124113, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 11}, mutation = Docidx <ud>(doc_3244)</ud>, Seqno 124113, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124034, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 12}, mutation = Docidx <ud>(doc_3536)</ud>, Seqno 124034, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124081, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 13}, mutation = Docidx <ud>(doc_4870)</ud>, Seqno 124081, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124129, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 14}, mutation = Docidx <ud>(doc_5174)</ud>, Seqno 124129, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124070, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 15}, mutation = Docidx <ud>(doc_5606)</ud>, Seqno 124070, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124074, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 16}, mutation = Docidx <ud>(doc_5796)</ud>, Seqno 124074, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124136, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 17}, mutation = Docidx <ud>(doc_6355)</ud>, Seqno 124136, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124138, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 18}, mutation = Docidx <ud>(doc_6427)</ud>, Seqno 124138, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124175, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 19}, mutation = Docidx <ud>(doc_8200)</ud>, Seqno 124175, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124186, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 20}, mutation = Docidx <ud>(doc_8390)</ud>, Seqno 124186, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 124188, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 21}, mutation = Docidx <ud>(doc_8572)</ud>, Seqno 124188, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:28.458-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:945, seq = 123984, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 22}, mutation = Docidx <ud>(doc_879)</ud>, Seqno 123984, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 3, command = 9, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 1}, mutation = Docidx <ud>()</ud>, Seqno 3, Ctime 1549015, Uuids [0], Commands [9]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200459, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 2}, mutation = Docidx <ud>(doc_10962)</ud>, Seqno 200459, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200472, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 3}, mutation = Docidx <ud>(doc_11066)</ud>, Seqno 200472, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200560, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 4}, mutation = Docidx <ud>(doc_11684)</ud>, Seqno 200560, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200506, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 5}, mutation = Docidx <ud>(doc_11714)</ud>, Seqno 200506, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200517, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 6}, mutation = Docidx <ud>(doc_12247)</ud>, Seqno 200517, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200498, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 7}, mutation = Docidx <ud>(doc_12535)</ud>, Seqno 200498, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 201337, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 8}, mutation = Docidx <ud>(doc_1359)</ud>, Seqno 201337, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200520, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 9}, mutation = Docidx <ud>(doc_14177)</ud>, Seqno 200520, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200569, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 10}, mutation = Docidx <ud>(doc_14605)</ud>, Seqno 200569, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200581, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 11}, mutation = Docidx <ud>(doc_14795)</ud>, Seqno 200581, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200618, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 12}, mutation = Docidx <ud>(doc_15873)</ud>, Seqno 200618, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200620, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 13}, mutation = Docidx <ud>(doc_17356)</ud>, Seqno 200620, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200624, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 14}, mutation = Docidx <ud>(doc_17424)</ud>, Seqno 200624, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200698, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 15}, mutation = Docidx <ud>(doc_19203)</ud>, Seqno 200698, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200636, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 16}, mutation = Docidx <ud>(doc_19393)</ud>, Seqno 200636, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200645, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 17}, mutation = Docidx <ud>(doc_19571)</ud>, Seqno 200645, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200674, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 18}, mutation = Docidx <ud>(doc_20820)</ud>, Seqno 200674, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 201353, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 19}, mutation = Docidx <ud>(doc_2178)</ud>, Seqno 201353, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 201306, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 20}, mutation = Docidx <ud>(doc_241)</ud>, Seqno 201306, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 201369, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 21}, mutation = Docidx <ud>(doc_4248)</ud>, Seqno 201369, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 201314, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 22}, mutation = Docidx <ud>(doc_533)</ud>, Seqno 201314, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200441, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 23}, mutation = Docidx <ud>(doc_7069)</ud>, Seqno 200441, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:23:36.314-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:976, seq = 200414, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 24}, mutation = Docidx <ud>(doc_8838)</ud>, Seqno 200414, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:24:22.108-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:958, seq = 3, command = 9, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 1}, mutation = Docidx <ud>()</ud>, Seqno 3, Ctime 1779869, Uuids [0], Commands [9]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:24:22.108-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:958, seq = 121935, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 2}, mutation = Docidx <ud>(doc_1170)</ud>, Seqno 121935, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:24:22.108-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:958, seq = 121950, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 3}, mutation = Docidx <ud>(doc_1602)</ud>, Seqno 121950, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:24:22.108-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:958, seq = 121956, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 4}, mutation = Docidx <ud>(doc_1792)</ud>, Seqno 121956, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:24:22.108-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:958, seq = 121313, command = 1, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 5}, mutation = Docidx <ud>(doc_19758)</ud>, Seqno 121313, Ctime 0, Uuids [6614192364350659055], Commands [1]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:24:22.108-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:958, seq = 121973, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 6}, mutation = Docidx <ud>(doc_2351)</ud>, Seqno 121973, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:24:22.108-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:958, seq = 121975, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 7}, mutation = Docidx <ud>(doc_2423)</ud>, Seqno 121975, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:24:22.108-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:958, seq = 122006, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 8}, mutation = Docidx <ud>(doc_4061)</ud>, Seqno 122006, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:24:22.108-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:958, seq = 122016, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 9}, mutation = Docidx <ud>(doc_4683)</ud>, Seqno 122016, Ctime 0, Uuids [6614192364350659055], Commands [2]
      /opt/couchbase/var/lib/couchbase/logs/projector.log:2023-06-14T20:24:22.108-07:00 [Fatal] ENDP[<-(172.23.104.137:9103,a11f)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_bcd1475004a6f3279cf516a1acfccbe5] seq order violation for vb = bucket7:scope_0:coll_0:958, seq = 122017, command = 2, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 10}, mutation = Docidx <ud>(doc_4713)</ud>, Seqno 122017, Ctime 0, Uuids [6614192364350659055], Commands [2]
      

      cbcollect_info attached. This is the first run we had on 7.2.1

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-57958
          # Subject Branch Project Status CR V

          Activity

            People

              varun.velamuri Varun Velamuri
              amit.kulkarni Amit Kulkarni
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty