Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
-
1
-
No
Description
Build : 7.0.0-5219
Test : -test tests/2i/cheshirecat/test_idx_clusterops_cheshire_cat_moi.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat_moi.yml (MOI)
Scale : 2
Iteration : 1st
Seeing occurrences of the keyword "seq order violation" on node 172.23.97.232 in the projector logs.
/opt/couchbase/var/lib/couchbase/logs/projector.log.5.gz:2021-05-22T00:47:33.103-07:00 [Fatal] ENDP[<-(172.23.107.3:9103,5ea3)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_c8f8e9eb7afb7781acb6269f4910760c] seq order violation for vb = bucket3:scope_2:coll_18:450, seq = 61160, command = 3, orderState = {snapStart: 61125, snapEnd 61125, snapStarted true, prevSeq: 61125, prevSeqValid: true, errCount: 1}, mutation = Docidx <ud>(doc_21246)</ud>, Seqno 61160, Ctime 0, Uuids [13588792686132421685], Commands [3]
|
/opt/couchbase/var/lib/couchbase/logs/projector.log.5.gz:2021-05-22T00:47:35.271-07:00 [Fatal] ENDP[<-(172.23.107.3:9103,5ea3)<-127.0.0.1:8091 #INIT_STREAM_TOPIC_c8f8e9eb7afb7781acb6269f4910760c] seq order violation for vb = bucket3:scope_2:coll_18:452, seq = 60356, command = 1, orderState = {snapStart: 60312, snapEnd 60312, snapStarted true, prevSeq: 60312, prevSeqValid: true, errCount: 1}, mutation = Docidx <ud>(doc_21203)</ud>, Seqno 60356, Ctime 205538582, Uuids [13588792686132421685], Commands [1]
|
Later in the test, in the 2nd iteration, it was seen on 3 nodes :
172.23.97.215 : projector
|
/opt/couchbase/var/lib/couchbase/logs/projector.log.1.gz:2021-05-22T06:45:10.534-07:00 [Fatal] ENDP[<-(172.23.107.5:9105,2ae5)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_ddcd87eb9e0baf13d2277c336d4a1f58] seq order violation for vb = bucket4:83, seq = 1199, command = 10, orderState = {snapStart: 1198, snapEnd 1198, snapStarted true, prevSeq: 1198, prevSeqValid: true, errCount: 1}, mutation = Docidx <ud>()</ud>, Seqno 1199, Ctime 0, Uuids [0], Commands [10]
|
/opt/couchbase/var/lib/couchbase/logs/projector.log.1.gz:2021-05-22T06:45:10.535-07:00 [Fatal] ENDP[<-(172.23.107.5:9105,2ae5)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_ddcd87eb9e0baf13d2277c336d4a1f58] seq order violation for vb = bucket4:134, seq = 1199, command = 10, orderState = {snapStart: 1198, snapEnd 1198, snapStarted true, prevSeq: 1198, prevSeqValid: true, errCount: 1}, mutation = Docidx <ud>()</ud>, Seqno 1199, Ctime 0, Uuids [0], Commands [10]
|
|
|
172.23.97.232 : projector
|
/opt/couchbase/var/lib/couchbase/logs/projector.log.1.gz:2021-05-22T06:35:08.434-07:00 [Fatal] ENDP[<-(172.23.107.5:9105,9139)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_ddcd87eb9e0baf13d2277c336d4a1f58] seq order violation for vb = bucket5:451, seq = 1130, command = 10, orderState = {snapStart: 0, snapEnd 0, snapStarted false, prevSeq: 0, prevSeqValid: false, errCount: 1}, mutation = Docidx <ud>()</ud>, Seqno 1130, Ctime 0, Uuids [0], Commands [10]
|
/opt/couchbase/var/lib/couchbase/logs/projector.log.1.gz:2021-05-22T06:40:13.381-07:00 [Fatal] ENDP[<-(172.23.97.217:9105,5877)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_9e1712193814e6c47b16f7786d773379] seq order violation for vb = bucket4:305, seq = 1182, command = 10, orderState = {snapStart: 1181, snapEnd 1181, snapStarted true, prevSeq: 1181, prevSeqValid: true, errCount: 1}, mutation = Docidx <ud>()</ud>, Seqno 1182, Ctime 0, Uuids [0], Commands [10]
|
|
|
172.23.97.237 : projector
|
/opt/couchbase/var/lib/couchbase/logs/projector.log.1.gz:2021-05-22T06:40:16.358-07:00 [Fatal] ENDP[<-(172.23.97.217:9105,1a16)<-127.0.0.1:8091 #MAINT_STREAM_TOPIC_9e1712193814e6c47b16f7786d773379] seq order violation for vb = bucket4:975, seq = 1185, command = 10, orderState = {snapStart: 1184, snapEnd 1184, snapStarted true, prevSeq: 1184, prevSeqValid: true, errCount: 1}, mutation = Docidx <ud>()</ud>, Seqno 1185, Ctime 0, Uuids [0], Commands [10]
|
This is not a regression because this is a new keyword added to the eagle-eye tool for scanning the logs in the system test.