Details
-
Bug
-
Resolution: Fixed
-
Major
-
Morpheus, 7.6.2
-
Untriaged
-
0
-
Yes
Description
Several XDCR runs on build 7.6.2-3550 don't work correctly. We see timeout messages in the job logs.
2024-04-16T04:37:23 [WARNING] XDCR sampling timed out after 600 seconds
2024-04-16T04:47:45 [WARNING] XDCR sampling timed out after 600 seconds
2024-04-16T04:58:06 [WARNING] XDCR sampling timed out after 600 seconds
2024-04-16T05:08:28 [WARNING] XDCR sampling timed out after 600 seconds
2024-04-16T05:18:50 [WARNING] XDCR sampling timed out after 600 seconds
In ns_server.goxdcr.log, there are many error messages.
2024-04-16T05:20:22.110-07:00 ERRO GOXDCR.XmemNozzle: pipelineFullTopic=00231cb2d2f898a399a8d45034e39a33/bucket-1/bucket-1, Received nil response for unique-key cd5a6d-00009269631610105%!s(MISSING)%!v(MISSING)
Build: 7.6.2-3550
Job: https://perf.jenkins.couchbase.com/job/titan_5node2/3422/
Logs:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan_5node2-3422/172.23.96.105.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan_5node2-3422/172.23.96.106.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan_5node2-3422/172.23.96.107.zip
Job: https://perf.jenkins.couchbase.com/job/titan_5node1/3091/
Logs:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan_5node1-3091/172.23.96.100.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan_5node1-3091/172.23.96.102.zip
2024-04-15T20:10:22.545-07:00 ERRO GOXDCR.XmemNozzle: pipelineFullTopic=1b7d9f2b5337e632951c3b5c7ad51c52/bucket-1/bucket-1, xmem_1b7d9f2b5337e632951c3b5c7ad51c52/bucket-1/bucket-1_172.23.96.102:11210_2 batchGet received fatal error and had to abort. Expected 493 responses, got 7 responses. err=Connection is bad
2024-04-15T20:10:22.545-07:00 INFO GOXDCR.XmemNozzle: pipelineFullTopic=1b7d9f2b5337e632951c3b5c7ad51c52/bucket-1/bucket-1, xmem_1b7d9f2b5337e632951c3b5c7ad51c52/bucket-1/bucket-1_172.23.96.102:11210_2 Expected=map[3416522752:[<ud>d2eb68-000000051204</ud> 1427 745 2024-04-15 20:10:22.498040419 -0700 PDT m=+1640.738805939 2] 3416522753:[<ud>d2eb68-000000059713</ud> 1445 745
2024-04-15 20:10:22.4980492 -0700 PDT m=+1640.738814720 2] 3416522754:[<ud>d2eb68-000000027459</ud> 1511 745 2024-04-15 20:10:22.498051159 -0700 PDT m=+1640.738816681 2] 3416522755:[<ud>...truncated due to encoding error
2024-04-15T20:10:22.546-07:00 ERRO GOXDCR.XmemNozzle: pipelineFullTopic=1b7d9f2b5337e632951c3b5c7ad51c52/bucket-1/bucket-1, Received nil response for unique-key d2eb68-0000000804923971%!s(MISSING)%!v(MISSING)