Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.2
-
Untriaged
-
0
-
Yes
Description
Several XDCR runs hit timeout threshold (6 hours) on build 7.6.2-3649. In these runs, xdcr_changes_left_total can't reach 0 for hours.
https://perf.jenkins.couchbase.com/job/titan/15946/
https://perf.jenkins.couchbase.com/job/titan/15948/
Good run: xdcr_changes_left_total reached 0 in 6-7 minutes.
Build 7.6.2-3623
https://perf.jenkins.couchbase.com/job/titan/15922/
2024-05-12T19:44:53 [INFO] Monitoring XDCR queues: bucket-1
2024-05-12T19:45:15 [INFO] Initial xdcr_mobile_docs_filtered_total = 0
2024-05-12T19:45:15 [INFO] xdcr_changes_left_total = 47,451,402
...
2024-05-12T19:51:24 [INFO] xdcr_changes_left_total reached 0
Bad runs: The value of xdcr_changes_left_total hasn't changed for 25 minutes.
Build: 7.6.2-3649
https://perf.jenkins.couchbase.com/job/titan/15950/
2024-05-20T10:00:03 [INFO] Monitoring XDCR queues: bucket-1
2024-05-20T10:00:25 [INFO] Initial xdcr_mobile_docs_filtered_total = 0
2024-05-20T10:00:25 [INFO] xdcr_changes_left_total = 50,049,393
...
2024-05-20T10:24:28 [INFO] xdcr_changes_left_total = 50,049,393
Log
Cluster 1:
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2024-05-20T172714-ns_1%40172.23.96.105.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2024-05-20T172714-ns_1%40172.23.96.106.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2024-05-20T172714-ns_1%40172.23.96.107.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2024-05-20T172714-ns_1%40172.23.96.108.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2024-05-20T172714-ns_1%40172.23.96.109.zip
Cluster 2:
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2024-05-20T172739-ns_1%40172.23.96.100.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2024-05-20T172739-ns_1%40172.23.96.101.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2024-05-20T172739-ns_1%40172.23.96.102.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2024-05-20T172739-ns_1%40172.23.96.103.zip
https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2024-05-20T172739-ns_1%40172.23.96.104.zip