Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50045

XDCR doesn't report replication_changes_left correctly on build 7.1.0-1831

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Test Blocker
    • 7.1.0
    • 7.1.0
    • XDCR
    • Untriaged
    • 1
    • Yes

    Description

      Test: Avg. initial XDCR rate (items/sec), 1 -> 2 changes to 2 -> 2 (2 source nozzles, 4 target nozzles), 1 bucket x 200M x 1KB

      Job: http://perf.jenkins.couchbase.com/job/titan/12833/ 

      Build: 7.1.0-1831

      Logs:

      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-12-10T201638-ns_1%40172.23.96.105.zip

      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-12-10T201638-ns_1%40172.23.96.106.zip

      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-12-10T201732-ns_1%40172.23.96.100.zip

      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-12-10T201732-ns_1%40172.23.96.101.zip

       

      In this test, we will monitor number of replication_changes_left to make sure replication is finished (replication_changes_left=0). In the runs with build 7.1.0-1831, we see the number of replication_changes_left becomes negative so the runs will never finish. The issue is reproducible.

      2021-12-10T11:27:33 [INFO] replication_changes_left = 12,859,568

      2021-12-10T11:27:35 [INFO] replication_changes_left = 11,318,963

      2021-12-10T11:27:38 [INFO] replication_changes_left = -2,798,374

      2021-12-10T11:27:40 [INFO] replication_changes_left = -2,798,374

      2021-12-10T11:27:42 [INFO] replication_changes_left = -2,798,374

      2021-12-10T11:27:44 [INFO] replication_changes_left = -2,798,374

      2021-12-10T11:27:46 [INFO] replication_changes_left = -6,127,946

      2021-12-10T11:27:48 [INFO] replication_changes_left = -6,127,946

      2021-12-10T11:27:50 [INFO] replication_changes_left = -6,127,946

       

      We get replication_changes_left from the xdcr stats.

      http://172.23.96.105:8091/pools/default/buckets/@xdcr-bucket-1/stats 

      "replication_changes_left":[-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000,-90625000],"

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              bo-chun.wang Bo-Chun Wang
              bo-chun.wang Bo-Chun Wang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty