Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49613

[System Test][XDCR] UI shows ~7.5M mutations remaining for XDCR in longevity

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • Neo
    • Neo
    • XDCR
    • Untriaged
    • 1
    • No

    Description

      7.1.0-1716

      Test:
      -test tests/integration/neo/test_neo_couchstore_milestone3.yml -scope tests/integration/neo/scope_couchstore.yml
      Scale 3
      Iteration 2

      This was run with p2p enabled - we can see from the UI that there are ~7.5M mutations remaining - attaching screenshot - latest logs from the cluster:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.106.134.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.120.58.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.120.73.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.120.74.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.120.75.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.120.77.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.120.81.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.120.86.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.121.77.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.123.24.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.123.25.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.123.26.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.123.31.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.123.32.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.123.33.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.96.122.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.96.14.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.96.243.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.96.254.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.96.48.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.97.110.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.97.112.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.97.148.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.97.149.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.97.150.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.97.241.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/xdcrp2p/collectinfo-2021-11-17T164511-ns_1%40172.23.97.74.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.1.0-1838 contains goxdcr commit 85eb796 with commit message:
            MB-49613 - Use unique subscriber ID for subscribing to bucket topology service to avoid collision

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1838 contains goxdcr commit 85eb796 with commit message: MB-49613 - Use unique subscriber ID for subscribing to bucket topology service to avoid collision
            neil.huang Neil Huang added a comment - - edited

            With Lilei's fix to MB-49480, we no longer see a build up of mutations... but instead the changes_left are now negative:

            [root@localhost logs]# grep "changes_left=" goxdcr.log | grep remote | tail
            2021-12-03T15:11:55.948-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2307512, docs_processed=2439915, changes_left=-132403
            2021-12-03T15:11:56.944-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2307831, docs_processed=2440258, changes_left=-132427
            2021-12-03T15:11:57.936-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2308186, docs_processed=2440588, changes_left=-132402
            2021-12-03T15:11:58.939-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2308420, docs_processed=2440972, changes_left=-132552
            2021-12-03T15:11:59.937-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2308782, docs_processed=2441221, changes_left=-132439
            2021-12-03T15:12:00.936-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2309083, docs_processed=2441604, changes_left=-132521
            2021-12-03T15:12:01.936-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2309376, docs_processed=2441923, changes_left=-132547
            2021-12-03T15:12:02.936-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2309589, docs_processed=2442225, changes_left=-132636
            2021-12-03T15:12:03.935-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2309694, docs_processed=2442447, changes_left=-132753
            2021-12-03T15:12:04.934-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2309918, docs_processed=2442562, changes_left=-132644
            

            At the very least, we know the pipelines are moving and not stuck (can see docs_processed moving forward) ... but the negative stats are still disconcerting.

            neil.huang Neil Huang added a comment - - edited With Lilei's fix to MB-49480 , we no longer see a build up of mutations... but instead the changes_left are now negative: [root@localhost logs]# grep "changes_left=" goxdcr.log | grep remote | tail 2021-12-03T15:11:55.948-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2307512, docs_processed=2439915, changes_left=-132403 2021-12-03T15:11:56.944-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2307831, docs_processed=2440258, changes_left=-132427 2021-12-03T15:11:57.936-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2308186, docs_processed=2440588, changes_left=-132402 2021-12-03T15:11:58.939-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2308420, docs_processed=2440972, changes_left=-132552 2021-12-03T15:11:59.937-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2308782, docs_processed=2441221, changes_left=-132439 2021-12-03T15:12:00.936-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2309083, docs_processed=2441604, changes_left=-132521 2021-12-03T15:12:01.936-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2309376, docs_processed=2441923, changes_left=-132547 2021-12-03T15:12:02.936-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2309589, docs_processed=2442225, changes_left=-132636 2021-12-03T15:12:03.935-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2309694, docs_processed=2442447, changes_left=-132753 2021-12-03T15:12:04.934-08:00 INFO GOXDCR.StatsMgr: f2325311edf4a39551dbb3fc52797641/default/remote total_docs=2309918, docs_processed=2442562, changes_left=-132644 At the very least, we know the pipelines are moving and not stuck (can see docs_processed moving forward) ... but the negative stats are still disconcerting.

            Build couchbase-server-7.1.0-1840 contains goxdcr commit 0d885a3 with commit message:
            MB-49613: add summary table for debugging and clean up warnings

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1840 contains goxdcr commit 0d885a3 with commit message: MB-49613 : add summary table for debugging and clean up warnings
            neil.huang Neil Huang added a comment -

            After a number of iterations, all related MBs have been checked in and fixed.

            neil.huang Neil Huang added a comment - After a number of iterations, all related MBs have been checked in and fixed.

            There is another task tracking this for testware changes - closing this one

            arunkumar Arunkumar Senthilnathan added a comment - There is another task tracking this for testware changes - closing this one

            People

              neil.huang Neil Huang
              arunkumar Arunkumar Senthilnathan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty