Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31379

XDCR - Uncompressed replication from 6.0.0-1643 to 5.0 fails

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 6.5.0
    • 6.0.0
    • XDCR
    • None
    • Triaged
    • Yes

    Description

      Build
      6.0.0-1643

      Testcase
      ./testrunner -i /tmp/testexec.5266.ini -p stop-on-failure=False,get-coredumps=True,fail_on_errors=1,upgrade_version=6.0.0-1643,get-cbcollect-info=True -t xdcr.upgradeXDCR.UpgradeTests.test_backward_compatibility,initial_version=5.0.0-3519,items=50000,bucket_topology=default:1>2;standard_bucket_1:1<2;sasl_bucket_1:1><2,expires=50,checkpoint_interval=60,GROUP=BACK_COMP

      This is a regression between builds 6.0.0-1614 and 6.0.0-1643
      Steps
      1.Load 3 buckets on source cluster running 5.0.0-3519 build. (default, sasl_bucket_1, standard_bucket_1)
      2. Install 6.0.0-1643 build on destination cluster.
      3. Setup C1->C2 XDCR for default bucket, C2->C1 for standard_bucket_1, C1 <-> C2 for sasl_bucket
      4.Perform replication verification on source-target. Verification fails only for standard_bucket_1. (0% replication in 300s)

      Attached screenshots

      1&2 : Last 10 errors seen on UI pointing to compression issues

      3:  Stats showing 50K outbound mutations pending on sasl and standard buckets

      2018-09-21 09:15:04,011 - root - INFO - wait for expiry pager to run on all these nodes
      2018-09-21 09:15:14,046 - root - INFO - http://172.23.105.152:8091/pools/default/buckets/default/stats?zoom=minute
      2018-09-21 09:15:14,090 - root - INFO - http://172.23.105.163:8091/pools/default/buckets/default/stats?zoom=minute
      2018-09-21 09:15:14,128 - root - INFO - Replication caught up for bucket default: 50000
      2018-09-21 09:15:14,128 - root - INFO - http://172.23.105.152:8091/pools/default/buckets/sasl_bucket_1/stats?zoom=minute
      2018-09-21 09:15:14,169 - root - INFO - http://172.23.105.163:8091/pools/default/buckets/sasl_bucket_1/stats?zoom=minute
      2018-09-21 09:15:14,207 - root - INFO - Replication caught up for bucket sasl_bucket_1: 50000
      2018-09-21 09:15:14,207 - root - INFO - http://172.23.105.152:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute
      2018-09-21 09:15:14,252 - root - INFO - http://172.23.105.163:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute
      2018-09-21 09:15:14,291 - root - INFO - sleep for 60 secs. Bucket: standard_bucket_1, count in one cluster : 0 items, another : 50000. Waiting for replication to catch up .. ...
      2018-09-21 09:16:14,311 - root - INFO - http://172.23.105.152:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute
      2018-09-21 09:16:14,352 - root - INFO - http://172.23.105.163:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute
      2018-09-21 09:16:14,389 - root - INFO - sleep for 60 secs. Bucket: standard_bucket_1, count in one cluster : 0 items, another : 50000. Waiting for replication to catch up .. ...
      2018-09-21 09:17:14,445 - root - INFO - http://172.23.105.152:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute
      2018-09-21 09:17:14,495 - root - INFO - http://172.23.105.163:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute
      2018-09-21 09:17:14,531 - root - INFO - sleep for 60 secs. Bucket: standard_bucket_1, count in one cluster : 0 items, another : 50000. Waiting for replication to catch up .. ...
      2018-09-21 09:18:14,583 - root - INFO - http://172.23.105.152:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute
      2018-09-21 09:18:14,625 - root - INFO - http://172.23.105.163:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute
      2018-09-21 09:18:14,660 - root - INFO - sleep for 60 secs. Bucket: standard_bucket_1, count in one cluster : 0 items, another : 50000. Waiting for replication to catch up .. ...
      2018-09-21 09:19:14,718 - root - INFO - http://172.23.105.152:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute
      2018-09-21 09:19:14,771 - root - INFO - http://172.23.105.163:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute
      2018-09-21 09:19:14,809 - root - INFO - sleep for 60 secs. Bucket: standard_bucket_1, count in one cluster : 0 items, another : 50000. Waiting for replication to catch up .. ...
      FAIL
      
      

      Attachments

        1. Screen Shot 2018-09-21 at 3.03.24 PM.png
          132 kB
          Pavithra Mahamani
        2. Screen Shot 2018-09-21 at 3.07.07 PM.png
          40 kB
          Pavithra Mahamani
        3. Screen Shot 2018-09-21 at 3.24.33 PM.png
          150 kB
          Pavithra Mahamani
        4. Screen Shot 2018-09-26 at 10.18.44 AM.png
          72 kB
          Pavithra Mahamani
        5. Screen Shot 2018-09-26 at 10.19.20 AM.png
          63 kB
          Pavithra Mahamani
        6. Screen Shot 2018-09-26 at 10.25.53 AM.png
          56 kB
          Pavithra Mahamani
        7. xdcr_compat_stdbkt.tar.gz
          6.23 MB
          Pavithra Mahamani

        Issue Links

          Activity

            People

              pavithra.mahamani Pavithra Mahamani (Inactive)
              pavithra.mahamani Pavithra Mahamani (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty