Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-18733

XDCR : Replication rate has regressed in build 4.5.0-1798.

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      Replication rate has regressed in build 4.5.0-1798.

      • 1 x 1, non-DGM (100M)
        4.5.0-1770 : 96K items / sec
        4.5.0-1798 : 42K items / sec.

      All other tests also regressed.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-18185 XDCR Average Initial Rate shows a regression from 4.1 to 4.5

          • Major - Major loss of function.
          • Closed
          mentioned in

          Page Loading...

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ascending order - Click to sort in descending order
            wayne Wayne Siu created issue -
            wayne Wayne Siu made changes -
            Field Original Value New Value
            Is this a Regression? Unknown [ 10452 ] Yes [ 10450 ]
            wayne Wayne Siu made changes -
            Fix Version/s watson [ 12540 ]
            jliang John Liang made changes -
            Assignee Yu Sui [ yu ] Ritesh Motlani [ ritesh.motlani ]
            ericcooper Eric Cooper (Inactive) made changes -
            Remote Link This issue links to "Page (Couchbase, Inc. Wiki)" [ 12064 ]
            ritesh.motlani Ritesh Motlani (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            wayne Wayne Siu made changes -
            Link This issue relates to MB-18185 [ MB-18185 ]
            wayne Wayne Siu made changes -
            Summary XDCR : Replication rate seems to have regressed in certain tests XDCR : Replication rate has regressed in build 4.5.0-1798.
            wayne Wayne Siu made changes -
            Description Replication rate seems to have regressed in some tests.
            - 1 x 1, non-DGM (100M)
            4.5.0-1770 : 96K items / sec
            4.5.0-1798 : 42K items / sec.

            Confirming other tests at the moment.             		Replication rate has regressed in build 4.5.0-1798.
            - 1 x 1, non-DGM (100M)
            4.5.0-1770 : 96K items / sec
            4.5.0-1798 : 42K items / sec.

            All other tests also regressed.
            ritesh.motlani Ritesh Motlani (Inactive) added a comment -

            It seems like on 1798 build, goxdcr is crashing couple of times and getting restarted. Restarting goxdcr leads to processing of mutation from the beginning every time goxdcr re-start.

            GOXDCR is crashing at statistic manager service due to race condition. I still working on finding solution.

            -Thanks
            Ritesh

            ritesh.motlani Ritesh Motlani (Inactive) added a comment - It seems like on 1798 build, goxdcr is crashing couple of times and getting restarted. Restarting goxdcr leads to processing of mutation from the beginning every time goxdcr re-start. GOXDCR is crashing at statistic manager service due to race condition. I still working on finding solution. -Thanks Ritesh
            keshav Keshav Murthy added a comment -

            Hi Ritesh, will this be addressed today/tomorrow? Thanks.

            keshav Keshav Murthy added a comment - Hi Ritesh, will this be addressed today/tomorrow? Thanks.
            ritesh.motlani Ritesh Motlani (Inactive) added a comment -

            This issue occurs with go 1.6 . I have not been able to reproduce locally ..
            I am working on reproducing this issue on dev sanity.

            -THanks
            Ritesh

            ritesh.motlani Ritesh Motlani (Inactive) added a comment - This issue occurs with go 1.6 . I have not been able to reproduce locally .. I am working on reproducing this issue on dev sanity. -THanks Ritesh
            keshav Keshav Murthy made changes -
            Labels perf perf_daily performance beta1 perf perf_daily performance
            ritesh.motlani Ritesh Motlani (Inactive) added a comment - - edited

            I was not able to reproduce these bugs locally with golang 1.6 version.
            but running golang race detector shows problem of "Data Race" Condition on the code as show in the attaced logs.
            so fix would be put locks on settingMap & registryMap and that should fix the problem of goxdcr crashing.
            I hope that should also fix the performance problem but need to test it out.

            -Thanks
            RItesh

            ritesh.motlani Ritesh Motlani (Inactive) added a comment - - edited I was not able to reproduce these bugs locally with golang 1.6 version. but running golang race detector shows problem of "Data Race" Condition on the code as show in the attaced logs. so fix would be put locks on settingMap & registryMap and that should fix the problem of goxdcr crashing. I hope that should also fix the performance problem but need to test it out. -Thanks RItesh
            wayne Wayne Siu added a comment -

            Build 4.5.0-1939 (back to go 1.5) seems to have addressed the regression in replication rate.
            MB-18185 is still open to track the slight regression from go 1.5.

            wayne Wayne Siu added a comment - Build 4.5.0-1939 (back to go 1.5) seems to have addressed the regression in replication rate. MB-18185 is still open to track the slight regression from go 1.5.
            wayne Wayne Siu added a comment -

            Verified in 4.5.0-1939.

            wayne Wayne Siu added a comment - Verified in 4.5.0-1939.
            wayne Wayne Siu made changes -
            Assignee Ritesh Motlani [ ritesh.motlani ] Wayne Siu [ wayne ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]

            People

              wayne Wayne Siu
              wayne Wayne Siu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty