Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49170

Replica item count lagging active in Magma insert test

    XMLWordPrintable

Details

    Description

      Note: Initially opened on performance variation between, that has been addressed, but now tracking the issue where replica item count does not reach active

      In Magma insert only tests, we see high performance variation. 

      http://172.23.123.237/#/timeline/Linux/hidd/S0/all

      In the latest runs with build 7.1.0-1558, the throughput changed from 122K to 226K.

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=rhea_710-1558_access_key_prefix_e3f5&&label=high_insert_rate&snapshot=rhea_710-1558_access_key_prefix_3994&label=low_insert_rate

      In the run having higher throughput, replica sync rate can't catch up.

        There are more sync write flushes after a certain point.

        Sarath Lakshman

      Please take a look. Is there a way we can change checkpoint settings? It looks like the runs can go to different modes (or code paths), even with the same build.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            bo-chun.wang Bo-Chun Wang created issue -
            sarath Sarath Lakshman made changes -
            Field Original Value New Value
            Assignee Sarath Lakshman [ sarath ] Bo-Chun Wang [ bo-chun.wang ]
            bo-chun.wang Bo-Chun Wang made changes -
            bo-chun.wang Bo-Chun Wang made changes -
            Assignee Bo-Chun Wang [ bo-chun.wang ] Sarath Lakshman [ sarath ]
            sarath Sarath Lakshman made changes -
            sarath Sarath Lakshman made changes -
            sarath Sarath Lakshman made changes -
            Component/s couchbase-bucket [ 10173 ]
            sarath Sarath Lakshman made changes -
            Assignee Sarath Lakshman [ sarath ] Daniel Owen [ owend ]
            drigby Dave Rigby made changes -
            Attachment Unknown.png [ 168141 ]
            drigby Dave Rigby made changes -
            Attachment Unknown.png [ 168141 ]
            drigby Dave Rigby made changes -
            Attachment 2c9be7495d498bf4ae151733781c8069.png [ 168142 ]
            drigby Dave Rigby made changes -
            Attachment fc98c640dcf022cd0036bb64ca36e284.png [ 168143 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            owend Daniel Owen made changes -
            Rank Ranked lower
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Bo-Chun Wang [ bo-chun.wang ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            bo-chun.wang Bo-Chun Wang made changes -
            bo-chun.wang Bo-Chun Wang made changes -
            bo-chun.wang Bo-Chun Wang made changes -
            Assignee Bo-Chun Wang [ bo-chun.wang ] Daniel Owen [ owend ]
            owend Daniel Owen made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Bo-Chun Wang [ bo-chun.wang ]
            drigby Dave Rigby made changes -
            drigby Dave Rigby made changes -
            drigby Dave Rigby made changes -
            owend Daniel Owen made changes -
            Assignee Bo-Chun Wang [ bo-chun.wang ] Daniel Owen [ owend ]
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Dave Rigby [ drigby ]
            drigby Dave Rigby made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            drigby Dave Rigby made changes -
            Link This issue relates to MB-49262 [ MB-49262 ]
            drigby Dave Rigby made changes -
            Description In Magma insert only tests, we see high performance variation. 

            [http://172.23.123.237/#/timeline/Linux/hidd/S0/all]

            !Screen Shot 2021-10-26 at 11.17.30 AM.png|width=1000!

            In the latest runs with build 7.1.0-1558, the throughput changed from 122K to 226K.
            ||Build||Throughput||Job||
            |7.1.0-1558|122,966|[http://perf.jenkins.couchbase.com/job/rhea-5node2/1538/]|
            |7.1.0-1558|226,157|[http://perf.jenkins.couchbase.com/job/rhea-5node2/1539/]|

            [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=rhea_710-1558_access_key_prefix_e3f5&&label=high_insert_rate&snapshot=rhea_710-1558_access_key_prefix_3994&label=low_insert_rate]

            In the run having higher throughput, replica sync rate can't catch up.
             !Screen Shot 2021-10-26 at 11.40.04 AM.png|width=600!

            !Screen Shot 2021-10-26 at 11.40.28 AM.png|width=600!   There are more sync write flushes after a certain point.
             !Screen Shot 2021-10-26 at 11.46.41 AM.png|width=600!
              [~sarath]

            Please take a look. Is there a way we can change checkpoint settings? It looks like the runs can go to different modes (or code paths), even with the same build.
            _Note: Initially opened on performance variation between, that has been addressed, but now tracking the issue where replica item count does not reach active_
            ---

            In Magma insert only tests, we see high performance variation. 

            [http://172.23.123.237/#/timeline/Linux/hidd/S0/all]

            !Screen Shot 2021-10-26 at 11.17.30 AM.png|width=1000!

            In the latest runs with build 7.1.0-1558, the throughput changed from 122K to 226K.
            ||Build||Throughput||Job||
            |7.1.0-1558|122,966|[http://perf.jenkins.couchbase.com/job/rhea-5node2/1538/]|
            |7.1.0-1558|226,157|[http://perf.jenkins.couchbase.com/job/rhea-5node2/1539/]|

            [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=rhea_710-1558_access_key_prefix_e3f5&&label=high_insert_rate&snapshot=rhea_710-1558_access_key_prefix_3994&label=low_insert_rate]

            In the run having higher throughput, replica sync rate can't catch up.
             !Screen Shot 2021-10-26 at 11.40.04 AM.png|width=600!

            !Screen Shot 2021-10-26 at 11.40.28 AM.png|width=600!   There are more sync write flushes after a certain point.
             !Screen Shot 2021-10-26 at 11.46.41 AM.png|width=600!
              [~sarath]

            Please take a look. Is there a way we can change checkpoint settings? It looks like the runs can go to different modes (or code paths), even with the same build.
            drigby Dave Rigby made changes -
            Summary High performance variation in Magma insert only tests Replica item count lagging active in Magma insert test
            paolo.cocchi Paolo Cocchi made changes -
            Assignee Dave Rigby [ drigby ] Paolo Cocchi [ paolo.cocchi ]
            paolo.cocchi Paolo Cocchi made changes -
            Epic Link MB-38441 [ 123649 ]
            paolo.cocchi Paolo Cocchi made changes -
            Sprint KV 2021-Nov [ 1866 ]
            paolo.cocchi Paolo Cocchi made changes -
            Status Reopened [ 4 ] In Progress [ 3 ]
            owend Daniel Owen made changes -
            Rank Ranked lower
            paolo.cocchi Paolo Cocchi made changes -
            Attachment MB-49170_build-1729.png [ 170525 ]
            paolo.cocchi Paolo Cocchi made changes -
            Attachment MB-49170_build-1729.png [ 170525 ]
            paolo.cocchi Paolo Cocchi made changes -
            Attachment MB-49170_build-1729.png [ 170526 ]
            paolo.cocchi Paolo Cocchi made changes -
            Triage Untriaged [ 10351 ] Triaged [ 10350 ]
            Resolution Not a Bug [ 10200 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            paolo.cocchi Paolo Cocchi made changes -
            Assignee Paolo Cocchi [ paolo.cocchi ] Sarath Lakshman [ sarath ]
            sarath Sarath Lakshman made changes -
            Assignee Sarath Lakshman [ sarath ] Bo-Chun Wang [ bo-chun.wang ]
            ritam.sharma Ritam Sharma made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              bo-chun.wang Bo-Chun Wang
              bo-chun.wang Bo-Chun Wang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty