Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7797

massive perf regression in 2.0.1 build 156 (observed in rerun of MB-6726 and MB-7797)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.1
    • Component/s: performance
    • Security Level: Public
    • Labels:
      None
    • Sprint:
      PCI Team - Sprint 1

      Description

      According to data at: https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdDV1SXhUZjJ0b0RhU3gtdlUzZGloUFE#gid=0

      we've regressed 2x in build 160. There were very tiny changes since build 156. With 160 itself no different at all compared to 159. http://builds.hq.northscale.net/latestbuilds/CHANGES_couchbase-server-2.0.1-160-rel.txt

      I need somebody answer very simply question. Have we regressed or not?

      I'd like to CC folks: Filipe, Damien, Jin, Dipti, Farshid, Pavel, Yaseen but jira doesn't allow me.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        Aliaksey Artamonau Aliaksey Artamonau added a comment -

        I hardly believe that my change could cause the regression. This code is exercised only when some of the cluster nodes are of pre 2.0 version.

        Show
        Aliaksey Artamonau Aliaksey Artamonau added a comment - I hardly believe that my change could cause the regression. This code is exercised only when some of the cluster nodes are of pre 2.0 version.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        And I've shared same opinion with Jin, Farshid, Steve, Ronnie and everyone else.

        So far we're waiting results of Pavel's second run. Which will either confirm there's some issue or indicate that it's some "noise". Of course 2x of noise is something to investigate.

        So as soon as we have any new data, I'd like Pavel or somebody else from perf team to give us diags and master events from good and bad runs so that we can see what may cause that and where this difference occurs.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - And I've shared same opinion with Jin, Farshid, Steve, Ronnie and everyone else. So far we're waiting results of Pavel's second run. Which will either confirm there's some issue or indicate that it's some "noise". Of course 2x of noise is something to investigate. So as soon as we have any new data, I'd like Pavel or somebody else from perf team to give us diags and master events from good and bad runs so that we can see what may cause that and where this difference occurs.
        Hide
        pavelpaulau Pavel Paulau added a comment - - edited

        Re-run took normal 5.5h, but this is not the first time when we observe 2x noise:

        http://dashboard.hq.couchbase.com/litmus/dashboard/?testcase=reb-vperf-60M-in-1d

        MB-7838 is a separate bug for investigation with all required information for Alk and other guys.

        Show
        pavelpaulau Pavel Paulau added a comment - - edited Re-run took normal 5.5h, but this is not the first time when we observe 2x noise: http://dashboard.hq.couchbase.com/litmus/dashboard/?testcase=reb-vperf-60M-in-1d MB-7838 is a separate bug for investigation with all required information for Alk and other guys.
        Show
        pavelpaulau Pavel Paulau added a comment - @Xiaoqin This is test spec: https://raw.github.com/couchbase/testrunner/master/conf/perf/reb-vperf-60M-in-1d.conf
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #481 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/481/)
        MB-7797: Lower bgfetch sleep time for 60s to 1s (Revision 62717fadb0f59cf50396908845625a4142b4a44f)
        Revert "MB-7797: Lower bgfetch sleep time for 60s to 1s" (Revision 1b81128a14d25bb62ed63ffc14618c3e3a9374c1)

        Result = SUCCESS
        Mike Wiederhold :
        Files :

        • src/bgfetcher.cc

        Mike Wiederhold :
        Files :

        • src/bgfetcher.cc
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #481 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/481/ ) MB-7797 : Lower bgfetch sleep time for 60s to 1s (Revision 62717fadb0f59cf50396908845625a4142b4a44f) Revert " MB-7797 : Lower bgfetch sleep time for 60s to 1s" (Revision 1b81128a14d25bb62ed63ffc14618c3e3a9374c1) Result = SUCCESS Mike Wiederhold : Files : src/bgfetcher.cc Mike Wiederhold : Files : src/bgfetcher.cc

          People

          • Assignee:
            pavelpaulau Pavel Paulau
            Reporter:
            alkondratenko Aleksey Kondratenko (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Agile

                Gerrit Reviews

                There are no open Gerrit changes