Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50327

Graceful failover time increased from 12K ms to 23K ms on build 7.1.0-2021

    XMLWordPrintable

Details

    • Task
    • Status: Resolved
    • Critical
    • Resolution: Cannot Reproduce
    • 7.1.0
    • None
    • performance
    • 1

    Description

      Compared to build 7.1.0-1985, the grace failover time on build 7.1.0-2021 is doubled. The regression is reproducible.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          owend Daniel Owen added a comment -

          Hi Bo-Chun Wang, Could you help with binary chopping the builds to try to isolate the patch?

          thanks

          owend Daniel Owen added a comment - Hi Bo-Chun Wang , Could you help with binary chopping the builds to try to isolate the patch? thanks
          owend Daniel Owen added a comment -

          Thanks Bo-Chun Wang- Could we also try build 2000 and 2010? As that should help us isolate.

          owend Daniel Owen added a comment - Thanks Bo-Chun Wang - Could we also try build 2000 and 2010? As that should help us isolate.
          bo-chun.wang Bo-Chun Wang added a comment -

          Yes. I am doing runs to finish the bisection.

          bo-chun.wang Bo-Chun Wang added a comment - Yes. I am doing runs to finish the bisection.
          owend Daniel Owen added a comment - - edited
          Build

          Graceful failover (ms)

          Job
          7.1.0-1993 29631 http://perf.jenkins.couchbase.com/job/hestia/7192
          7.1.0-1993 22439 http://perf.jenkins.couchbase.com/job/hestia/7193
          7.1.0-2000 23845 http://perf.jenkins.couchbase.com/job/hestia/7194
          7.1.0-2000 12179 http://perf.jenkins.couchbase.com/job/hestia/7195

          Looking at the changes between 1985 and 1993 we see none that would explain the slowdown:

          * Commit: fc8b9f9dafaaa4177bfcd3ad588c74bbbbdccf47 in build: couchbase-server-7.1.0-1993
          MB-49977: Add configuration for auxio and nonio threads
           
          * Commit: f9a4e466c018e28f9660150f8ee5fadb77a02c67 in build: couchbase-server-7.1.0-1992
          MB-49140: Update BSL license version/change date for Neo
           
          * Commit: 38d33c9d55cb38ae0518e5a0d26269a61f2fec3b in build: couchbase-server-7.1.0-1992
          Throw the created SslContextException
           
          * Commit: a109c265a8663c50e7816a33058152adde6277ce in build: couchbase-server-7.1.0-1991
          MB-50110: Rename ClosedUnrefCheckpointRemoverTask
          

          Therefore doing more runs at 1985:

          Build

          Graceful failover (ms)

          Job
          7.1.0-1985 12216 http://perf.jenkins.couchbase.com/job/hestia/7196
          7.1.0-1985 20708 http://perf.jenkins.couchbase.com/job/hestia/7197
          7.1.0-1985 32825 http://perf.jenkins.couchbase.com/job/hestia/7198
          owend Daniel Owen added a comment - - edited Build Graceful failover (ms) Job 7.1.0-1993 29631 http://perf.jenkins.couchbase.com/job/hestia/7192 7.1.0-1993 22439 http://perf.jenkins.couchbase.com/job/hestia/7193 7.1.0-2000 23845 http://perf.jenkins.couchbase.com/job/hestia/7194 7.1.0-2000 12179 http://perf.jenkins.couchbase.com/job/hestia/7195 Looking at the changes between 1985 and 1993 we see none that would explain the slowdown: * Commit: fc8b9f9dafaaa4177bfcd3ad588c74bbbbdccf47 in build: couchbase-server-7.1.0-1993 MB-49977: Add configuration for auxio and nonio threads   * Commit: f9a4e466c018e28f9660150f8ee5fadb77a02c67 in build: couchbase-server-7.1.0-1992 MB-49140: Update BSL license version/change date for Neo   * Commit: 38d33c9d55cb38ae0518e5a0d26269a61f2fec3b in build: couchbase-server-7.1.0-1992 Throw the created SslContextException   * Commit: a109c265a8663c50e7816a33058152adde6277ce in build: couchbase-server-7.1.0-1991 MB-50110: Rename ClosedUnrefCheckpointRemoverTask Therefore doing more runs at 1985: Build Graceful failover (ms) Job 7.1.0-1985 12216 http://perf.jenkins.couchbase.com/job/hestia/7196 7.1.0-1985 20708 http://perf.jenkins.couchbase.com/job/hestia/7197 7.1.0-1985 32825 http://perf.jenkins.couchbase.com/job/hestia/7198
          owend Daniel Owen added a comment -

          Hi Bo-Chun Wang, Unfortunately the test is not stable - hence changing the component to performance.

          owend Daniel Owen added a comment - Hi Bo-Chun Wang , Unfortunately the test is not stable - hence changing the component to performance.

          People

            bo-chun.wang Bo-Chun Wang
            bo-chun.wang Bo-Chun Wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty