Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-33074

Intermittent failure in ep_testsuite_dcp 'dcp early termination'

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 6.5.0
    • 6.5.0
    • couchbase-bucket
    • None
    • Untriaged
    • Yes

    Description

      As seen during CV on Windows:

      test dcp early termination (couchstore)...Exceeded maximum wait time of 60000000us waiting for stat 'ep_dcp_num_running_backfills(dcp)' to be 0 (last value:1) - aborting.
      

      Note this frequency of this (intermittent) failure has increased significantly in recent runs - below are the occurrences in the last 100 runs of kv_engine-windows-master:

      ‡ ./scripts/jenkins_console_log_search.py --job kv_engine-windows-master --search "Exceeded maximum wait time of 60000000us waiting for stat 'ep_dcp_num_running_backfills\(dcp\)'" --build-no 17299
      Searching for RegEx "Exceeded maximum wait time of 60000000us waiting for stat 'ep_dcp_num_running_backfills\(dcp\)'" in console logs of job: kv_engine-windows-master between build 17200 and 17299
       Completed search in 47.76758289337158 s
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17299/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17298/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17297/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17296/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17295/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17293/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17292/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17290/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17289/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17284/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17283/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17281/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17280/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17279/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17277/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17276/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17273/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17272/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17270/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17269/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17267/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17265/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17264/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17263/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17262/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17261/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17260/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17257/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17255/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17249/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17245/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17244/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17243/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17239/console/
      http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17236/console/
      

      A summary of instances per 10 builds shows the frequency ramped up somewhere near 17240:

      ‡ pbpaste | histogram.py -m 17200 -x 17300 -b 10 -f %d --no-mvsd
      # NumSamples = 35; Min = 17200.00; Max = 17300.00
      # each ∎ represents a count of 1
      17200 - 17210 [     0]: 
      17210 - 17220 [     0]: 
      17220 - 17230 [     0]: 
      17230 - 17240 [     2]: ∎∎
      17240 - 17250 [     4]: ∎∎∎∎
      17250 - 17260 [     3]: ∎∎∎
      17260 - 17270 [     8]: ∎∎∎∎∎∎∎∎
      17270 - 17280 [     6]: ∎∎∎∎∎∎
      17280 - 17290 [     5]: ∎∎∎∎∎
      17290 - 17300 [     7]: ∎∎∎∎∎∎∎
      

      Note this only fails on Windows; it seems reliable on Linux.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Build couchbase-server-6.5.0-2357 contains kv_engine commit d960e13 with commit message:
          MB-33074: Skip test_dcp_early_termination (WIN32)

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-2357 contains kv_engine commit d960e13 with commit message: MB-33074 : Skip test_dcp_early_termination (WIN32)
          drigby Dave Rigby added a comment -

          From discussions on Slack, appears this is related to http://review.couchbase.org/#/c/103963/12/engines/ep/src/dcp/backfill-manager.cc

          JimW investigating...

          drigby Dave Rigby added a comment - From discussions on Slack, appears this is related to http://review.couchbase.org/#/c/103963/12/engines/ep/src/dcp/backfill-manager.cc JimW investigating...
          jwalker Jim Walker added a comment -

          Submitted a fix under - MB-33078

          jwalker Jim Walker added a comment - Submitted a fix under - MB-33078

          Build couchbase-server-6.5.0-2377 contains kv_engine commit 14dc732 with commit message:
          MB-33074: Reinstate decrement of stat from backfill-manager DTOR

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-2377 contains kv_engine commit 14dc732 with commit message: MB-33074 : Reinstate decrement of stat from backfill-manager DTOR

          Build couchbase-server-6.5.0-2383 contains kv_engine commit 366e2ce with commit message:
          Revert "MB-33074: Skip test_dcp_early_termination (WIN32)"

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-2383 contains kv_engine commit 366e2ce with commit message: Revert " MB-33074 : Skip test_dcp_early_termination (WIN32)"

          People

            jwalker Jim Walker
            drigby Dave Rigby
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty