Details
-
Bug
-
Resolution: Fixed
-
Major
-
6.5.0
-
None
-
Untriaged
-
Yes
Description
As seen during CV on Windows:
test dcp early termination (couchstore)...Exceeded maximum wait time of 60000000us waiting for stat 'ep_dcp_num_running_backfills(dcp)' to be 0 (last value:1) - aborting.
|
Note this frequency of this (intermittent) failure has increased significantly in recent runs - below are the occurrences in the last 100 runs of kv_engine-windows-master:
‡ ./scripts/jenkins_console_log_search.py --job kv_engine-windows-master --search "Exceeded maximum wait time of 60000000us waiting for stat 'ep_dcp_num_running_backfills\(dcp\)'" --build-no 17299
|
Searching for RegEx "Exceeded maximum wait time of 60000000us waiting for stat 'ep_dcp_num_running_backfills\(dcp\)'" in console logs of job: kv_engine-windows-master between build 17200 and 17299
|
Completed search in 47.76758289337158 s
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17299/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17298/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17297/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17296/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17295/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17293/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17292/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17290/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17289/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17284/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17283/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17281/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17280/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17279/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17277/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17276/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17273/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17272/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17270/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17269/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17267/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17265/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17264/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17263/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17262/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17261/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17260/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17257/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17255/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17249/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17245/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17244/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17243/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17239/console/
|
http://cv.jenkins.couchbase.com/job/kv_engine-windows-master/17236/console/
|
A summary of instances per 10 builds shows the frequency ramped up somewhere near 17240:
‡ pbpaste | histogram.py -m 17200 -x 17300 -b 10 -f %d --no-mvsd
|
# NumSamples = 35; Min = 17200.00; Max = 17300.00
|
# each ∎ represents a count of 1
|
17200 - 17210 [ 0]:
|
17210 - 17220 [ 0]:
|
17220 - 17230 [ 0]:
|
17230 - 17240 [ 2]: ∎∎
|
17240 - 17250 [ 4]: ∎∎∎∎
|
17250 - 17260 [ 3]: ∎∎∎
|
17260 - 17270 [ 8]: ∎∎∎∎∎∎∎∎
|
17270 - 17280 [ 6]: ∎∎∎∎∎∎
|
17280 - 17290 [ 5]: ∎∎∎∎∎
|
17290 - 17300 [ 7]: ∎∎∎∎∎∎∎
|
Note this only fails on Windows; it seems reliable on Linux.