Details
-
Task
-
Resolution: Fixed
-
Major
-
master
-
None
Description
Background
Couchbase Nightly Test (CBNT) - https://hub.internal.couchbase.com/confluence/display/cbeng/CBNT+-+KV+Engine+Performance+Commit+Validation - is a microbenchmark test runner which runs microbenchmark performance tests as part of commit validation.
However, it was never "made voting" when originally developed, and then when we merged the memcached & ep-engine repos it was not updated 
As such it's been unused for a while.
However, recent perf regressions (e.g. MB-27514) have (re) highlighted the value of such a test framework, and we should get it working again.
Task
TL;DR: Fix so spock and master branches work again; and report to Gerrit.
Roughly what needs doing is the following:
- Update CBNT to use just a single project (kv_engine).
- Update / create new Jenkins job (http://factory.couchbase.com/view/build/view/commit_validation/view/memcached/job/memcached-cv-perf/) to watch the kv_engine project.
- Side task: factory is the "old" CV host and ideally no new jobs should be created on it; so it would be good to move the new kv_engine CBNT jobs (and their dedicated builder) over to cv.jenkins.couchbase.com.
- Review the set of benchmarks which CBNT runs; adding any new ones (which have stable scores) and disabling any which arn't stable so we can rely on the output of CBNT (minimise any false regressions).
- Make the new kv_engine CBNT job voting, so regressions are seen via Gerrit.