dcp observe replication regression between 4.5.1-2801 and 4.5.1-2802

This has been identified as part of the daily sanity:
python -u perfSanity/scripts/perf_regression_runner_alpha.py -e -v 4.5.1-2802 -r 2016-07-14:13:18 -q "testName='dcp_observe'" -n -e

This test measures the replication latency. With build 2801 the 95% percentile replica is 5 msec, and in build 2802 it is 800 msec. There are a number of epengine related changes that went into this build.

This issue is not seen in the weekly crank, possibly it is showing up with the daily sanity because the servers have less RAM and fewer cores (though is a supported configuration).

Initially assigning to Jim because he has worked on similar issues in the past.

Jim, if this does indeed belong to you let know a good time to meet to discuss this.

I will attached logs shortly.