3.1.4 KV Performance Regression

A performance sanity test on 3.1.4-1834 showed performance degraded in the following manner:

• Inability to sustain the same number of front-end ops
• Higher DCP queue on all nodes
• Higher DWQ on all nodes

• Test designed to antagonise the front-end threads, DWQ and the DCP.
• 3 node cluster in GCE, 2 replicas, 300M active docs (900M total)
• ep-engine configured for 128 vbuckets and 8 shards, writer threads.
• One client running a 64 thread 100% set workload

  cbc-pillowfight -p DH_KEY_ -t 64 -m 20 -M 20 -U couchbase://dhaikney-server-1/charlie --no-population -I 300000000 -r100 --rate-limit 1000

See attached screenshots - cmd_set screenshot shows the timeline when each test was run.
You can see the distinct workloads being run and see how, after the upgrade the DCP and DWQ explode for an equivalent number of operations.

Logs after 3.1.4 upgrade:
https://s3.amazonaws.com/cb-customers/davidH/collectinfo-2016-01-15T214910-ns_1%40dhaikney-server-1.c.cb-googbench-101.internal.zip
https://s3.amazonaws.com/cb-customers/davidH/collectinfo-2016-01-15T214910-ns_1%40dhaikney-server-2.c.cb-googbench-101.internal.zip
https://s3.amazonaws.com/cb-customers/davidH/collectinfo-2016-01-15T214910-ns_1%40dhaikney-server-3.c.cb-googbench-101.internal.zip

Logs on vanilla 3.1.3
https://s3.amazonaws.com/cb-customers/davidH/collectinfo-2016-01-15T182926-ns_1%40dhaikney-server-3.c.cb-googbench-101.internal.zip
https://s3.amazonaws.com/cb-customers/davidH/collectinfo-2016-01-15T182926-ns_1%40dhaikney-server-2.c.cb-googbench-101.internal.zip
https://s3.amazonaws.com/cb-customers/davidH/collectinfo-2016-01-15T182926-ns_1%40dhaikney-server-1.c.cb-googbench-101.internal.zip

I captured some top output from each run, will post below.