Investigate persist majority throughput when varying changing shard and thread count

I ran some experiments where I changed num_auxio_threads and max_num_shards. These paramter seems to have little affect unless they are set to low values like 1 and 2 or to extremely high values like 512. For the tests, I set the two parameters to the exact same value via the following: 

curl -i -u Administrator:password --data 'ns_bucket:update_bucket_props("bucket-1", [\{extra_config_string, "num_auxio_threads=64;max_num_shards=64"}]).' http://localhost:8091/diag/eval

and then each couchbase node was restarted via:

sudo service couchbase-server restart

Here are some results.

no mod: 7843  - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4558_access_70b1

1: 2463 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_9e2e

2: 4538 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_3f16

4: 7674 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_2d5b

8: 7990 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_14b2

12: 8089 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_2b33

15: 7708 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_a354

16: 8244 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_7ae6

20: 7975 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_db62

32: 7926 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_ec74

64: 7919 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_1267

128: 7257 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_38e5

256: 7536 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_d4aa

512: memcached crashes

I have collect logs for 3 parameters:

16: 8047 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_cbf7
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.11.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.12.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.13.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.14.zip

32: 7923 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_9f7f
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.11.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.12.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.13.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.14.zip

64: 7721 - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-4676_access_9855
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.11.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.12.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.13.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.133.14.zip

These tests were all run on ares: https://github.com/couchbase/perfrunner/blob/master/clusters/ares.spec

And the test is: https://github.com/couchbase/perfrunner/blob/master/tests/durability/throughput/kv/enhanced/kv_max_ops_writes_512_durability_persist_majority.test

One thing to note is that we are using libcouchbase 3.0.0-alpha.5. We are yet to upgrade to beta since beta.1 had a bug in it where libcouchbase was not picking up libevent properly. The fix will go into beta.2 which will be released late next week. This could be a potential issue and I will rerun the tests once beta.2 is released.