HiDD: Scaling Bucket Eviction

When we ran YCSB tests on NVMe cluster, we saw average load throughput was limited at 150K ops/s. It happened to Magma and Couchstore.

The ops rate was about 400K ops/s at the beginning, and it dropped to 150K quickly.

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=magma-nvme_700-3154_custom_load_953b&label=magma&snapshot=magma-nvme_700-3154_custom_load_b32b&label=couchstore#7d953b61958286f86d6bacb30330e47f

When the memory was filled up, it triggered the tmp oom, and ops rate was throttled

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=magma-nvme_700-3154_custom_load_953b&label=magma&snapshot=magma-nvme_700-3154_custom_load_b32b&label=couchstore#629579b6bf3a05996bd8689da648aa19

We observed the ejections rate was 150K/s so the throughput dropped to 150k/s.
http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=magma-nvme_700-3154_custom_load_953b&label=magma&snapshot=magma-nvme_700-3154_custom_load_b32b&label=couchstore#020da40581d2c3c889ce8f943193a3e8

Test: Single node YCSB load (replica_number = 0)

Build: 7.0.0-3154

Magma: http://perf.jenkins.couchbase.com/job/magma-nvme-ycsb/321/ 

Log: https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-magma-nvme-ycsb-321/172.23.100.135.zip 

Couchstore: http://perf.jenkins.couchbase.com/job/magma-nvme-ycsb/322/

Log: https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-magma-nvme-ycsb-322/172.23.100.135.zip