Memory used stuck above high water mark at ~25% resident

We were running a ycsb test with magma (http://perf.jenkins.couchbase.com/job/rhea-ycsb/263/console) and 128 shards(to increase the number of reader threads). The test loads 100 million 6kb items with a single replica and updates the whole data set couple of times over. The memory usage spikes over the high water mark on a few nodes causing ops to significantly drop and gets stuck there around an hour into the update phase, this is despite being at ~25% resident.

The toy build being used is branch-master + http://review.couchbase.org/#/c/114897/5 + http://review.couchbase.org/#/c/111966/ (this patch turns off magma's write cache memory accounting which is a max of 128MB per shard)checked out on kv_engine.

Logs -
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-rhea-ycsb-263/172.23.97.25.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-rhea-ycsb-263/172.23.97.26.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-rhea-ycsb-263/172.23.97.28.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-rhea-ycsb-263/172.23.97.29.zip

Supportal -
https://supportal.couchbase.com/customer/apaar_gupta/cluster/8e3deb56914a4003d7399760536fa238