Issue with recovering RAM after Hard OOM error

When writing a flurry of large documents to Couchbase, when writing docs until Hard OOM is observed, we see that Total User Data In Ram is a multiple of (Active + Replica + Pending ) User Data in RAM, and that this RAM is not released

Both 3.0.3 and 4.0.0-3570 were tested.

Steps to reproduce:

1. Set up 2 node cluster
2. Install libcouchbase on one node:

wget http://packages.couchbase.com/clients/c/couchbase-csdk-setup
sudo perl couchbase-csdk-setup

3. Set up bucket with 1 replica and Value ejection, 500 MB quota
4. Run pillowfight:

/usr/bin/cbc-pillowfight -B 1 -t 1 -r 100 -m 5120000 -M 5120000 -p MEM:: -U couchbase://localhost/default

5. When Hard OOM is observed from pillowfight and from UI, stop pillowfight.

Operation(s) failed: [0x8] The server is out of memory. Try again later

6. Observe metrics:

• curr_items is low
• mem_used is much larger than couch_docs_data_size
• In 3.x, steady value for ep_dcp_replica_items_remaining ( Intra-Replication queue )
• DCP backoffs/sec increasing
• Significant free RAM
• Very low metadata usage