Compaction of large vbuckets causes a significant increase in memory usage

If vbuckets are large (e.g., 100M or 200M keys per vbucket), then compaction of individual vbuckets triggers a noticeable spike in memory usage.

For instance, this relatively small vbucket temporary increased memory usage by ~15GB:

DB Info (/data/bucket-1/0.couch.2) - header at 33279463424

   file format version: 12

   update_seq: 100000000

   purge_seq: 0

   crc: CRC-32C

   doc count: 100000000

   deleted doc count: 0

   data size: 30.993 GB

   B-tree size: 7.045 GB

   total disk size: 30.994 GB

Larger vbuckets sent my 64GB servers to swap.

The amount of RAM allocated during the compaction process depends on the size of the index. I am pretty sure we can come up with a good formula for that.

Steps:

• Create a cluster with 4 nodes, 1 bucket, and 4 vbuckets (no replicas)
• Disable auto-compaction
• Load 800M items (~512B)
• Trigger compaction

Some graphs: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hera_510-1137_compact_5197