Saturated disk i/o causes compaction to fail then crash and fill disk

Compaction stuck at retrying over 200 times, then crashes the compactor. This causes the node's disk to fill up 100%.

couchdb:info,2014-06-22T16:29:47.545,ns_1@node73:<0.30281.439>:couch_log:info:39]Compaction file still behind main file (update seq=7344516. compact update seq=7344463). Retrying.

... repeats around 200 times

[couchdb:info,2014-06-22T18:09:23.561,ns_1@node73:<0.30281.439>:couch_log:info:39]Compaction file still behind main file (update seq=7510615. compact update seq=7507648). Retrying.
[couchdb:info,2014-06-22T18:09:59.432,ns_1@node73:<0.30281.439>:couch_log:info:39]Database Bucket/976 compactor died with reason: {noproc,
{gen_server,call,
[<0.9214.952>,
{append_bin,
[<<

from the iostat output in the log files it looks like the device where the vbuckets reside, xvdb is completely saturated for I/O.

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
xvdb 0.00 0.00 0.00 3349.00 0.00 293490.00 87.64 144.09 43.29 0.30 100.00
xvdbp1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4294966.30 0.00 0.00 100.00
xvdbp2 0.00 0.00 0.00 3349.00 0.00 293490.00 87.64 144.09 43.29 0.30 100.00