[longevity] noticed that on a cluster running for few days that disk_commit time 90th percentile takes 10m when index is building

cluster : 6 nodes
50M items , sets per second per node 50 , gets per second per node 500 , one index is building , compaction is set to 30 %
cluster IP : https://docs.google.com/spreadsheet/ccc?key=0AqiOtilGYQU4dEh6V0JTTlItX2phMXg1VDN6OW9pSWc
Aaron,Damien and Chiyoung looked at high disk commit time . we ran iostat and confirmed that one process is doing mostly reads and memcached does not have a chance to do a commit .
Then Aaron killed the indexing process through erlang shell and then observed that memcached started draining very rapidly aftewards and it continued draining even when indexing was restarted so this appears to be a combination of index building running for hours or days and eventually it impacts draining

the environment is a VM byt the disk is not shared by any other VM.