GSI: Very high indexer RAM usage under heavy load

Build 1615

I have a pretty high load going through a cluster and I see very high RAM usage by one indexer process (>10G and growing) though the quota is set to 2500mb. I suspect we're heading towards an OOM situation on the box, but even if not, we would be if it didn't have enough spare RAM.

These are ForestDB backed GSI, not memory-optimized.

Logs are here:
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214507-ns_1%40ec2-52-12-176-202.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214507-ns_1%40ec2-54-185-192-98.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214507-ns_1%40ec2-54-185-235-98.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214507-ns_1%40ec2-54-185-63-41.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214507-ns_1%40ec2-54-203-124-219.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214507-ns_1%40ec2-54-203-170-211.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214507-ns_1%40ec2-54-214-70-167.us-west-2.compute.amazonaws.com.zip

The indexer node is: ec2-54-185-63-41.us-west-2.compute.amazonaws.com

And another set of logs are here:
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214711-ns_1%40ec2-54-185-246-33.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214711-ns_1%40ec2-54-188-24-147.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214711-ns_1%40ec2-54-188-55-160.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214711-ns_1%40ec2-54-188-86-59.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214711-ns_1%40ec2-54-189-132-205.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214711-ns_1%40ec2-54-203-189-1.us-west-2.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-customers/perry/indexerram/collectinfo-2016-03-02T214711-ns_1%40ec2-54-212-51-126.us-west-2.compute.amazonaws.com.zip

The indexer node is: ec2-54-189-132-205.us-west-2.compute.amazonaws.com