[system test] Huge(10x) memory spike on indexer process on reboot cluster

1. Set compaction to default 30% for indexes
2. Load tpcc workload ~ 10M items on few buckets
3. Wait for indexes to be built
4. Cluster in steady state ( no load, no queries) shows indexer process at 3G ( which seems high as well)
5. Reboot the cluster — The Ram quota on the indexer node is nearly 100%, the indexer process hovers at 19G of memory on both the nodes.

• The CPU on the system is high, but this is from the index-recovery mode?

Logs

10.6.2.164:/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2015-04-28T202328-ns_1@10.6.2.164.zip
10.6.2.167:/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2015-04-28T202328-ns_1@10.6.2.167.zip
10.6.2.168:/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2015-04-28T202328-ns_1@10.6.2.168.zip
10.6.2.194:/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2015-04-28T202328-ns_1@10.6.2.194.zip
10.6.2.195:/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2015-04-28T202328-ns_1@10.6.2.195.zip
10.6.2.234:/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2015-04-28T202328-ns_1@10.6.2.234.zip

screenshots from atop on the indexer node.