[Projector][GSI Volume Test] projector OOM crash

Projector crashes during volume testing at ~9.5M docs.

Service 'projector' exited with status 137. Restarting. Messages:
2019-10-30T11:13:30.403-07:00 [Info] memstats

{"Alloc":12835040992, "TotalAlloc":501474914896, "Sys":15916891080, "Lookups":352, "Mallocs":6435022197,"Frees":6360526893, "HeapAlloc":12835040992, "HeapSys":15102771200, "HeapIdle":1088233472, "HeapInuse":14014537728,"HeapReleased":0, "HeapObjects":74495304,"GCSys":543643648, "LastGC":1572459057591351451,"PauseTotalNs":1511960792, "PauseNs":[], "NumGC":241}

2019-10-30T11:14:31.924-07:00 [Info] memstats

{"Alloc":11968056248, "TotalAlloc":504215705840, "Sys":15918463944, "Lookups":352, "Mallocs":6470816228,"Frees":6402128189, "HeapAlloc":11968056248, "HeapSys":15102541824, "HeapIdle":1759313920, "HeapInuse":13343227904,"HeapReleased":273825792, "HeapObjects":68688039,"GCSys":545421312, "LastGC":1572459244908791935,"PauseTotalNs":1532496377, "PauseNs":[20535585], "NumGC":242}

2019-10-30T11:15:35.460-07:00 [Info] memstats

{"Alloc":12000283456, "TotalAlloc":506514499128, "Sys":15918463944, "Lookups":352, "Mallocs":6500429595,"Frees":6430330758, "HeapAlloc":12000283456, "HeapSys":15101853696, "HeapIdle":1881628672, "HeapInuse":13220225024,"HeapReleased":270876672, "HeapObjects":70098837,"GCSys":545421312, "LastGC":1572459244908791935,"PauseTotalNs":1532496377, "PauseNs":[], "NumGC":242}

[goport(/opt/couchbase/bin/projector)] 2019/10/30 11:15:53 child process exited with status 137

There are 128 indexes, built before the data loading occurs. There are 10 fakeit procceses loading a total of 25M docs per bucket, each bucket got ~4.7M docs before projector crashed. Projector crashes were noted in the UI logs at 11:15:53 AM (node 172.23.97.74) , 11:20:34 AM (node 172.23.96.14), 11:22:59 AM (node 172.23.96.122).

There are
10 kv nodes (172.23.97.74,172.23.96.48,172.23.96.209,172.23.96.207,172.23.96.191,172.23.96.190,172.23.96.183,172.23.96.18,172.23.96.14,172.23.96.122)

12 index nodes(172.23.104.68, 172.23.104.78, 172.23.96.210, 172.23.96.212, 172.23.96.214, 172.23.96.215, u172.23.96.216, 172.23.96.219, 172.23.96.220, 172.23.96.221, 172.23.96.223, 172.23.96.254)

2 n1ql nodes(172.23.104.113,172.23.104.114)

https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.104.68.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.104.78.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.210.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.212.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.214.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.215.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.216.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.219.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.220.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.221.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.223.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.254.zip - index node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.48.zip - kv node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.97.74.zip - kv node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.122.zip - kv node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.14.zip - kv node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.18.zip - kv node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.183.zip - kv node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.190.zip - kv node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.191.zip - kv node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.207.zip - kv node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.96.209.zip - kv node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.104.113.zip -n1ql node
https://cb-jira.s3.us-east-2.amazonaws.com/logs/gsivoltesting/collectinfo-2019-10-30T185925-ns_1%40172.23.104.114.zip - n1ql node