Indexer crashed multiple times

Build 2009

6-hour test running with moderate write load (12k/sec). Towards the very end, the indexer seems to have crashed and restarted a few times with different messages, seems like every hour:

1st:
Service 'indexer' exited with status 1. Restarting. Messages: 2016-04-05T08:02:49.728+00:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1306510000
2016-04-05T08:02:49.731+00:00 [Info] logWriterStat:: 12700678987715329722 FlushedCount 156230000 QueuedCount 2964
2016-04-05T08:02:54.428+00:00 [Info] memstats

{"Alloc":22350014008, "TotalAlloc":564469253848, "Sys":34526916120, "Lookups":18648, "Mallocs":17767307455,"Frees":17413373120, "HeapAlloc":22350014008, "HeapSys":32903168000, "HeapIdle":7959461888, "HeapInuse":24943706112,"HeapReleased":245899264, "HeapObjects":353934335,"GCSys":1036216320, "LastGC":1459843259554470173,"PauseTotalNs":237614839, "PauseNs":[], "NumGC":111}

2016-04-05T08:02:52.148+00:00 [Info] connected with 2 indexers
[goport] 2016/04/05 08:03:03 /opt/couchbase/bin/indexer terminated: signal: killed

2nd (1hr later):
Service 'indexer' exited with status 1. Restarting. Messages: 2016-04-05T08:58:30.189+00:00 [Info] logWriterStat:: 17945607957096596504 FlushedCount 152900000 QueuedCount 0
2016-04-05T08:58:30.188+00:00 [Info] logWriterStat:: 12700678987715329722 FlushedCount 152900000 QueuedCount 3
2016-04-05T08:58:30.564+00:00 [Info] logWriterStat:: 18391313911218615949 FlushedCount 152900000 QueuedCount 554
2016-04-05T08:58:30.642+00:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM default SnapType NO_SNAP
[goport] 2016/04/05 08:58:36 /opt/couchbase/bin/indexer terminated: signal: killed

3rd (1hr later):
Service 'indexer' exited with status 1. Restarting. Messages: 2016-04-05T10:04:01.142+00:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM default SnapType NO_SNAP
2016-04-05T10:04:01.108+00:00 [Info] logWriterStat:: 18391313911218615949 FlushedCount 151680000 QueuedCount 384
2016-04-05T10:04:01.297+00:00 [Info] logWriterStat:: 14239020311441188784 FlushedCount 151640000 QueuedCount 40384
2016-04-05T10:04:01.563+00:00 [Info] logWriterStat:: 14239020311441188784 FlushedCount 151650000 QueuedCount 30384
[goport] 2016/04/05 10:04:07 /opt/couchbase/bin/indexer terminated: signal: killed

4th:(1hr later)
Service 'indexer' exited with status 1. Restarting. Messages: gs 0x0

---

[goport] 2016/04/05 10:59:51 /opt/couchbase/bin/indexer terminated: signal: aborted

5th: (1 hr later)
Service 'indexer' exited with status 1. Restarting. Messages: 2016-04-05T11:54:02.102+00:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1266890000
2016-04-05T11:54:01.862+00:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM default SnapType NO_SNAP
2016-04-05T11:54:02.944+00:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1266900000
2016-04-05T11:54:03.285+00:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1266910000
[goport] 2016/04/05 11:54:08 /opt/couchbase/bin/indexer terminated: signal: killed

6th:
Service 'indexer' exited with status 1. Restarting. Messages: 2016-04-05T12:46:51.386+00:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1235230000
2016-04-05T12:46:51.391+00:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1235240000
2016-04-05T12:46:54.129+00:00 [Info] DATP[->dataport ":9105"] DATP -> Indexer 0.902669% blocked
2016-04-05T12:46:53.789+00:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1235250000
[goport] 2016/04/05 12:47:03 /opt/couchbase/bin/indexer terminated: signal: killed

7th:
Service 'indexer' exited with status 1. Restarting. Messages: /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:262 +0x163 fp=0xcfd43ba838 sp=0xcfd43ba810
runtime.goparkunlock(0xc8237e6bf8, 0x10b0ed0, 0x9, 0x16, 0x3)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:268 +0x54 fp=0xcfd43ba870 sp=0xcfd43ba838
runtime.chansend(0xcb4ac0, 0xc8237e6ba0, 0xcfd43ba958, 0xcf4b272801, 0x4d81ba, 0x8dbd912)
[goport] 2016/04/05 13:40:13 /opt/couchbase/bin/indexer terminated: signal: killed

Logs are at:

https://s3.amazonaws.com/cb-engineering/perry/samsung/collectinfo-2016-04-05T135115-ns_1%40ec2-54-215-123-219.us-west-1.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-engineering/perry/samsung/collectinfo-2016-04-05T135115-ns_1%40ec2-54-215-27-253.us-west-1.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-engineering/perry/samsung/collectinfo-2016-04-05T135115-ns_1%40ec2-54-219-195-14.us-west-1.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-engineering/perry/samsung/collectinfo-2016-04-05T135115-ns_1%40ec2-54-219-197-161.us-west-1.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-engineering/perry/samsung/collectinfo-2016-04-05T135115-ns_1%40ec2-54-219-255-167.us-west-1.compute.amazonaws.com.zip
https://s3.amazonaws.com/cb-engineering/perry/samsung/collectinfo-2016-04-05T135115-ns_1%40ec2-54-219-64-21.us-west-1.compute.amazonaws.com.zip