Memcached killed by O/S (oomkiller) even without any load or replication going on (erlang too much memory)

Live clusters:
c1: http://ec2-184-169-184-36.us-west-1.compute.amazonaws.com:8091/
https://s3.amazonaws.com/bugdb/MB-6962/c1-ec2-184-169-184-36.us-west-1.compute.amazonaws.com.txt.zip
c2: http://ec2-122-248-222-144.ap-southeast-1.compute.amazonaws.com:8091/
https://s3.amazonaws.com/bugdb/MB-6962/c2-ec2-122-248-222-144.ap-southeast-1.compute.amazonaws.com.txt.zip

sbucket:: c1 -> c2
default:: c2 -> c1

The logs showed some errors where the memcached seems to have crashed (with errors listed below), however the nodes seem to have recovered. And as per the timestamp on the errors I had no operation running on any of the nodes on the 2 clusters.

>>

Control connection to memcached on 'ns_1@ec2-175-41-154-35.ap-southeast-1.compute.amazonaws.com' disconnected: {{badmatch,
{error,
closed}},
[

{mc_client_binary, cmd_binary_vocal_recv, 5}

,

{mc_client_binary, select_bucket, 2}

,

{ns_memcached, ensure_bucket, 2}

,

{ns_memcached, handle_info, 2}

,

{gen_server, handle_msg, 5}

,

{proc_lib, init_p_do_apply, 3}

]}

>>

Port server memcached on node 'ns_1@ec2-175-41-154-35.ap-southeast-1.compute.amazonaws.com' exited with status 137. Restarting. Messages: Thu Oct 18 01:41:55.581708 UTC 3: Retry notify CouchDB of update, vbucket=659 rev=12
Thu Oct 18 01:41:55.618654 UTC 3: Retry notify CouchDB of update, vbucket=661 rev=15
Thu Oct 18 01:43:27.778758 UTC 3: Retry notify CouchDB of update, vbucket=292 rev=16
Thu Oct 18 01:43:38.478364 UTC 3: Retry notify CouchDB of update, vbucket=781 rev=13
Thu Oct 18 01:45:40.055815 UTC 3: Retry notify CouchDB of update, vbucket=723 rev=17
Thu Oct 18 01:48:45.465898 UTC 3: Retry notify CouchDB of update, vbucket=674 rev=18
Thu Oct 18 01:54:08.388407 UTC 3: 113 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.383466 UTC 3: 114 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.380544 UTC 3: 112 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.395011 UTC 3: 121 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.395056 UTC 3: 117 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.395080 UTC 3: 125 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.395101 UTC 3: 109 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.395118 UTC 3: 116 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.379943 UTC 3: 111 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.395295 UTC 3: 128 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.395434 UTC 3: 115 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.399902 UTC 3: 123 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.400350 UTC 3: 122 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:08.401823 UTC 3: 124 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:47.349826 UTC 3: 111 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:47.349951 UTC 3: 117 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:47.350795 UTC 3: 120 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:47.353536 UTC 3: 122 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:47.358016 UTC 3: 124 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:47.358265 UTC 3: 99 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:47.358886 UTC 3: 123 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:47.360482 UTC 3: 125 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:47.360639 UTC 3: 110 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:47.369350 UTC 3: 114 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:54:47.370345 UTC 3: 127 Closing connection due to read error: Connection reset by peer
Thu Oct 18 01:55:29.358381 UTC 3: Retry notify CouchDB of update, vbucket=387 rev=7
Thu Oct 18 02:42:36.876817 UTC 3: Retry notify CouchDB of update, vbucket=338 rev=20
Thu Oct 18 02:42:52.987684 UTC 3: Retry notify CouchDB of update, vbucket=1001 rev=20
Thu Oct 18 02:52:52.069148 UTC 3: Retry notify CouchDB of update, vbucket=168 rev=24
Thu Oct 18 02:54:53.776831 UTC 3: Retry notify CouchDB of update, vbucket=34 rev=25
Thu Oct 18 02:58:43.409834 UTC 3: Retry notify CouchDB of update, vbucket=275 rev=27
Thu Oct 18 02:58:59.152387 UTC 3: Retry notify CouchDB of update, vbucket=767 rev=27
Thu Oct 18 03:01:16.673561 UTC 3: Retry notify CouchDB of update, vbucket=106 rev=28
Thu Oct 18 03:03:27.838852 UTC 3: Retry notify CouchDB of update, vbucket=202 rev=29
Thu Oct 18 03:03:42.433468 UTC 3: Retry notify CouchDB of update, vbucket=725 rev=29
Thu Oct 18 03:03:53.612380 UTC 3: Retry notify CouchDB of update, vbucket=1008 rev=29
Thu Oct 18 03:06:08.602318 UTC 3: Retry notify CouchDB of update, vbucket=370 rev=30
Thu Oct 18 03:11:42.116476 UTC 3: Retry notify CouchDB of update, vbucket=965 rev=32
Thu Oct 18 03:13:24.345347 UTC 3: Retry notify CouchDB of update, vbucket=171 rev=33
Thu Oct 18 03:13:49.958533 UTC 3: Retry notify CouchDB of update, vbucket=977 rev=33
Thu Oct 18 03:16:56.512504 UTC 3: Retry notify CouchDB of update, vbucket=1004 rev=34
Thu Oct 18 03:19:00.228060 UTC 3: Retry notify CouchDB of update, vbucket=20 rev=35
Thu Oct 18 03:21:33.283037 UTC 3: Retry notify CouchDB of update, vbucket=38 rev=36
Thu Oct 18 03:21:48.640589 UTC 3: Retry notify CouchDB of update, vbucket=434 rev=36
Thu Oct 18 03:33:38.108769 UTC 3: Retry notify CouchDB of update, vbucket=782 rev=4
Thu Oct 18 03:38:24.256185 UTC 3: Retry notify CouchDB of update, vbucket=273 rev=37
Thu Oct 18 03:38:50.597278 UTC 3: Retry notify CouchDB of update, vbucket=954 rev=37
Thu Oct 18 03:38:50.680538 UTC 3: Retry notify CouchDB of update, vbucket=955 rev=37
Thu Oct 18 03:44:17.957375 UTC 3: Retry notify CouchDB of update, vbucket=1006 rev=39
Thu Oct 18 03:49:15.370507 UTC 3: Retry notify CouchDB of update, vbucket=440 rev=41
Thu Oct 18 03:49:29.837810 UTC 3: Retry notify CouchDB of update, vbucket=910 rev=41
Thu Oct 18 03:54:52.751376 UTC 3: Retry notify CouchDB of update, vbucket=201 rev=43
Thu Oct 18 03:58:03.398599 UTC 3: Retry notify CouchDB of update, vbucket=176 rev=44
Thu Oct 18 04:01:32.506965 UTC 3: Retry notify CouchDB of update, vbucket=717 rev=45
Thu Oct 18 04:01:32.688488 UTC 3: Retry notify CouchDB of update, vbucket=718 rev=45
Thu Oct 18 04:03:53.669318 UTC 3: Retry notify CouchDB of update, vbucket=106 rev=46
Thu Oct 18 04:04:08.714662 UTC 3: Retry notify CouchDB of update, vbucket=516 rev=46
Thu Oct 18 04:06:05.273179 UTC 3: Retry notify CouchDB of update, vbucket=35 rev=47
Thu Oct 18 04:09:08.835290 UTC 3: Retry notify CouchDB of update, vbucket=672 rev=48
Thu Oct 18 04:09:22.000950 UTC 3: Retry notify CouchDB of update, vbucket=1010 rev=48
Thu Oct 18 04:09:22.116066 UTC 3: Retry notify CouchDB of update, vbucket=1011 rev=48
Thu Oct 18 04:17:03.322536 UTC 3: Retry notify CouchDB of update, vbucket=129 rev=51
Thu Oct 18 04:25:00.483079 UTC 3: Retry notify CouchDB of update, vbucket=276 rev=9
Thu Oct 18 04:28:11.313991 UTC 3: Retry notify CouchDB of update, vbid=434 rev=9
Thu Oct 18 04:29:56.353390 UTC 3: Retry notify CouchDB of update, vbid=656 rev=4
Thu Oct 18 04:32:10.875985 UTC 3: Retry notify CouchDB of update, vbucket=722 rev=10
Thu Oct 18 04:39:32.988816 UTC 3: Retry notify CouchDB of update, vbucket=513 rev=52
Thu Oct 18 04:42:38.588116 UTC 3: Retry notify CouchDB of update, vbucket=887 rev=53
Thu Oct 18 04:45:18.460222 UTC 3: Retry notify CouchDB of update, vbucket=21 rev=54
Thu Oct 18 04:45:36.239587 UTC 3: Retry notify CouchDB of update, vbucket=439 rev=54
Thu Oct 18 04:45:56.723741 UTC 3: Retry notify CouchDB of update, vbucket=957 rev=54
Thu Oct 18 04:45:56.942498 UTC 3: Retry notify CouchDB of update, vbucket=958 rev=54
Thu Oct 18 04:49:16.963767 UTC 3: Retry notify CouchDB of update, vbucket=983 rev=55
Thu Oct 18 04:58:26.052117 UTC 3: Retry notify CouchDB of update, vbucket=126 rev=58
Thu Oct 18 04:59:06.744012 UTC 3: Retry notify CouchDB of update, vbucket=1002 rev=58
Thu Oct 18 05:02:09.367206 UTC 3: Retry notify CouchDB of update, vbucket=12 rev=59
Thu Oct 18 05:02:26.076587 UTC 3: Retry notify CouchDB of update, vbucket=296 rev=59
Thu Oct 18 05:02:50.390077 UTC 3: Retry notify CouchDB of update, vbucket=834 rev=59
Thu Oct 18 05:06:12.621689 UTC 3: Retry notify CouchDB of update, vbucket=191 rev=60
Thu Oct 18 05:06:43.614801 UTC 3: Retry notify CouchDB of update, vbucket=1004 rev=60
Thu Oct 18 05:06:43.664449 UTC 3: Retry notify CouchDB of update, vbucket=1005 rev=60
Thu Oct 18 05:09:21.193324 UTC 3: Retry notify CouchDB of update, vbucket=101 rev=61
Thu Oct 18 05:12:59.101229 UTC 3: Retry notify CouchDB of update, vbucket=658 rev=59
Thu Oct 18 05:12:59.643282 UTC 3: Retry notify CouchDB of update, vbucket=661 rev=62
Thu Oct 18 05:13:24.149292 UTC 3: Retry notify CouchDB of update, vbucket=985 rev=59
Thu Oct 18 05:19:37.670807 UTC 3: Retry notify CouchDB of update, vbucket=520 rev=64
Thu Oct 18 05:47:15.857101 UTC 3: Retry notify CouchDB of update, vbucket=809 rev=10
Thu Oct 18 05:59:33.695917 UTC 3: Retry notify CouchDB of update, vbucket=277 rev=67
Thu Oct 18 05:59:47.744716 UTC 3: Retry notify CouchDB of update, vbucket=515 rev=67
Thu Oct 18 06:00:16.791461 UTC 3: Retry notify CouchDB of update, vbucket=910 rev=67
Thu Oct 18 06:06:15.058250 UTC 3: Retry notify CouchDB of update, vbucket=20 rev=68
Thu Oct 18 06:07:33.633925 UTC 3: Retry notify CouchDB of update, vbucket=951 rev=68
Thu Oct 18 06:12:59.540660 UTC 3: Retry notify CouchDB of update, vbucket=17 rev=69
Thu Oct 18 06:25:32.613461 UTC 3: Retry notify CouchDB of update, vbucket=662 rev=70