Dec 13 17:56:15 ip-172-31-37-130 kernel: 13_scheduler invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Dec 13 17:56:15 ip-172-31-37-130 kernel: CPU: 12 PID: 19955 Comm: 13_scheduler Not tainted 5.10.75-79.358.amzn2.aarch64 #1 Dec 13 17:56:15 ip-172-31-37-130 kernel: Hardware name: Amazon EC2 c6gd.4xlarge/, BIOS 1.0 11/1/2018 Dec 13 17:56:15 ip-172-31-37-130 kernel: Call trace: Dec 13 17:56:15 ip-172-31-37-130 kernel: dump_backtrace+0x0/0x1fc Dec 13 17:56:16 ip-172-31-37-130 kernel: show_stack+0x24/0x30 Dec 13 17:56:16 ip-172-31-37-130 kernel: dump_stack+0xd0/0x12c Dec 13 17:56:16 ip-172-31-37-130 kernel: dump_header+0x50/0x1f8 Dec 13 17:56:16 ip-172-31-37-130 kernel: oom_kill_process+0x228/0x238 Dec 13 17:56:16 ip-172-31-37-130 kernel: out_of_memory.part.0+0xbc/0x2c8 Dec 13 17:56:16 ip-172-31-37-130 kernel: out_of_memory+0x54/0xa8 Dec 13 17:56:16 ip-172-31-37-130 kernel: __alloc_pages_may_oom+0x118/0x1a0 Dec 13 17:56:16 ip-172-31-37-130 kernel: __alloc_pages_slowpath.constprop.0+0x524/0x7b0 Dec 13 17:56:16 ip-172-31-37-130 kernel: __alloc_pages_nodemask+0x2b8/0x30c Dec 13 17:56:16 ip-172-31-37-130 kernel: alloc_pages_current+0x94/0x14c Dec 13 17:56:16 ip-172-31-37-130 kernel: pagecache_get_page+0x24c/0x3b8 Dec 13 17:56:16 ip-172-31-37-130 kernel: filemap_fault+0x528/0x6e0 Dec 13 17:56:16 ip-172-31-37-130 kernel: __xfs_filemap_fault+0x70/0x2b0 Dec 13 17:56:16 ip-172-31-37-130 kernel: xfs_filemap_fault+0x60/0x6c Dec 13 17:56:16 ip-172-31-37-130 kernel: __do_fault+0x44/0x114 Dec 13 17:56:16 ip-172-31-37-130 kernel: do_fault+0xc4/0x1d8 Dec 13 17:56:16 ip-172-31-37-130 kernel: handle_pte_fault+0x5c/0x200 Dec 13 17:56:16 ip-172-31-37-130 kernel: __handle_mm_fault+0x1dc/0x37c Dec 13 17:56:16 ip-172-31-37-130 kernel: handle_mm_fault+0xd0/0x240 Dec 13 17:56:16 ip-172-31-37-130 kernel: do_page_fault+0x154/0x420 Dec 13 17:56:16 ip-172-31-37-130 kernel: do_translation_fault+0xbc/0xe0 Dec 13 17:56:16 ip-172-31-37-130 kernel: do_mem_abort+0x4c/0xb8 Dec 13 17:56:16 ip-172-31-37-130 kernel: el0_ia+0x64/0xb4 Dec 13 17:56:16 ip-172-31-37-130 kernel: el0_sync_handler+0x11c/0x120 Dec 13 17:56:16 ip-172-31-37-130 kernel: Mem-Info: Dec 13 17:56:16 ip-172-31-37-130 kernel: active_anon:2405 inactive_anon:7951701 isolated_anon:0#012 active_file:82 inactive_file:1197 isolated_file:0#012 unevictable:1 dirty:1 writeback:0#012 slab_reclaimable:14280 slab_unreclaimable:14012#012 mapped:287 shmem:110 pagetables:17561 bounce:0#012 free:43367 free_pcp:638 free_cma:892 Dec 13 17:56:16 ip-172-31-37-130 kernel: Node 0 active_anon:9620kB inactive_anon:31806804kB active_file:328kB inactive_file:5028kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:1308kB dirty:4kB writeback:0kB shmem:440kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:11472kB all_unreclaimable? no Dec 13 17:56:16 ip-172-31-37-130 kernel: Node 0 DMA free:124136kB min:1292kB low:2216kB high:3140kB reserved_highatomic:0KB active_anon:24kB inactive_anon:798992kB active_file:4kB inactive_file:96kB unevictable:0kB writepending:0kB present:1048576kB managed:942328kB mlocked:0kB pagetables:1244kB bounce:0kB free_pcp:8kB local_pcp:8kB free_cma:3568kB Dec 13 17:56:16 ip-172-31-37-130 kernel: lowmem_reserve[]: 0 0 30590 30590 Dec 13 17:56:16 ip-172-31-37-130 kernel: Node 0 Normal free:47820kB min:43760kB low:75084kB high:106408kB reserved_highatomic:2048KB active_anon:9596kB inactive_anon:31007812kB active_file:2344kB inactive_file:3740kB unevictable:4kB writepending:4kB present:31965184kB managed:31324612kB mlocked:4kB pagetables:69000kB bounce:0kB free_pcp:3020kB local_pcp:180kB free_cma:0kB Dec 13 17:56:16 ip-172-31-37-130 kernel: lowmem_reserve[]: 0 0 0 0 Dec 13 17:56:16 ip-172-31-37-130 kernel: Node 0 DMA: 144*4kB (UMEC) 159*8kB (UC) 291*16kB (UEC) 242*32kB (UEC) 201*64kB (UEC) 180*128kB (UE) 121*256kB (UE) 58*512kB (UE) 13*1024kB (E) 0*2048kB 0*4096kB = 124136kB Dec 13 17:56:16 ip-172-31-37-130 kernel: Node 0 Normal: 493*4kB (UMEH) 797*8kB (UMEH) 1122*16kB (UMEH) 323*32kB (UMEH) 131*64kB (UMEH) 6*128kB (MH) 4*256kB (MH) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46812kB Dec 13 17:56:16 ip-172-31-37-130 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Dec 13 17:56:16 ip-172-31-37-130 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=32768kB Dec 13 17:56:16 ip-172-31-37-130 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Dec 13 17:56:16 ip-172-31-37-130 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=64kB Dec 13 17:56:16 ip-172-31-37-130 kernel: 2139 total pagecache pages Dec 13 17:56:16 ip-172-31-37-130 kernel: 0 pages in swap cache Dec 13 17:56:16 ip-172-31-37-130 kernel: Swap cache stats: add 0, delete 0, find 0/0 Dec 13 17:56:16 ip-172-31-37-130 kernel: Free swap = 0kB Dec 13 17:56:16 ip-172-31-37-130 kernel: Total swap = 0kB Dec 13 17:56:16 ip-172-31-37-130 kernel: 8253440 pages RAM Dec 13 17:56:16 ip-172-31-37-130 kernel: 0 pages HighMem/MovableOnly Dec 13 17:56:16 ip-172-31-37-130 kernel: 186705 pages reserved Dec 13 17:56:16 ip-172-31-37-130 kernel: 16384 pages cma reserved Dec 13 17:56:16 ip-172-31-37-130 kernel: Tasks state (memory values in pages): Dec 13 17:56:16 ip-172-31-37-130 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 478] 0 478 5390 86 65536 0 0 systemd-journal Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 497] 0 497 20694 68 61440 0 0 lvmetad Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 503] 0 503 3733 216 61440 0 -1000 systemd-udevd Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 667] 0 667 4445 112 53248 0 -1000 auditd Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 691] 0 691 20376 99 53248 0 0 irqbalance Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 694] 81 694 2166 136 57344 0 -900 dbus-daemon Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 699] 32 699 2480 134 57344 0 0 rpcbind Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 703] 0 703 1174 104 49152 0 0 systemd-logind Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 711] 999 711 643 41 49152 0 0 lsmd Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 712] 998 712 80472 206 110592 0 0 rngd Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 714] 997 714 20634 138 57344 0 0 chronyd Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 910] 0 910 4896 514 77824 0 0 dhclient Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 954] 0 954 4896 508 77824 0 0 dhclient Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 1065] 0 1065 42198 118 86016 0 0 gssproxy Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 1092] 0 1092 5244 261 81920 0 0 master Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 1096] 89 1096 5287 257 86016 0 0 qmgr Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 1146] 0 1146 66338 301 188416 0 0 rsyslogd Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 1147] 0 1147 179673 2227 131072 0 0 amazon-ssm-agen Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 1153] 0 1153 4552 260 77824 0 -1000 sshd Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 1163] 0 1163 1005 51 49152 0 0 atd Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 1165] 0 1165 28721 152 53248 0 0 crond Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 1185] 0 1185 28305 31 45056 0 0 agetty Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 1186] 0 1186 28217 32 53248 0 0 agetty Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 1302] 0 1302 182650 3677 151552 0 0 ssm-agent-worke Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 6729] 1000 6729 771 39 40960 0 0 epmd Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 14269] 0 14269 5253 330 81920 0 0 sshd Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 14301] 1000 14301 5422 489 81920 0 0 sshd Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 14302] 1000 14302 28748 274 69632 0 0 bash Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 14326] 1000 14326 1031 70 40960 0 0 tmux Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 14328] 1000 14328 4321 3306 65536 0 0 tmux Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 14329] 1000 14329 28756 309 57344 0 0 bash Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19334] 1000 19334 28723 280 61440 0 0 bash Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19419] 1000 19419 28781 324 57344 0 0 bash Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 8907] 1000 8907 7603584 7282351 58626048 0 0 cbc-pillowfight Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 9126] 1000 9126 28789 333 53248 0 0 bash Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 10050] 1000 10050 37441 3861 126976 0 0 emacs Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19786] 1000 19786 33512 2079 106496 0 0 python3 Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19846] 1000 19846 1303018 9657 528384 0 0 beam.smp Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19868] 1000 19868 502 16 40960 0 0 erl_child_setup Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19914] 1000 19914 1595 841 57344 0 0 gosecrets Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19920] 1000 19920 1327412 24821 770048 0 0 beam.smp Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19942] 1000 19942 503 17 40960 0 0 erl_child_setup Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19987] 1000 19987 28453 53 53248 0 0 sh Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19989] 1000 19989 468 20 36864 0 0 memsup Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19990] 1000 19990 501 18 40960 0 0 cpu_sup Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 19994] 1000 19994 1092509 16184 651264 0 0 beam.smp Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20001] 1000 20001 504 18 40960 0 0 erl_child_setup Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20048] 1000 20048 28454 54 57344 0 0 sh Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20050] 1000 20050 469 21 36864 0 0 memsup Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20051] 1000 20051 502 18 36864 0 0 cpu_sup Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20062] 1000 20062 772 25 45056 0 0 inet_gethost Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20063] 1000 20063 772 30 45056 0 0 inet_gethost Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20064] 1000 20064 2462 289 69632 0 0 godu Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20068] 1000 20068 28453 52 53248 0 0 sh Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20071] 1000 20071 828 93 53248 0 0 godu Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20077] 1000 20077 1639 185 57344 0 0 goport Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20082] 1000 20082 273281 10497 327680 0 0 prometheus Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20085] 1000 20085 771 30 40960 0 0 inet_gethost Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20086] 1000 20086 771 32 40960 0 0 inet_gethost Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20091] 1000 20091 141095 221 122880 0 0 saslauthd-port Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20098] 1000 20098 1337 69 45056 0 0 sigar_port Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20104] 1000 20104 1877 749 57344 0 0 goport Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20110] 1000 20110 755612 6098 622592 0 0 goxdcr Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20128] 1000 20128 1221 270 53248 0 0 goport Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20134] 1000 20134 353389 1697 225280 0 0 projector Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20142] 1000 20142 1776314 576103 6610944 0 0 memcached Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 22391] 89 22391 5267 252 81920 0 0 pickup Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 25517] 0 25517 28755 277 57344 0 0 dhclient-script Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 25656] 0 25656 29570 94 73728 0 0 ps Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 25724] 0 25724 29028 171 53248 0 0 crond Dec 13 17:56:16 ip-172-31-37-130 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=cbc-pillowfight,pid=8907,uid=1000 Dec 13 17:56:16 ip-172-31-37-130 kernel: Out of memory: Killed process 8907 (cbc-pillowfight) total-vm:30414336kB, anon-rss:29129404kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:57252kB oom_score_adj:0 Dec 13 17:56:16 ip-172-31-37-130 systemd: Created slice User Slice of root. Dec 13 17:56:16 ip-172-31-37-130 ec2net: [get_meta] Querying IMDS for meta-data/network/interfaces/macs/0a:fa:69:31:5d:e9/local-ipv4s Dec 13 17:56:16 ip-172-31-37-130 systemd: Started Session 46 of user root. Dec 13 17:56:16 ip-172-31-37-130 dhclient[910]: bound to 172.31.37.130 -- renewal in 627 seconds. Dec 13 17:56:16 ip-172-31-37-130 ec2net: [get_meta] Getting token for IMDSv2. Dec 13 17:56:16 ip-172-31-37-130 systemd: Removed slice User Slice of root. Dec 13 17:56:16 ip-172-31-37-130 ec2net: [get_meta] Trying to get http://169.254.169.254/latest/meta-data/network/interfaces/macs/0a:fa:69:31:5d:e9/local-ipv4s Dec 13 17:56:16 ip-172-31-37-130 ec2net: [remove_aliases] Removing aliases of eth0 Dec 13 17:56:18 ip-172-31-37-130 kernel: oom_reaper: reaped process 8907 (cbc-pillowfight), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB