Jan 16 00:00:01 kvm-s63702 kernel: [474381.765945] traps: atop[4821] trap divide error ip:4073e6 sp:7fffcd7d13c0 error:0 in atop[400000+26000] Jan 16 05:42:31 kvm-s63702 kernel: [494931.278853] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.278861] indexer cpuset=/ mems_allowed=0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.278881] CPU: 7 PID: 6666 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 16 05:42:31 kvm-s63702 kernel: [494931.278882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 16 05:42:31 kvm-s63702 kernel: [494931.278884] 0000000000000000 ffff88079c4ff9d8 ffffffff817aed00 0000000000000000 Jan 16 05:42:31 kvm-s63702 kernel: [494931.278887] ffff8806938bebf0 ffff88079c4ffa78 ffffffff817a9c3f 0000000000000000 Jan 16 05:42:31 kvm-s63702 kernel: [494931.278889] 0000000000000000 0000000000000000 0000000000000000 ffff88079c4ffac8 Jan 16 05:42:31 kvm-s63702 kernel: [494931.278891] Call Trace: Jan 16 05:42:31 kvm-s63702 kernel: [494931.278932] [] dump_stack+0x45/0x57 Jan 16 05:42:31 kvm-s63702 kernel: [494931.278936] [] dump_header+0x7f/0x1f1 Jan 16 05:42:31 kvm-s63702 kernel: [494931.278957] [] oom_kill_process+0x22b/0x390 Jan 16 05:42:31 kvm-s63702 kernel: [494931.278977] [] out_of_memory+0x4d2/0x520 Jan 16 05:42:31 kvm-s63702 kernel: [494931.278982] [] __alloc_pages_nodemask+0x940/0xa60 Jan 16 05:42:31 kvm-s63702 kernel: [494931.278993] [] alloc_pages_vma+0x97/0x150 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279002] [] handle_mm_fault+0xd13/0x1100 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279022] [] __do_page_fault+0x1c4/0x5b0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279037] [] ? __enqueue_entity+0x78/0x80 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279063] [] ? do_futex+0x10a/0x5d0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279069] [] ? pick_next_task_fair+0x61b/0x880 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279082] [] ? __switch_to+0x15f/0x570 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279089] [] ? sched_clock_cpu+0x98/0xc0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279094] [] trace_do_page_fault+0x45/0x100 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279098] [] do_async_page_fault+0x1e/0xd0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279109] [] async_page_fault+0x28/0x30 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279113] Mem-Info: Jan 16 05:42:31 kvm-s63702 kernel: [494931.279117] Node 0 DMA per-cpu: Jan 16 05:42:31 kvm-s63702 kernel: [494931.279121] CPU 0: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279123] CPU 1: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279126] CPU 2: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279127] CPU 3: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279128] CPU 4: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279129] CPU 5: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279130] CPU 6: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279131] CPU 7: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279132] CPU 8: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279133] CPU 9: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279134] CPU 10: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279135] CPU 11: hi: 0, btch: 1 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279136] Node 0 DMA32 per-cpu: Jan 16 05:42:31 kvm-s63702 kernel: [494931.279137] CPU 0: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279141] CPU 1: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279142] CPU 2: hi: 186, btch: 31 usd: 1 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279143] CPU 3: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279144] CPU 4: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279145] CPU 5: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279146] CPU 6: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279147] CPU 7: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279148] CPU 8: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279149] CPU 9: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279150] CPU 10: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279151] CPU 11: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279152] Node 0 Normal per-cpu: Jan 16 05:42:31 kvm-s63702 kernel: [494931.279153] CPU 0: hi: 186, btch: 31 usd: 27 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279154] CPU 1: hi: 186, btch: 31 usd: 29 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279155] CPU 2: hi: 186, btch: 31 usd: 12 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279156] CPU 3: hi: 186, btch: 31 usd: 24 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279157] CPU 4: hi: 186, btch: 31 usd: 23 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279158] CPU 5: hi: 186, btch: 31 usd: 12 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279159] CPU 6: hi: 186, btch: 31 usd: 45 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279160] CPU 7: hi: 186, btch: 31 usd: 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279161] CPU 8: hi: 186, btch: 31 usd: 11 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279162] CPU 9: hi: 186, btch: 31 usd: 18 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279163] CPU 10: hi: 186, btch: 31 usd: 6 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279164] CPU 11: hi: 186, btch: 31 usd: 14 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279167] active_anon:7596021 inactive_anon:73 isolated_anon:0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279167] active_file:166 inactive_file:78 isolated_file:78 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279167] unevictable:0 dirty:38 writeback:0 unstable:0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279167] free:48060 slab_reclaimable:8034 slab_unreclaimable:9256 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279167] mapped:0 shmem:94 pagetables:18565 bounce:0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279167] free_cma:0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279170] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 16 05:42:31 kvm-s63702 kernel: [494931.279178] lowmem_reserve[]: 0 2978 30143 30143 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279180] Node 0 DMA32 free:115188kB min:6668kB low:8332kB high:10000kB active_anon:2908524kB inactive_anon:32kB active_file:116kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:0kB dirty:116kB writeback:0kB mapped:0kB shmem:32kB slab_reclaimable:2680kB slab_unreclaimable:4156kB kernel_stack:2208kB pagetables:6760kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2828 all_unreclaimable? yes Jan 16 05:42:31 kvm-s63702 kernel: [494931.279184] lowmem_reserve[]: 0 0 27165 27165 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279188] Node 0 Normal free:61160kB min:60876kB low:76092kB high:91312kB active_anon:27475560kB inactive_anon:260kB active_file:548kB inactive_file:668kB unevictable:0kB isolated(anon):0kB isolated(file):312kB present:28311552kB managed:27817664kB mlocked:0kB dirty:36kB writeback:0kB mapped:0kB shmem:344kB slab_reclaimable:29456kB slab_unreclaimable:32852kB kernel_stack:21712kB pagetables:67500kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4816 all_unreclaimable? no Jan 16 05:42:31 kvm-s63702 kernel: [494931.279206] lowmem_reserve[]: 0 0 0 0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279208] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 16 05:42:31 kvm-s63702 kernel: [494931.279221] Node 0 DMA32: 187*4kB (UEM) 126*8kB (UEM) 164*16kB (UEM) 130*32kB (UEM) 74*64kB (UEM) 35*128kB (UEM) 16*256kB (UE) 11*512kB (UEM) 2*1024kB (UM) 4*2048kB (UEM) 19*4096kB (MR) = 115548kB Jan 16 05:42:31 kvm-s63702 kernel: [494931.279229] Node 0 Normal: 436*4kB (UEM) 297*8kB (UEM) 404*16kB (UEM) 342*32kB (UEM) 269*64kB (UEM) 95*128kB (UEM) 21*256kB (EM) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60376kB Jan 16 05:42:31 kvm-s63702 kernel: [494931.279238] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 16 05:42:31 kvm-s63702 kernel: [494931.279239] 587 total pagecache pages Jan 16 05:42:31 kvm-s63702 kernel: [494931.279240] 0 pages in swap cache Jan 16 05:42:31 kvm-s63702 kernel: [494931.279241] Swap cache stats: add 0, delete 0, find 0/0 Jan 16 05:42:31 kvm-s63702 kernel: [494931.279242] Free swap = 0kB Jan 16 05:42:31 kvm-s63702 kernel: [494931.279243] Total swap = 0kB Jan 16 05:42:31 kvm-s63702 kernel: [494931.279244] 7864218 pages RAM Jan 16 05:42:31 kvm-s63702 kernel: [494931.279245] 0 pages HighMem/MovableOnly Jan 16 05:42:31 kvm-s63702 kernel: [494931.279245] 143432 pages reserved Jan 16 05:42:31 kvm-s63702 kernel: [494931.279246] 0 pages cma reserved Jan 16 05:42:31 kvm-s63702 kernel: [494931.279247] 0 pages hwpoisoned Jan 16 05:42:31 kvm-s63702 kernel: [494931.279248] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 16 05:42:31 kvm-s63702 kernel: [494931.279255] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 16 05:42:31 kvm-s63702 kernel: [494931.279258] [ 381] 0 381 12939 220 28 0 -1000 systemd-udevd Jan 16 05:42:31 kvm-s63702 kernel: [494931.279260] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 16 05:42:31 kvm-s63702 kernel: [494931.279262] [ 724] 0 724 2559 597 8 0 0 dhclient Jan 16 05:42:31 kvm-s63702 kernel: [494931.279263] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 16 05:42:31 kvm-s63702 kernel: [494931.279265] [ 932] 101 932 65020 200 29 0 0 rsyslogd Jan 16 05:42:31 kvm-s63702 kernel: [494931.279267] [ 961] 102 961 9809 103 23 0 0 dbus-daemon Jan 16 05:42:31 kvm-s63702 kernel: [494931.279269] [ 964] 0 964 3956 44 12 0 0 getty Jan 16 05:42:31 kvm-s63702 kernel: [494931.279270] [ 967] 0 967 3956 43 13 0 0 getty Jan 16 05:42:31 kvm-s63702 kernel: [494931.279272] [ 972] 0 972 3956 43 13 0 0 getty Jan 16 05:42:31 kvm-s63702 kernel: [494931.279273] [ 973] 0 973 3956 42 13 0 0 getty Jan 16 05:42:31 kvm-s63702 kernel: [494931.279275] [ 975] 0 975 3956 42 13 0 0 getty Jan 16 05:42:31 kvm-s63702 kernel: [494931.279277] [ 995] 0 995 15343 174 33 0 -1000 sshd Jan 16 05:42:31 kvm-s63702 kernel: [494931.279278] [ 998] 0 998 1093 38 8 0 0 acpid Jan 16 05:42:31 kvm-s63702 kernel: [494931.279280] [ 1016] 0 1016 5915 68 18 0 0 cron Jan 16 05:42:31 kvm-s63702 kernel: [494931.279282] [ 1017] 0 1017 4786 41 13 0 0 atd Jan 16 05:42:31 kvm-s63702 kernel: [494931.279283] [ 1046] 0 1046 10864 97 27 0 0 systemd-logind Jan 16 05:42:31 kvm-s63702 kernel: [494931.279285] [ 1052] 0 1052 4799 77 14 0 0 irqbalance Jan 16 05:42:31 kvm-s63702 kernel: [494931.279286] [ 1520] 0 1520 3956 43 13 0 0 getty Jan 16 05:42:31 kvm-s63702 kernel: [494931.279289] [ 5353] 1000 5353 1875 43 8 0 0 epmd Jan 16 05:42:31 kvm-s63702 kernel: [494931.279291] [ 5396] 1000 5396 351373 7524 86 0 0 beam.smp Jan 16 05:42:31 kvm-s63702 kernel: [494931.279292] [ 5432] 1000 5432 2884 1503 10 0 0 gosecrets Jan 16 05:42:31 kvm-s63702 kernel: [494931.279294] [ 5436] 1000 5436 617342 161543 637 0 0 beam.smp Jan 16 05:42:31 kvm-s63702 kernel: [494931.279296] [ 5474] 1000 5474 1113 31 7 0 0 sh Jan 16 05:42:31 kvm-s63702 kernel: [494931.279297] [ 5476] 1000 5476 1084 29 7 0 0 memsup Jan 16 05:42:31 kvm-s63702 kernel: [494931.279299] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 16 05:42:31 kvm-s63702 kernel: [494931.279300] [ 5479] 1000 5479 1867 23 9 0 0 inet_gethost Jan 16 05:42:31 kvm-s63702 kernel: [494931.279314] [ 5480] 1000 5480 1867 33 9 0 0 inet_gethost Jan 16 05:42:31 kvm-s63702 kernel: [494931.279316] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 16 05:42:31 kvm-s63702 kernel: [494931.279318] [ 5571] 1000 5571 428648 5384 112 0 0 memcached Jan 16 05:42:31 kvm-s63702 kernel: [494931.279319] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 16 05:42:31 kvm-s63702 kernel: [494931.279321] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 16 05:42:31 kvm-s63702 kernel: [494931.279323] [ 5786] 1000 5786 276493 16485 136 0 0 beam.smp Jan 16 05:42:31 kvm-s63702 kernel: [494931.279324] [ 5854] 1000 5854 1113 22 5 0 0 sh Jan 16 05:42:31 kvm-s63702 kernel: [494931.279326] [ 5856] 1000 5856 1084 29 7 0 0 memsup Jan 16 05:42:31 kvm-s63702 kernel: [494931.279327] [ 5857] 1000 5857 1084 21 8 0 0 cpu_sup Jan 16 05:42:31 kvm-s63702 kernel: [494931.279329] [ 5863] 1000 5863 1331 110 8 0 0 godu Jan 16 05:42:31 kvm-s63702 kernel: [494931.279331] [ 5867] 1000 5867 1113 31 7 0 0 sh Jan 16 05:42:31 kvm-s63702 kernel: [494931.279332] [ 5868] 1000 5868 1331 109 8 0 0 godu Jan 16 05:42:31 kvm-s63702 kernel: [494931.279334] [ 5878] 1000 5878 2156 59 9 0 0 sigar_port Jan 16 05:42:31 kvm-s63702 kernel: [494931.279335] [ 5879] 1000 5879 1462 163 8 0 0 goport Jan 16 05:42:31 kvm-s63702 kernel: [494931.279336] [ 5884] 1000 5884 72275 2044 31 0 0 goxdcr Jan 16 05:42:31 kvm-s63702 kernel: [494931.279338] [ 5893] 1000 5893 233207 6558 76 0 0 beam.smp Jan 16 05:42:31 kvm-s63702 kernel: [494931.279339] [ 5925] 1000 5925 1462 151 8 0 0 goport Jan 16 05:42:31 kvm-s63702 kernel: [494931.279341] [ 5930] 1000 5930 13419752 7355153 16978 0 0 indexer Jan 16 05:42:31 kvm-s63702 kernel: [494931.279343] [ 7058] 1000 7058 1867 33 9 0 0 inet_gethost Jan 16 05:42:31 kvm-s63702 kernel: [494931.279345] Out of memory: Kill process 11743 (indexer) score 954 or sacrifice child Jan 16 05:42:31 kvm-s63702 kernel: [494931.279456] Killed process 5930 (indexer) total-vm:53679008kB, anon-rss:29420876kB, file-rss:0kB Jan 16 08:36:25 kvm-s63702 kernel: [505365.458322] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458329] indexer cpuset=/ mems_allowed=0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458345] CPU: 4 PID: 12861 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 16 08:36:25 kvm-s63702 kernel: [505365.458346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458348] 0000000000000000 ffff88064a7f39d8 ffffffff817aed00 0000000000000000 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458350] ffff880663e8d850 ffff88064a7f3a78 ffffffff817a9c3f 0000000000000000 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458352] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458354] Call Trace: Jan 16 08:36:25 kvm-s63702 kernel: [505365.458395] [] dump_stack+0x45/0x57 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458398] [] dump_header+0x7f/0x1f1 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458416] [] oom_kill_process+0x22b/0x390 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458419] [] out_of_memory+0x4d2/0x520 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458423] [] __alloc_pages_nodemask+0x940/0xa60 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458433] [] alloc_pages_vma+0x97/0x150 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458439] [] handle_mm_fault+0xd13/0x1100 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458451] [] ? x2apic_send_IPI_mask+0x13/0x20 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458459] [] ? try_to_wake_up+0x1f1/0x340 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458466] [] __do_page_fault+0x1c4/0x5b0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458472] [] ? __enqueue_entity+0x78/0x80 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458475] [] ? set_next_entity+0x9d/0xb0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458477] [] ? pick_next_task_fair+0x61b/0x880 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458488] [] ? __switch_to+0x15f/0x570 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458490] [] ? sched_clock_cpu+0x98/0xc0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458492] [] trace_do_page_fault+0x45/0x100 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458496] [] do_async_page_fault+0x1e/0xd0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458506] [] async_page_fault+0x28/0x30 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458508] Mem-Info: Jan 16 08:36:25 kvm-s63702 kernel: [505365.458511] Node 0 DMA per-cpu: Jan 16 08:36:25 kvm-s63702 kernel: [505365.458515] CPU 0: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458516] CPU 1: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458517] CPU 2: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458518] CPU 3: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458539] CPU 4: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458540] CPU 5: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458541] CPU 6: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458542] CPU 7: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458543] CPU 8: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458544] CPU 9: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458545] CPU 10: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458546] CPU 11: hi: 0, btch: 1 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458546] Node 0 DMA32 per-cpu: Jan 16 08:36:25 kvm-s63702 kernel: [505365.458548] CPU 0: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458549] CPU 1: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458550] CPU 2: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458551] CPU 3: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458551] CPU 4: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458552] CPU 5: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458553] CPU 6: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458554] CPU 7: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458555] CPU 8: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458556] CPU 9: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458557] CPU 10: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458558] CPU 11: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458558] Node 0 Normal per-cpu: Jan 16 08:36:25 kvm-s63702 kernel: [505365.458559] CPU 0: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458560] CPU 1: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458561] CPU 2: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458562] CPU 3: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458563] CPU 4: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458564] CPU 5: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458565] CPU 6: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458566] CPU 7: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458566] CPU 8: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458567] CPU 9: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458568] CPU 10: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458569] CPU 11: hi: 186, btch: 31 usd: 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458572] active_anon:7595859 inactive_anon:73 isolated_anon:0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458572] active_file:225 inactive_file:0 isolated_file:0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458572] unevictable:0 dirty:88 writeback:0 unstable:0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458572] free:47921 slab_reclaimable:7293 slab_unreclaimable:9519 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458572] mapped:119 shmem:94 pagetables:19292 bounce:0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458572] free_cma:0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458575] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 16 08:36:25 kvm-s63702 kernel: [505365.458582] lowmem_reserve[]: 0 2978 30143 30143 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458584] Node 0 DMA32 free:115264kB min:6668kB low:8332kB high:10000kB active_anon:2908008kB inactive_anon:32kB active_file:128kB inactive_file:44kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:0kB dirty:4kB writeback:0kB mapped:0kB shmem:32kB slab_reclaimable:2176kB slab_unreclaimable:3580kB kernel_stack:2336kB pagetables:7892kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1120 all_unreclaimable? yes Jan 16 08:36:25 kvm-s63702 kernel: [505365.458588] lowmem_reserve[]: 0 0 27165 27165 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458590] Node 0 Normal free:60528kB min:60876kB low:76092kB high:91312kB active_anon:27475428kB inactive_anon:260kB active_file:772kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:0kB dirty:348kB writeback:0kB mapped:528kB shmem:344kB slab_reclaimable:26996kB slab_unreclaimable:34480kB kernel_stack:21664kB pagetables:69276kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:106956 all_unreclaimable? yes Jan 16 08:36:25 kvm-s63702 kernel: [505365.458593] lowmem_reserve[]: 0 0 0 0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458595] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 16 08:36:25 kvm-s63702 kernel: [505365.458605] Node 0 DMA32: 239*4kB (UEM) 120*8kB (UEM) 127*16kB (UEM) 102*32kB (UEM) 68*64kB (UEM) 29*128kB (UEM) 18*256kB (UEM) 12*512kB (UE) 2*1024kB (UE) 3*2048kB (UE) 20*4096kB (MR) = 116140kB Jan 16 08:36:25 kvm-s63702 kernel: [505365.458613] Node 0 Normal: 486*4kB (UEM) 339*8kB (UEM) 418*16kB (UEM) 329*32kB (UEM) 203*64kB (UEM) 112*128kB (UEM) 32*256kB (UE) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61488kB Jan 16 08:36:25 kvm-s63702 kernel: [505365.458631] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 16 08:36:25 kvm-s63702 kernel: [505365.458632] 785 total pagecache pages Jan 16 08:36:25 kvm-s63702 kernel: [505365.458633] 0 pages in swap cache Jan 16 08:36:25 kvm-s63702 kernel: [505365.458634] Swap cache stats: add 0, delete 0, find 0/0 Jan 16 08:36:25 kvm-s63702 kernel: [505365.458635] Free swap = 0kB Jan 16 08:36:25 kvm-s63702 kernel: [505365.458636] Total swap = 0kB Jan 16 08:36:25 kvm-s63702 kernel: [505365.458637] 7864218 pages RAM Jan 16 08:36:25 kvm-s63702 kernel: [505365.458638] 0 pages HighMem/MovableOnly Jan 16 08:36:25 kvm-s63702 kernel: [505365.458638] 143432 pages reserved Jan 16 08:36:25 kvm-s63702 kernel: [505365.458639] 0 pages cma reserved Jan 16 08:36:25 kvm-s63702 kernel: [505365.458640] 0 pages hwpoisoned Jan 16 08:36:25 kvm-s63702 kernel: [505365.458641] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 16 08:36:25 kvm-s63702 kernel: [505365.458647] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 16 08:36:25 kvm-s63702 kernel: [505365.458649] [ 381] 0 381 12939 220 28 0 -1000 systemd-udevd Jan 16 08:36:25 kvm-s63702 kernel: [505365.458651] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 16 08:36:25 kvm-s63702 kernel: [505365.458653] [ 724] 0 724 2559 597 8 0 0 dhclient Jan 16 08:36:25 kvm-s63702 kernel: [505365.458655] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 16 08:36:25 kvm-s63702 kernel: [505365.458657] [ 932] 101 932 65020 242 29 0 0 rsyslogd Jan 16 08:36:25 kvm-s63702 kernel: [505365.458658] [ 961] 102 961 9809 106 23 0 0 dbus-daemon Jan 16 08:36:25 kvm-s63702 kernel: [505365.458660] [ 964] 0 964 3956 44 12 0 0 getty Jan 16 08:36:25 kvm-s63702 kernel: [505365.458661] [ 967] 0 967 3956 43 13 0 0 getty Jan 16 08:36:25 kvm-s63702 kernel: [505365.458663] [ 972] 0 972 3956 43 13 0 0 getty Jan 16 08:36:25 kvm-s63702 kernel: [505365.458664] [ 973] 0 973 3956 42 13 0 0 getty Jan 16 08:36:25 kvm-s63702 kernel: [505365.458665] [ 975] 0 975 3956 42 13 0 0 getty Jan 16 08:36:25 kvm-s63702 kernel: [505365.458667] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 16 08:36:25 kvm-s63702 kernel: [505365.458669] [ 998] 0 998 1093 38 8 0 0 acpid Jan 16 08:36:25 kvm-s63702 kernel: [505365.458670] [ 1016] 0 1016 5915 71 18 0 0 cron Jan 16 08:36:25 kvm-s63702 kernel: [505365.458672] [ 1017] 0 1017 4786 41 13 0 0 atd Jan 16 08:36:25 kvm-s63702 kernel: [505365.458673] [ 1046] 0 1046 10864 101 27 0 0 systemd-logind Jan 16 08:36:25 kvm-s63702 kernel: [505365.458675] [ 1052] 0 1052 4799 79 14 0 0 irqbalance Jan 16 08:36:25 kvm-s63702 kernel: [505365.458676] [ 1520] 0 1520 3956 43 13 0 0 getty Jan 16 08:36:25 kvm-s63702 kernel: [505365.458679] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 16 08:36:25 kvm-s63702 kernel: [505365.458680] [ 5396] 1000 5396 352269 7683 87 0 0 beam.smp Jan 16 08:36:25 kvm-s63702 kernel: [505365.458682] [ 5432] 1000 5432 2884 1229 10 0 0 gosecrets Jan 16 08:36:25 kvm-s63702 kernel: [505365.458683] [ 5436] 1000 5436 620343 146241 590 0 0 beam.smp Jan 16 08:36:25 kvm-s63702 kernel: [505365.458685] [ 5474] 1000 5474 1113 31 7 0 0 sh Jan 16 08:36:25 kvm-s63702 kernel: [505365.458686] [ 5476] 1000 5476 1084 33 7 0 0 memsup Jan 16 08:36:25 kvm-s63702 kernel: [505365.458688] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 16 08:36:25 kvm-s63702 kernel: [505365.458689] [ 5479] 1000 5479 1867 23 9 0 0 inet_gethost Jan 16 08:36:25 kvm-s63702 kernel: [505365.458691] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 16 08:36:25 kvm-s63702 kernel: [505365.458692] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 16 08:36:25 kvm-s63702 kernel: [505365.458694] [ 5571] 1000 5571 428648 5384 112 0 0 memcached Jan 16 08:36:25 kvm-s63702 kernel: [505365.458695] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 16 08:36:25 kvm-s63702 kernel: [505365.458697] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 16 08:36:25 kvm-s63702 kernel: [505365.458698] [ 5786] 1000 5786 277642 16185 139 0 0 beam.smp Jan 16 08:36:25 kvm-s63702 kernel: [505365.458700] [ 5854] 1000 5854 1113 27 5 0 0 sh Jan 16 08:36:25 kvm-s63702 kernel: [505365.458701] [ 5856] 1000 5856 1084 33 7 0 0 memsup Jan 16 08:36:25 kvm-s63702 kernel: [505365.458712] [ 5857] 1000 5857 1084 21 8 0 0 cpu_sup Jan 16 08:36:25 kvm-s63702 kernel: [505365.458714] [ 5863] 1000 5863 1331 110 8 0 0 godu Jan 16 08:36:25 kvm-s63702 kernel: [505365.458716] [ 5867] 1000 5867 1113 34 7 0 0 sh Jan 16 08:36:25 kvm-s63702 kernel: [505365.458717] [ 5868] 1000 5868 1331 109 8 0 0 godu Jan 16 08:36:25 kvm-s63702 kernel: [505365.458719] [ 5878] 1000 5878 2156 64 9 0 0 sigar_port Jan 16 08:36:25 kvm-s63702 kernel: [505365.458720] [ 5879] 1000 5879 1462 163 8 0 0 goport Jan 16 08:36:25 kvm-s63702 kernel: [505365.458722] [ 5884] 1000 5884 74324 2133 32 0 0 goxdcr Jan 16 08:36:25 kvm-s63702 kernel: [505365.458723] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 16 08:36:25 kvm-s63702 kernel: [505365.458725] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 16 08:36:25 kvm-s63702 kernel: [505365.458726] [12029] 1000 12029 1462 151 8 0 0 goport Jan 16 08:36:25 kvm-s63702 kernel: [505365.458728] [12033] 1000 12033 13586668 7368762 17759 0 0 indexer Jan 16 08:36:25 kvm-s63702 kernel: [505365.458730] Out of memory: Kill process 12033 (indexer) score 956 or sacrifice child Jan 16 08:36:25 kvm-s63702 kernel: [505365.458831] Killed process 12033 (indexer) total-vm:54346672kB, anon-rss:29474884kB, file-rss:164kB Jan 16 08:44:57 kvm-s63702 kernel: [505877.079272] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079277] indexer cpuset=/ mems_allowed=0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079293] CPU: 8 PID: 19618 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 16 08:44:57 kvm-s63702 kernel: [505877.079294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079296] 0000000000000000 ffff8804c8fb79d8 ffffffff817aed00 0000000000000000 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079298] ffff88054e826bf0 ffff8804c8fb7a78 ffffffff817a9c3f 0000000000000000 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079300] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079302] Call Trace: Jan 16 08:44:57 kvm-s63702 kernel: [505877.079342] [] dump_stack+0x45/0x57 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079345] [] dump_header+0x7f/0x1f1 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079361] [] oom_kill_process+0x22b/0x390 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079364] [] out_of_memory+0x4d2/0x520 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079368] [] __alloc_pages_nodemask+0x940/0xa60 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079377] [] alloc_pages_vma+0x97/0x150 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079383] [] handle_mm_fault+0xd13/0x1100 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079394] [] ? x2apic_send_IPI_mask+0x13/0x20 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079402] [] ? try_to_wake_up+0x1f1/0x340 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079408] [] __do_page_fault+0x1c4/0x5b0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079414] [] ? __enqueue_entity+0x78/0x80 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079416] [] ? set_next_entity+0x9d/0xb0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079419] [] ? pick_next_task_fair+0x61b/0x880 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079428] [] ? __switch_to+0x15f/0x570 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079430] [] ? sched_clock_cpu+0x98/0xc0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079432] [] trace_do_page_fault+0x45/0x100 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079435] [] do_async_page_fault+0x1e/0xd0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079444] [] async_page_fault+0x28/0x30 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079446] Mem-Info: Jan 16 08:44:57 kvm-s63702 kernel: [505877.079449] Node 0 DMA per-cpu: Jan 16 08:44:57 kvm-s63702 kernel: [505877.079453] CPU 0: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079454] CPU 1: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079455] CPU 2: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079456] CPU 3: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079457] CPU 4: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079458] CPU 5: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079459] CPU 6: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079460] CPU 7: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079461] CPU 8: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079461] CPU 9: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079462] CPU 10: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079463] CPU 11: hi: 0, btch: 1 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079464] Node 0 DMA32 per-cpu: Jan 16 08:44:57 kvm-s63702 kernel: [505877.079465] CPU 0: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079466] CPU 1: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079467] CPU 2: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079468] CPU 3: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079469] CPU 4: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079485] CPU 5: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079487] CPU 6: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079488] CPU 7: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079488] CPU 8: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079489] CPU 9: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079490] CPU 10: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079491] CPU 11: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079492] Node 0 Normal per-cpu: Jan 16 08:44:57 kvm-s63702 kernel: [505877.079493] CPU 0: hi: 186, btch: 31 usd: 25 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079494] CPU 1: hi: 186, btch: 31 usd: 30 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079495] CPU 2: hi: 186, btch: 31 usd: 24 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079496] CPU 3: hi: 186, btch: 31 usd: 14 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079496] CPU 4: hi: 186, btch: 31 usd: 13 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079497] CPU 5: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079498] CPU 6: hi: 186, btch: 31 usd: 42 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079499] CPU 7: hi: 186, btch: 31 usd: 25 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079500] CPU 8: hi: 186, btch: 31 usd: 15 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079501] CPU 9: hi: 186, btch: 31 usd: 26 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079502] CPU 10: hi: 186, btch: 31 usd: 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079503] CPU 11: hi: 186, btch: 31 usd: 20 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079506] active_anon:7600219 inactive_anon:73 isolated_anon:0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079506] active_file:0 inactive_file:3 isolated_file:0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079506] unevictable:0 dirty:0 writeback:7 unstable:0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079506] free:48518 slab_reclaimable:7219 slab_unreclaimable:9790 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079506] mapped:163 shmem:94 pagetables:18526 bounce:0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079506] free_cma:0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079508] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 16 08:44:57 kvm-s63702 kernel: [505877.079520] lowmem_reserve[]: 0 2978 30143 30143 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079523] Node 0 DMA32 free:116856kB min:6668kB low:8332kB high:10000kB active_anon:2914140kB inactive_anon:32kB active_file:112kB inactive_file:196kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:0kB dirty:0kB writeback:0kB mapped:336kB shmem:32kB slab_reclaimable:2076kB slab_unreclaimable:3820kB kernel_stack:3280kB pagetables:5140kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:748 all_unreclaimable? no Jan 16 08:44:57 kvm-s63702 kernel: [505877.079526] lowmem_reserve[]: 0 0 27165 27165 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079528] Node 0 Normal free:61324kB min:60876kB low:76092kB high:91312kB active_anon:27486736kB inactive_anon:260kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:0kB dirty:0kB writeback:28kB mapped:316kB shmem:344kB slab_reclaimable:26800kB slab_unreclaimable:35324kB kernel_stack:35680kB pagetables:68964kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7428 all_unreclaimable? yes Jan 16 08:44:57 kvm-s63702 kernel: [505877.079531] lowmem_reserve[]: 0 0 0 0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079533] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 16 08:44:57 kvm-s63702 kernel: [505877.079543] Node 0 DMA32: 253*4kB (UEM) 113*8kB (UEM) 124*16kB (UEM) 121*32kB (UEM) 105*64kB (UEM) 54*128kB (UE) 24*256kB (UEM) 13*512kB (UEM) 3*1024kB (UEM) 3*2048kB (UE) 18*4096kB (MR) = 117148kB Jan 16 08:44:57 kvm-s63702 kernel: [505877.079552] Node 0 Normal: 333*4kB (UEM) 237*8kB (UE) 281*16kB (UEM) 268*32kB (UEM) 159*64kB (UEM) 80*128kB (UEM) 80*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61292kB Jan 16 08:44:57 kvm-s63702 kernel: [505877.079560] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 16 08:44:57 kvm-s63702 kernel: [505877.079561] 386 total pagecache pages Jan 16 08:44:57 kvm-s63702 kernel: [505877.079562] 0 pages in swap cache Jan 16 08:44:57 kvm-s63702 kernel: [505877.079563] Swap cache stats: add 0, delete 0, find 0/0 Jan 16 08:44:57 kvm-s63702 kernel: [505877.079564] Free swap = 0kB Jan 16 08:44:57 kvm-s63702 kernel: [505877.079565] Total swap = 0kB Jan 16 08:44:57 kvm-s63702 kernel: [505877.079566] 7864218 pages RAM Jan 16 08:44:57 kvm-s63702 kernel: [505877.079567] 0 pages HighMem/MovableOnly Jan 16 08:44:57 kvm-s63702 kernel: [505877.079567] 143432 pages reserved Jan 16 08:44:57 kvm-s63702 kernel: [505877.079568] 0 pages cma reserved Jan 16 08:44:57 kvm-s63702 kernel: [505877.079569] 0 pages hwpoisoned Jan 16 08:44:57 kvm-s63702 kernel: [505877.079569] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 16 08:44:57 kvm-s63702 kernel: [505877.079586] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 16 08:44:57 kvm-s63702 kernel: [505877.079589] [ 381] 0 381 12939 220 28 0 -1000 systemd-udevd Jan 16 08:44:57 kvm-s63702 kernel: [505877.079591] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 16 08:44:57 kvm-s63702 kernel: [505877.079593] [ 724] 0 724 2559 593 8 0 0 dhclient Jan 16 08:44:57 kvm-s63702 kernel: [505877.079594] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 16 08:44:57 kvm-s63702 kernel: [505877.079596] [ 932] 101 932 65020 244 29 0 0 rsyslogd Jan 16 08:44:57 kvm-s63702 kernel: [505877.079598] [ 961] 102 961 9809 100 23 0 0 dbus-daemon Jan 16 08:44:57 kvm-s63702 kernel: [505877.079599] [ 964] 0 964 3956 44 12 0 0 getty Jan 16 08:44:57 kvm-s63702 kernel: [505877.079601] [ 967] 0 967 3956 43 13 0 0 getty Jan 16 08:44:57 kvm-s63702 kernel: [505877.079602] [ 972] 0 972 3956 43 13 0 0 getty Jan 16 08:44:57 kvm-s63702 kernel: [505877.079604] [ 973] 0 973 3956 42 13 0 0 getty Jan 16 08:44:57 kvm-s63702 kernel: [505877.079605] [ 975] 0 975 3956 42 13 0 0 getty Jan 16 08:44:57 kvm-s63702 kernel: [505877.079607] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 16 08:44:57 kvm-s63702 kernel: [505877.079608] [ 998] 0 998 1093 38 8 0 0 acpid Jan 16 08:44:57 kvm-s63702 kernel: [505877.079610] [ 1016] 0 1016 5915 64 18 0 0 cron Jan 16 08:44:57 kvm-s63702 kernel: [505877.079612] [ 1017] 0 1017 4786 41 13 0 0 atd Jan 16 08:44:57 kvm-s63702 kernel: [505877.079613] [ 1046] 0 1046 10864 95 27 0 0 systemd-logind Jan 16 08:44:57 kvm-s63702 kernel: [505877.079615] [ 1052] 0 1052 4799 74 14 0 0 irqbalance Jan 16 08:44:57 kvm-s63702 kernel: [505877.079616] [ 1520] 0 1520 3956 43 13 0 0 getty Jan 16 08:44:57 kvm-s63702 kernel: [505877.079619] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 16 08:44:57 kvm-s63702 kernel: [505877.079621] [ 5396] 1000 5396 353344 7697 88 0 0 beam.smp Jan 16 08:44:57 kvm-s63702 kernel: [505877.079622] [ 5432] 1000 5432 2884 1464 10 0 0 gosecrets Jan 16 08:44:57 kvm-s63702 kernel: [505877.079624] [ 5436] 1000 5436 629010 163208 612 0 0 beam.smp Jan 16 08:44:57 kvm-s63702 kernel: [505877.079625] [ 5474] 1000 5474 1113 31 7 0 0 sh Jan 16 08:44:57 kvm-s63702 kernel: [505877.079627] [ 5476] 1000 5476 1084 29 7 0 0 memsup Jan 16 08:44:57 kvm-s63702 kernel: [505877.079628] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 16 08:44:57 kvm-s63702 kernel: [505877.079630] [ 5479] 1000 5479 1867 23 9 0 0 inet_gethost Jan 16 08:44:57 kvm-s63702 kernel: [505877.079631] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 16 08:44:57 kvm-s63702 kernel: [505877.079633] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 16 08:44:57 kvm-s63702 kernel: [505877.079634] [ 5571] 1000 5571 428648 5404 112 0 0 memcached Jan 16 08:44:57 kvm-s63702 kernel: [505877.079636] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 16 08:44:57 kvm-s63702 kernel: [505877.079637] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 16 08:44:57 kvm-s63702 kernel: [505877.079639] [ 5786] 1000 5786 276336 15521 134 0 0 beam.smp Jan 16 08:44:57 kvm-s63702 kernel: [505877.079640] [ 5854] 1000 5854 1113 27 5 0 0 sh Jan 16 08:44:57 kvm-s63702 kernel: [505877.079642] [ 5856] 1000 5856 1084 29 7 0 0 memsup Jan 16 08:44:57 kvm-s63702 kernel: [505877.079643] [ 5857] 1000 5857 1084 21 8 0 0 cpu_sup Jan 16 08:44:57 kvm-s63702 kernel: [505877.079645] [ 5863] 1000 5863 1331 110 8 0 0 godu Jan 16 08:44:57 kvm-s63702 kernel: [505877.079646] [ 5867] 1000 5867 1113 31 7 0 0 sh Jan 16 08:44:57 kvm-s63702 kernel: [505877.079648] [ 5868] 1000 5868 1331 109 8 0 0 godu Jan 16 08:44:57 kvm-s63702 kernel: [505877.079649] [ 5878] 1000 5878 2156 59 9 0 0 sigar_port Jan 16 08:44:57 kvm-s63702 kernel: [505877.079650] [ 5879] 1000 5879 1462 163 8 0 0 goport Jan 16 08:44:57 kvm-s63702 kernel: [505877.079652] [ 5884] 1000 5884 74324 2167 32 0 0 goxdcr Jan 16 08:44:57 kvm-s63702 kernel: [505877.079654] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 16 08:44:57 kvm-s63702 kernel: [505877.079664] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 16 08:44:57 kvm-s63702 kernel: [505877.079666] [18261] 1000 18261 1462 159 8 0 0 goport Jan 16 08:44:57 kvm-s63702 kernel: [505877.079668] [18266] 1000 18266 15641423 7328008 16963 0 0 indexer Jan 16 08:44:57 kvm-s63702 kernel: [505877.079669] Out of memory: Kill process 18266 (indexer) score 951 or sacrifice child Jan 16 08:44:57 kvm-s63702 kernel: [505877.079794] Killed process 18266 (indexer) total-vm:62565692kB, anon-rss:29312032kB, file-rss:0kB Jan 16 11:38:59 kvm-s63702 kernel: [516319.633666] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633682] indexer cpuset=/ mems_allowed=0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633701] CPU: 5 PID: 21174 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 16 11:39:00 kvm-s63702 kernel: [516319.633703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633705] 0000000000000000 ffff88079c4539d8 ffffffff817aed00 0000000000000000 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633707] ffff88079b9989d0 ffff88079c453a78 ffffffff817a9c3f 0000000000000000 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633709] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633712] Call Trace: Jan 16 11:39:00 kvm-s63702 kernel: [516319.633775] [] dump_stack+0x45/0x57 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633779] [] dump_header+0x7f/0x1f1 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633798] [] oom_kill_process+0x22b/0x390 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633800] [] out_of_memory+0x4d2/0x520 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633809] [] __alloc_pages_nodemask+0x940/0xa60 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633828] [] alloc_pages_vma+0x97/0x150 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633837] [] handle_mm_fault+0xd13/0x1100 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633853] [] ? ttwu_do_activate.constprop.94+0x5d/0x70 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633856] [] ? try_to_wake_up+0x1f1/0x340 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633867] [] __do_page_fault+0x1c4/0x5b0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633873] [] ? __enqueue_entity+0x78/0x80 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633875] [] ? set_next_entity+0x9d/0xb0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633902] [] ? pick_next_task_fair+0x61b/0x880 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633925] [] ? __switch_to+0x15f/0x570 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633927] [] ? sched_clock_cpu+0x98/0xc0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633929] [] trace_do_page_fault+0x45/0x100 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633934] [] do_async_page_fault+0x1e/0xd0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633953] [] async_page_fault+0x28/0x30 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633955] Mem-Info: Jan 16 11:39:00 kvm-s63702 kernel: [516319.633960] Node 0 DMA per-cpu: Jan 16 11:39:00 kvm-s63702 kernel: [516319.633964] CPU 0: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633965] CPU 1: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633967] CPU 2: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633968] CPU 3: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633969] CPU 4: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633971] CPU 5: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633972] CPU 6: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633973] CPU 7: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633974] CPU 8: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633976] CPU 9: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633977] CPU 10: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633978] CPU 11: hi: 0, btch: 1 usd: 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633979] Node 0 DMA32 per-cpu: Jan 16 11:39:00 kvm-s63702 kernel: [516319.633981] CPU 0: hi: 186, btch: 31 usd: 31 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633982] CPU 1: hi: 186, btch: 31 usd: 46 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633983] CPU 2: hi: 186, btch: 31 usd: 2 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633984] CPU 3: hi: 186, btch: 31 usd: 35 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633994] CPU 4: hi: 186, btch: 31 usd: 31 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633995] CPU 5: hi: 186, btch: 31 usd: 30 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633996] CPU 6: hi: 186, btch: 31 usd: 36 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633998] CPU 7: hi: 186, btch: 31 usd: 42 Jan 16 11:39:00 kvm-s63702 kernel: [516319.633999] CPU 8: hi: 186, btch: 31 usd: 12 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634000] CPU 9: hi: 186, btch: 31 usd: 3 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634001] CPU 10: hi: 186, btch: 31 usd: 44 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634002] CPU 11: hi: 186, btch: 31 usd: 22 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634003] Node 0 Normal per-cpu: Jan 16 11:39:00 kvm-s63702 kernel: [516319.634004] CPU 0: hi: 186, btch: 31 usd: 3 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634005] CPU 1: hi: 186, btch: 31 usd: 19 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634006] CPU 2: hi: 186, btch: 31 usd: 22 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634008] CPU 3: hi: 186, btch: 31 usd: 27 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634009] CPU 4: hi: 186, btch: 31 usd: 10 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634010] CPU 5: hi: 186, btch: 31 usd: 58 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634064] CPU 6: hi: 186, btch: 31 usd: 39 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634065] CPU 7: hi: 186, btch: 31 usd: 8 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634066] CPU 8: hi: 186, btch: 31 usd: 11 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634067] CPU 9: hi: 186, btch: 31 usd: 51 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634069] CPU 10: hi: 186, btch: 31 usd: 21 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634070] CPU 11: hi: 186, btch: 31 usd: 135 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634073] active_anon:7596610 inactive_anon:73 isolated_anon:0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634073] active_file:258 inactive_file:118 isolated_file:0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634073] unevictable:0 dirty:194 writeback:0 unstable:0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634073] free:47960 slab_reclaimable:7383 slab_unreclaimable:8915 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634073] mapped:159 shmem:94 pagetables:18949 bounce:0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634073] free_cma:0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634077] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 16 11:39:00 kvm-s63702 kernel: [516319.634084] lowmem_reserve[]: 0 2978 30143 30143 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634086] Node 0 DMA32 free:115168kB min:6668kB low:8332kB high:10000kB active_anon:2911604kB inactive_anon:32kB active_file:64kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:0kB dirty:20kB writeback:0kB mapped:124kB shmem:32kB slab_reclaimable:2272kB slab_unreclaimable:2888kB kernel_stack:2480kB pagetables:6292kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:98872 all_unreclaimable? yes Jan 16 11:39:00 kvm-s63702 kernel: [516319.634090] lowmem_reserve[]: 0 0 27165 27165 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634117] Node 0 Normal free:60780kB min:60876kB low:76092kB high:91312kB active_anon:27474836kB inactive_anon:260kB active_file:968kB inactive_file:508kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:0kB dirty:756kB writeback:0kB mapped:512kB shmem:344kB slab_reclaimable:27260kB slab_unreclaimable:32756kB kernel_stack:20992kB pagetables:69504kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:169896 all_unreclaimable? yes Jan 16 11:39:00 kvm-s63702 kernel: [516319.634120] lowmem_reserve[]: 0 0 0 0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634122] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 16 11:39:00 kvm-s63702 kernel: [516319.634134] Node 0 DMA32: 78*4kB (UEM) 56*8kB (EM) 119*16kB (UEM) 102*32kB (UE) 58*64kB (UE) 35*128kB (UEM) 27*256kB (UEM) 14*512kB (UEM) 2*1024kB (UE) 5*2048kB (UEM) 18*4096kB (MR) = 114216kB Jan 16 11:39:00 kvm-s63702 kernel: [516319.634155] Node 0 Normal: 367*4kB (UEM) 288*8kB (UEM) 327*16kB (UEM) 285*32kB (UEM) 184*64kB (UEM) 88*128kB (UEM) 49*256kB (UEM) 3*512kB (E) 0*1024kB 0*2048kB 1*4096kB (R) = 59340kB Jan 16 11:39:00 kvm-s63702 kernel: [516319.634165] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 16 11:39:00 kvm-s63702 kernel: [516319.634166] 745 total pagecache pages Jan 16 11:39:00 kvm-s63702 kernel: [516319.634168] 0 pages in swap cache Jan 16 11:39:00 kvm-s63702 kernel: [516319.634170] Swap cache stats: add 0, delete 0, find 0/0 Jan 16 11:39:00 kvm-s63702 kernel: [516319.634170] Free swap = 0kB Jan 16 11:39:00 kvm-s63702 kernel: [516319.634171] Total swap = 0kB Jan 16 11:39:00 kvm-s63702 kernel: [516319.634172] 7864218 pages RAM Jan 16 11:39:00 kvm-s63702 kernel: [516319.634173] 0 pages HighMem/MovableOnly Jan 16 11:39:00 kvm-s63702 kernel: [516319.634174] 143432 pages reserved Jan 16 11:39:00 kvm-s63702 kernel: [516319.634174] 0 pages cma reserved Jan 16 11:39:00 kvm-s63702 kernel: [516319.634175] 0 pages hwpoisoned Jan 16 11:39:00 kvm-s63702 kernel: [516319.634176] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 16 11:39:00 kvm-s63702 kernel: [516319.634223] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 16 11:39:00 kvm-s63702 kernel: [516319.634227] [ 381] 0 381 12939 220 28 0 -1000 systemd-udevd Jan 16 11:39:00 kvm-s63702 kernel: [516319.634231] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 16 11:39:00 kvm-s63702 kernel: [516319.634235] [ 724] 0 724 2559 593 8 0 0 dhclient Jan 16 11:39:00 kvm-s63702 kernel: [516319.634239] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 16 11:39:00 kvm-s63702 kernel: [516319.634242] [ 932] 101 932 65020 246 29 0 0 rsyslogd Jan 16 11:39:00 kvm-s63702 kernel: [516319.634245] [ 961] 102 961 9809 115 23 0 0 dbus-daemon Jan 16 11:39:00 kvm-s63702 kernel: [516319.634249] [ 964] 0 964 3956 44 12 0 0 getty Jan 16 11:39:00 kvm-s63702 kernel: [516319.634283] [ 967] 0 967 3956 43 13 0 0 getty Jan 16 11:39:00 kvm-s63702 kernel: [516319.634286] [ 972] 0 972 3956 43 13 0 0 getty Jan 16 11:39:00 kvm-s63702 kernel: [516319.634289] [ 973] 0 973 3956 42 13 0 0 getty Jan 16 11:39:00 kvm-s63702 kernel: [516319.634291] [ 975] 0 975 3956 42 13 0 0 getty Jan 16 11:39:00 kvm-s63702 kernel: [516319.634294] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 16 11:39:00 kvm-s63702 kernel: [516319.634297] [ 998] 0 998 1093 38 8 0 0 acpid Jan 16 11:39:00 kvm-s63702 kernel: [516319.634315] [ 1016] 0 1016 5915 77 18 0 0 cron Jan 16 11:39:00 kvm-s63702 kernel: [516319.634318] [ 1017] 0 1017 4786 41 13 0 0 atd Jan 16 11:39:00 kvm-s63702 kernel: [516319.634321] [ 1046] 0 1046 10864 111 27 0 0 systemd-logind Jan 16 11:39:00 kvm-s63702 kernel: [516319.634324] [ 1052] 0 1052 4799 89 14 0 0 irqbalance Jan 16 11:39:00 kvm-s63702 kernel: [516319.634327] [ 1520] 0 1520 3956 43 13 0 0 getty Jan 16 11:39:00 kvm-s63702 kernel: [516319.634332] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 16 11:39:00 kvm-s63702 kernel: [516319.634335] [ 5396] 1000 5396 351866 8046 89 0 0 beam.smp Jan 16 11:39:00 kvm-s63702 kernel: [516319.634338] [ 5432] 1000 5432 2884 1440 10 0 0 gosecrets Jan 16 11:39:00 kvm-s63702 kernel: [516319.634340] [ 5436] 1000 5436 643408 162241 627 0 0 beam.smp Jan 16 11:39:00 kvm-s63702 kernel: [516319.634343] [ 5474] 1000 5474 1113 37 7 0 0 sh Jan 16 11:39:00 kvm-s63702 kernel: [516319.634345] [ 5476] 1000 5476 1084 41 7 0 0 memsup Jan 16 11:39:00 kvm-s63702 kernel: [516319.634347] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 16 11:39:00 kvm-s63702 kernel: [516319.634360] [ 5479] 1000 5479 1867 23 9 0 0 inet_gethost Jan 16 11:39:00 kvm-s63702 kernel: [516319.634362] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 16 11:39:00 kvm-s63702 kernel: [516319.634366] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 16 11:39:00 kvm-s63702 kernel: [516319.634368] [ 5571] 1000 5571 428648 5404 112 0 0 memcached Jan 16 11:39:00 kvm-s63702 kernel: [516319.634371] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 16 11:39:00 kvm-s63702 kernel: [516319.634373] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 16 11:39:00 kvm-s63702 kernel: [516319.634375] [ 5786] 1000 5786 278912 17106 144 0 0 beam.smp Jan 16 11:39:00 kvm-s63702 kernel: [516319.634378] [ 5854] 1000 5854 1113 40 5 0 0 sh Jan 16 11:39:00 kvm-s63702 kernel: [516319.634380] [ 5856] 1000 5856 1084 41 7 0 0 memsup Jan 16 11:39:00 kvm-s63702 kernel: [516319.634382] [ 5857] 1000 5857 1084 21 8 0 0 cpu_sup Jan 16 11:39:00 kvm-s63702 kernel: [516319.634385] [ 5863] 1000 5863 1331 110 8 0 0 godu Jan 16 11:39:00 kvm-s63702 kernel: [516319.634402] [ 5867] 1000 5867 1113 40 7 0 0 sh Jan 16 11:39:00 kvm-s63702 kernel: [516319.634406] [ 5868] 1000 5868 1331 109 8 0 0 godu Jan 16 11:39:00 kvm-s63702 kernel: [516319.634408] [ 5878] 1000 5878 2156 74 9 0 0 sigar_port Jan 16 11:39:00 kvm-s63702 kernel: [516319.634410] [ 5879] 1000 5879 1462 163 8 0 0 goport Jan 16 11:39:00 kvm-s63702 kernel: [516319.634412] [ 5884] 1000 5884 74324 2196 32 0 0 goxdcr Jan 16 11:39:00 kvm-s63702 kernel: [516319.634415] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 16 11:39:00 kvm-s63702 kernel: [516319.634418] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 16 11:39:00 kvm-s63702 kernel: [516319.634422] [20334] 1000 20334 1526 154 8 0 0 goport Jan 16 11:39:00 kvm-s63702 kernel: [516319.634424] [20339] 1000 20339 13786398 7354818 17369 0 0 indexer Jan 16 11:39:00 kvm-s63702 kernel: [516319.634426] Out of memory: Kill process 26116 (indexer) score 954 or sacrifice child Jan 16 11:39:00 kvm-s63702 kernel: [516319.634754] Killed process 20339 (indexer) total-vm:55145592kB, anon-rss:29419272kB, file-rss:0kB Jan 16 12:03:29 kvm-s63702 kernel: [517789.091398] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091403] beam.smp cpuset=/ mems_allowed=0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091421] CPU: 3 PID: 5826 Comm: beam.smp Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 16 12:03:29 kvm-s63702 kernel: [517789.091423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091449] 0000000000000000 ffff88061ea7b858 ffffffff817aed00 0000000000000000 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091474] ffff88079c656bf0 ffff88061ea7b8f8 ffffffff817a9c3f 0000000000000000 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091476] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091495] Call Trace: Jan 16 12:03:29 kvm-s63702 kernel: [517789.091535] [] dump_stack+0x45/0x57 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091538] [] dump_header+0x7f/0x1f1 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091562] [] oom_kill_process+0x22b/0x390 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091564] [] out_of_memory+0x4d2/0x520 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091569] [] __alloc_pages_nodemask+0x940/0xa60 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091583] [] alloc_pages_current+0x91/0x100 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091585] [] __page_cache_alloc+0xa7/0xc0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091587] [] filemap_fault+0x1a8/0x430 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091590] [] __do_fault+0x39/0x90 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091592] [] do_read_fault.isra.55+0x1d7/0x2f0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091595] [] handle_mm_fault+0x7e5/0x1100 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091598] [] ? __vm_enough_memory+0x140/0x150 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091612] [] ? cap_vm_enough_memory+0x47/0x50 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091615] [] ? security_vm_enough_memory_mm+0x13/0x20 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091627] [] __do_page_fault+0x1c4/0x5b0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091630] [] ? do_mmap_pgoff+0x2f5/0x3c0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091634] [] ? vm_mmap_pgoff+0x99/0xc0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091636] [] trace_do_page_fault+0x45/0x100 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091639] [] do_async_page_fault+0x1e/0xd0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091648] [] async_page_fault+0x28/0x30 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091649] Mem-Info: Jan 16 12:03:29 kvm-s63702 kernel: [517789.091653] Node 0 DMA per-cpu: Jan 16 12:03:29 kvm-s63702 kernel: [517789.091657] CPU 0: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091658] CPU 1: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091659] CPU 2: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091660] CPU 3: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091661] CPU 4: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091662] CPU 5: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091663] CPU 6: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091664] CPU 7: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091665] CPU 8: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091666] CPU 9: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091667] CPU 10: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091668] CPU 11: hi: 0, btch: 1 usd: 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091669] Node 0 DMA32 per-cpu: Jan 16 12:03:29 kvm-s63702 kernel: [517789.091670] CPU 0: hi: 186, btch: 31 usd: 57 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091671] CPU 1: hi: 186, btch: 31 usd: 44 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091672] CPU 2: hi: 186, btch: 31 usd: 78 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091673] CPU 3: hi: 186, btch: 31 usd: 39 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091674] CPU 4: hi: 186, btch: 31 usd: 35 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091675] CPU 5: hi: 186, btch: 31 usd: 30 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091676] CPU 6: hi: 186, btch: 31 usd: 38 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091677] CPU 7: hi: 186, btch: 31 usd: 50 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091677] CPU 8: hi: 186, btch: 31 usd: 31 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091678] CPU 9: hi: 186, btch: 31 usd: 30 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091679] CPU 10: hi: 186, btch: 31 usd: 29 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091680] CPU 11: hi: 186, btch: 31 usd: 162 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091681] Node 0 Normal per-cpu: Jan 16 12:03:29 kvm-s63702 kernel: [517789.091682] CPU 0: hi: 186, btch: 31 usd: 44 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091683] CPU 1: hi: 186, btch: 31 usd: 25 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091684] CPU 2: hi: 186, btch: 31 usd: 45 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091685] CPU 3: hi: 186, btch: 31 usd: 32 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091686] CPU 4: hi: 186, btch: 31 usd: 30 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091686] CPU 5: hi: 186, btch: 31 usd: 37 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091687] CPU 6: hi: 186, btch: 31 usd: 40 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091688] CPU 7: hi: 186, btch: 31 usd: 73 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091689] CPU 8: hi: 186, btch: 31 usd: 58 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091690] CPU 9: hi: 186, btch: 31 usd: 27 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091691] CPU 10: hi: 186, btch: 31 usd: 31 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091692] CPU 11: hi: 186, btch: 31 usd: 171 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091695] active_anon:7578682 inactive_anon:73 isolated_anon:0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091695] active_file:76 inactive_file:40 isolated_file:0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091695] unevictable:0 dirty:20 writeback:0 unstable:0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091695] free:48114 slab_reclaimable:7514 slab_unreclaimable:14028 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091695] mapped:176 shmem:94 pagetables:21538 bounce:0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091695] free_cma:0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091697] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 16 12:03:29 kvm-s63702 kernel: [517789.091732] lowmem_reserve[]: 0 2978 30143 30143 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091735] Node 0 DMA32 free:115184kB min:6668kB low:8332kB high:10000kB active_anon:2907932kB inactive_anon:32kB active_file:284kB inactive_file:136kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:0kB dirty:0kB writeback:0kB mapped:428kB shmem:32kB slab_reclaimable:2096kB slab_unreclaimable:4112kB kernel_stack:6480kB pagetables:6844kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4160 all_unreclaimable? yes Jan 16 12:03:29 kvm-s63702 kernel: [517789.091738] lowmem_reserve[]: 0 0 27165 27165 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091740] Node 0 Normal free:61380kB min:60876kB low:76092kB high:91312kB active_anon:27406796kB inactive_anon:260kB active_file:20kB inactive_file:24kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:0kB dirty:80kB writeback:0kB mapped:276kB shmem:344kB slab_reclaimable:27960kB slab_unreclaimable:51984kB kernel_stack:78368kB pagetables:79308kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1160 all_unreclaimable? yes Jan 16 12:03:29 kvm-s63702 kernel: [517789.091743] lowmem_reserve[]: 0 0 0 0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091745] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 16 12:03:29 kvm-s63702 kernel: [517789.091769] Node 0 DMA32: 31*4kB (EM) 34*8kB (EM) 89*16kB (UE) 89*32kB (UEM) 66*64kB (UEM) 27*128kB (UE) 24*256kB (UEM) 12*512kB (UE) 4*1024kB (UEM) 6*2048kB (UEM) 18*4096kB (MR) = 114748kB Jan 16 12:03:29 kvm-s63702 kernel: [517789.091777] Node 0 Normal: 297*4kB (EM) 221*8kB (EM) 127*16kB (EM) 203*32kB (UEM) 168*64kB (UEM) 92*128kB (UEM) 42*256kB (UE) 6*512kB (E) 2*1024kB (UM) 3*2048kB (M) 1*4096kB (R) = 60124kB Jan 16 12:03:29 kvm-s63702 kernel: [517789.091787] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 16 12:03:29 kvm-s63702 kernel: [517789.091788] 1135 total pagecache pages Jan 16 12:03:29 kvm-s63702 kernel: [517789.091789] 0 pages in swap cache Jan 16 12:03:29 kvm-s63702 kernel: [517789.091790] Swap cache stats: add 0, delete 0, find 0/0 Jan 16 12:03:29 kvm-s63702 kernel: [517789.091791] Free swap = 0kB Jan 16 12:03:29 kvm-s63702 kernel: [517789.091792] Total swap = 0kB Jan 16 12:03:29 kvm-s63702 kernel: [517789.091793] 7864218 pages RAM Jan 16 12:03:29 kvm-s63702 kernel: [517789.091793] 0 pages HighMem/MovableOnly Jan 16 12:03:29 kvm-s63702 kernel: [517789.091794] 143432 pages reserved Jan 16 12:03:29 kvm-s63702 kernel: [517789.091795] 0 pages cma reserved Jan 16 12:03:29 kvm-s63702 kernel: [517789.091796] 0 pages hwpoisoned Jan 16 12:03:29 kvm-s63702 kernel: [517789.091796] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 16 12:03:29 kvm-s63702 kernel: [517789.091806] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 16 12:03:29 kvm-s63702 kernel: [517789.091808] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 16 12:03:29 kvm-s63702 kernel: [517789.091810] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 16 12:03:29 kvm-s63702 kernel: [517789.091812] [ 724] 0 724 2559 593 8 0 0 dhclient Jan 16 12:03:29 kvm-s63702 kernel: [517789.091813] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 16 12:03:29 kvm-s63702 kernel: [517789.091816] [ 932] 101 932 65020 246 29 0 0 rsyslogd Jan 16 12:03:29 kvm-s63702 kernel: [517789.091818] [ 961] 102 961 9809 96 23 0 0 dbus-daemon Jan 16 12:03:29 kvm-s63702 kernel: [517789.091819] [ 964] 0 964 3956 41 12 0 0 getty Jan 16 12:03:29 kvm-s63702 kernel: [517789.091821] [ 967] 0 967 3956 40 13 0 0 getty Jan 16 12:03:29 kvm-s63702 kernel: [517789.091823] [ 972] 0 972 3956 40 13 0 0 getty Jan 16 12:03:29 kvm-s63702 kernel: [517789.091824] [ 973] 0 973 3956 39 13 0 0 getty Jan 16 12:03:29 kvm-s63702 kernel: [517789.091826] [ 975] 0 975 3956 39 13 0 0 getty Jan 16 12:03:29 kvm-s63702 kernel: [517789.091827] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 16 12:03:29 kvm-s63702 kernel: [517789.091829] [ 998] 0 998 1093 37 8 0 0 acpid Jan 16 12:03:29 kvm-s63702 kernel: [517789.091841] [ 1016] 0 1016 5915 64 18 0 0 cron Jan 16 12:03:29 kvm-s63702 kernel: [517789.091843] [ 1017] 0 1017 4786 41 13 0 0 atd Jan 16 12:03:29 kvm-s63702 kernel: [517789.091845] [ 1046] 0 1046 10864 91 27 0 0 systemd-logind Jan 16 12:03:29 kvm-s63702 kernel: [517789.091847] [ 1052] 0 1052 4799 70 14 0 0 irqbalance Jan 16 12:03:29 kvm-s63702 kernel: [517789.091848] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 16 12:03:29 kvm-s63702 kernel: [517789.091852] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 16 12:03:29 kvm-s63702 kernel: [517789.091853] [ 5396] 1000 5396 353837 8025 89 0 0 beam.smp Jan 16 12:03:29 kvm-s63702 kernel: [517789.091855] [ 5432] 1000 5432 2884 1509 10 0 0 gosecrets Jan 16 12:03:29 kvm-s63702 kernel: [517789.091856] [ 5436] 1000 5436 644910 148474 605 0 0 beam.smp Jan 16 12:03:29 kvm-s63702 kernel: [517789.091858] [ 5474] 1000 5474 1113 28 7 0 0 sh Jan 16 12:03:29 kvm-s63702 kernel: [517789.091859] [ 5476] 1000 5476 1084 25 7 0 0 memsup Jan 16 12:03:29 kvm-s63702 kernel: [517789.091861] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 16 12:03:29 kvm-s63702 kernel: [517789.091863] [ 5479] 1000 5479 1867 23 9 0 0 inet_gethost Jan 16 12:03:29 kvm-s63702 kernel: [517789.091864] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 16 12:03:29 kvm-s63702 kernel: [517789.091866] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 16 12:03:29 kvm-s63702 kernel: [517789.091868] [ 5571] 1000 5571 428648 5410 112 0 0 memcached Jan 16 12:03:29 kvm-s63702 kernel: [517789.091869] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 16 12:03:29 kvm-s63702 kernel: [517789.091871] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 16 12:03:29 kvm-s63702 kernel: [517789.091873] [ 5786] 1000 5786 275364 15294 140 0 0 beam.smp Jan 16 12:03:29 kvm-s63702 kernel: [517789.091874] [ 5854] 1000 5854 1113 27 5 0 0 sh Jan 16 12:03:29 kvm-s63702 kernel: [517789.091876] [ 5856] 1000 5856 1084 26 7 0 0 memsup Jan 16 12:03:29 kvm-s63702 kernel: [517789.091877] [ 5857] 1000 5857 1084 21 8 0 0 cpu_sup Jan 16 12:03:29 kvm-s63702 kernel: [517789.091879] [ 5863] 1000 5863 1331 110 8 0 0 godu Jan 16 12:03:29 kvm-s63702 kernel: [517789.091881] [ 5867] 1000 5867 1113 28 7 0 0 sh Jan 16 12:03:29 kvm-s63702 kernel: [517789.091882] [ 5868] 1000 5868 1331 109 8 0 0 godu Jan 16 12:03:29 kvm-s63702 kernel: [517789.091883] [ 5878] 1000 5878 2156 55 9 0 0 sigar_port Jan 16 12:03:29 kvm-s63702 kernel: [517789.091885] [ 5879] 1000 5879 1462 164 8 0 0 goport Jan 16 12:03:29 kvm-s63702 kernel: [517789.091886] [ 5884] 1000 5884 74324 2236 32 0 0 goxdcr Jan 16 12:03:29 kvm-s63702 kernel: [517789.091888] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 16 12:03:29 kvm-s63702 kernel: [517789.091889] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 16 12:03:29 kvm-s63702 kernel: [517789.091892] [26330] 1000 26330 1526 156 8 0 0 goport Jan 16 12:03:29 kvm-s63702 kernel: [517789.091893] [26335] 1000 26335 21757528 7251485 19975 0 0 indexer Jan 16 12:03:29 kvm-s63702 kernel: [517789.091895] Out of memory: Kill process 26335 (indexer) score 941 or sacrifice child Jan 16 12:03:29 kvm-s63702 kernel: [517789.092154] Killed process 26335 (indexer) total-vm:87030112kB, anon-rss:29005940kB, file-rss:0kB Jan 16 14:35:47 kvm-s63702 kernel: [526927.392238] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392243] indexer cpuset=/ mems_allowed=0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392259] CPU: 6 PID: 32320 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 16 14:35:47 kvm-s63702 kernel: [526927.392260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392262] 0000000000000000 ffff8800951479d8 ffffffff817aed00 0000000000000000 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392264] ffff88065b074e80 ffff880095147a78 ffffffff817a9c3f 0000000000000000 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392266] 0000000000000000 0000000000000000 0000000000000000 ffff880095147ac8 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392268] Call Trace: Jan 16 14:35:47 kvm-s63702 kernel: [526927.392308] [] dump_stack+0x45/0x57 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392311] [] dump_header+0x7f/0x1f1 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392329] [] oom_kill_process+0x22b/0x390 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392332] [] out_of_memory+0x4d2/0x520 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392336] [] __alloc_pages_nodemask+0x940/0xa60 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392361] [] alloc_pages_vma+0x97/0x150 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392368] [] handle_mm_fault+0xd13/0x1100 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392381] [] __do_page_fault+0x1c4/0x5b0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392391] [] ? __enqueue_entity+0x78/0x80 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392393] [] ? set_next_entity+0x9d/0xb0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392395] [] ? pick_next_task_fair+0x61b/0x880 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392406] [] ? __switch_to+0x15f/0x570 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392408] [] ? sched_clock_cpu+0x98/0xc0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392410] [] trace_do_page_fault+0x45/0x100 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392414] [] do_async_page_fault+0x1e/0xd0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392437] [] async_page_fault+0x28/0x30 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392438] Mem-Info: Jan 16 14:35:47 kvm-s63702 kernel: [526927.392442] Node 0 DMA per-cpu: Jan 16 14:35:47 kvm-s63702 kernel: [526927.392445] CPU 0: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392446] CPU 1: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392447] CPU 2: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392448] CPU 3: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392449] CPU 4: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392450] CPU 5: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392451] CPU 6: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392452] CPU 7: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392453] CPU 8: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392454] CPU 9: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392455] CPU 10: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392456] CPU 11: hi: 0, btch: 1 usd: 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392457] Node 0 DMA32 per-cpu: Jan 16 14:35:47 kvm-s63702 kernel: [526927.392458] CPU 0: hi: 186, btch: 31 usd: 59 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392459] CPU 1: hi: 186, btch: 31 usd: 50 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392460] CPU 2: hi: 186, btch: 31 usd: 150 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392461] CPU 3: hi: 186, btch: 31 usd: 36 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392462] CPU 4: hi: 186, btch: 31 usd: 58 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392463] CPU 5: hi: 186, btch: 31 usd: 47 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392464] CPU 6: hi: 186, btch: 31 usd: 31 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392465] CPU 7: hi: 186, btch: 31 usd: 31 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392466] CPU 8: hi: 186, btch: 31 usd: 89 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392467] CPU 9: hi: 186, btch: 31 usd: 22 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392468] CPU 10: hi: 186, btch: 31 usd: 23 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392469] CPU 11: hi: 186, btch: 31 usd: 19 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392469] Node 0 Normal per-cpu: Jan 16 14:35:47 kvm-s63702 kernel: [526927.392471] CPU 0: hi: 186, btch: 31 usd: 17 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392471] CPU 1: hi: 186, btch: 31 usd: 31 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392472] CPU 2: hi: 186, btch: 31 usd: 132 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392473] CPU 3: hi: 186, btch: 31 usd: 30 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392474] CPU 4: hi: 186, btch: 31 usd: 164 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392475] CPU 5: hi: 186, btch: 31 usd: 5 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392476] CPU 6: hi: 186, btch: 31 usd: 30 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392477] CPU 7: hi: 186, btch: 31 usd: 25 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392478] CPU 8: hi: 186, btch: 31 usd: 64 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392479] CPU 9: hi: 186, btch: 31 usd: 49 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392479] CPU 10: hi: 186, btch: 31 usd: 106 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392480] CPU 11: hi: 186, btch: 31 usd: 6 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392484] active_anon:7594254 inactive_anon:73 isolated_anon:0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392484] active_file:302 inactive_file:100 isolated_file:0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392484] unevictable:0 dirty:0 writeback:0 unstable:0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392484] free:47924 slab_reclaimable:6873 slab_unreclaimable:8959 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392484] mapped:207 shmem:94 pagetables:19494 bounce:0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392484] free_cma:0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392486] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 16 14:35:47 kvm-s63702 kernel: [526927.392493] lowmem_reserve[]: 0 2978 30143 30143 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392495] Node 0 DMA32 free:115264kB min:6668kB low:8332kB high:10000kB active_anon:2908180kB inactive_anon:40kB active_file:12kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:40kB slab_reclaimable:2108kB slab_unreclaimable:2832kB kernel_stack:2384kB pagetables:8404kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:40740 all_unreclaimable? yes Jan 16 14:35:47 kvm-s63702 kernel: [526927.392508] lowmem_reserve[]: 0 0 27165 27165 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392510] Node 0 Normal free:60540kB min:60876kB low:76092kB high:91312kB active_anon:27468836kB inactive_anon:252kB active_file:1196kB inactive_file:388kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:0kB dirty:0kB writeback:0kB mapped:824kB shmem:336kB slab_reclaimable:25384kB slab_unreclaimable:32988kB kernel_stack:20976kB pagetables:69572kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:23484 all_unreclaimable? yes Jan 16 14:35:47 kvm-s63702 kernel: [526927.392513] lowmem_reserve[]: 0 0 0 0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392515] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 16 14:35:47 kvm-s63702 kernel: [526927.392525] Node 0 DMA32: 60*4kB (E) 62*8kB (UE) 102*16kB (UE) 88*32kB (UE) 58*64kB (UE) 23*128kB (UEM) 20*256kB (UE) 12*512kB (UE) 4*1024kB (UE) 7*2048kB (UEM) 18*4096kB (MR) = 115264kB Jan 16 14:35:47 kvm-s63702 kernel: [526927.392533] Node 0 Normal: 325*4kB (UEM) 245*8kB (UEM) 349*16kB (UEM) 244*32kB (UEM) 178*64kB (UE) 106*128kB (UEM) 43*256kB (UEM) 5*512kB (EM) 0*1024kB 0*2048kB 1*4096kB (R) = 59276kB Jan 16 14:35:47 kvm-s63702 kernel: [526927.392542] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 16 14:35:47 kvm-s63702 kernel: [526927.392543] 977 total pagecache pages Jan 16 14:35:47 kvm-s63702 kernel: [526927.392544] 0 pages in swap cache Jan 16 14:35:47 kvm-s63702 kernel: [526927.392545] Swap cache stats: add 0, delete 0, find 0/0 Jan 16 14:35:47 kvm-s63702 kernel: [526927.392546] Free swap = 0kB Jan 16 14:35:47 kvm-s63702 kernel: [526927.392547] Total swap = 0kB Jan 16 14:35:47 kvm-s63702 kernel: [526927.392548] 7864218 pages RAM Jan 16 14:35:47 kvm-s63702 kernel: [526927.392548] 0 pages HighMem/MovableOnly Jan 16 14:35:47 kvm-s63702 kernel: [526927.392549] 143432 pages reserved Jan 16 14:35:47 kvm-s63702 kernel: [526927.392550] 0 pages cma reserved Jan 16 14:35:47 kvm-s63702 kernel: [526927.392550] 0 pages hwpoisoned Jan 16 14:35:47 kvm-s63702 kernel: [526927.392551] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 16 14:35:47 kvm-s63702 kernel: [526927.392573] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 16 14:35:47 kvm-s63702 kernel: [526927.392575] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 16 14:35:47 kvm-s63702 kernel: [526927.392577] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 16 14:35:47 kvm-s63702 kernel: [526927.392579] [ 724] 0 724 2559 593 8 0 0 dhclient Jan 16 14:35:47 kvm-s63702 kernel: [526927.392580] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 16 14:35:47 kvm-s63702 kernel: [526927.392582] [ 932] 101 932 65020 246 29 0 0 rsyslogd Jan 16 14:35:47 kvm-s63702 kernel: [526927.392584] [ 961] 102 961 9809 96 23 0 0 dbus-daemon Jan 16 14:35:47 kvm-s63702 kernel: [526927.392585] [ 964] 0 964 3956 41 12 0 0 getty Jan 16 14:35:47 kvm-s63702 kernel: [526927.392587] [ 967] 0 967 3956 40 13 0 0 getty Jan 16 14:35:47 kvm-s63702 kernel: [526927.392588] [ 972] 0 972 3956 40 13 0 0 getty Jan 16 14:35:47 kvm-s63702 kernel: [526927.392590] [ 973] 0 973 3956 39 13 0 0 getty Jan 16 14:35:47 kvm-s63702 kernel: [526927.392591] [ 975] 0 975 3956 39 13 0 0 getty Jan 16 14:35:47 kvm-s63702 kernel: [526927.392593] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 16 14:35:47 kvm-s63702 kernel: [526927.392594] [ 998] 0 998 1093 37 8 0 0 acpid Jan 16 14:35:47 kvm-s63702 kernel: [526927.392596] [ 1016] 0 1016 5915 65 18 0 0 cron Jan 16 14:35:47 kvm-s63702 kernel: [526927.392597] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 16 14:35:47 kvm-s63702 kernel: [526927.392599] [ 1046] 0 1046 10864 91 27 0 0 systemd-logind Jan 16 14:35:47 kvm-s63702 kernel: [526927.392600] [ 1052] 0 1052 4799 70 14 0 0 irqbalance Jan 16 14:35:47 kvm-s63702 kernel: [526927.392602] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 16 14:35:47 kvm-s63702 kernel: [526927.392604] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 16 14:35:47 kvm-s63702 kernel: [526927.392606] [ 5396] 1000 5396 352397 8244 89 0 0 beam.smp Jan 16 14:35:47 kvm-s63702 kernel: [526927.392607] [ 5432] 1000 5432 2884 1467 10 0 0 gosecrets Jan 16 14:35:47 kvm-s63702 kernel: [526927.392609] [ 5436] 1000 5436 661031 162325 609 0 0 beam.smp Jan 16 14:35:47 kvm-s63702 kernel: [526927.392611] [ 5474] 1000 5474 1113 28 7 0 0 sh Jan 16 14:35:47 kvm-s63702 kernel: [526927.392624] [ 5476] 1000 5476 1084 26 7 0 0 memsup Jan 16 14:35:47 kvm-s63702 kernel: [526927.392626] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 16 14:35:47 kvm-s63702 kernel: [526927.392627] [ 5479] 1000 5479 1867 23 9 0 0 inet_gethost Jan 16 14:35:47 kvm-s63702 kernel: [526927.392629] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 16 14:35:47 kvm-s63702 kernel: [526927.392630] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 16 14:35:47 kvm-s63702 kernel: [526927.392632] [ 5571] 1000 5571 428648 5410 112 0 0 memcached Jan 16 14:35:47 kvm-s63702 kernel: [526927.392633] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 16 14:35:47 kvm-s63702 kernel: [526927.392635] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 16 14:35:47 kvm-s63702 kernel: [526927.392637] [ 5786] 1000 5786 294124 16651 141 0 0 beam.smp Jan 16 14:35:47 kvm-s63702 kernel: [526927.392638] [ 5854] 1000 5854 1113 28 5 0 0 sh Jan 16 14:35:47 kvm-s63702 kernel: [526927.392640] [ 5856] 1000 5856 1084 26 7 0 0 memsup Jan 16 14:35:47 kvm-s63702 kernel: [526927.392641] [ 5857] 1000 5857 1084 21 8 0 0 cpu_sup Jan 16 14:35:47 kvm-s63702 kernel: [526927.392643] [ 5863] 1000 5863 1331 110 8 0 0 godu Jan 16 14:35:47 kvm-s63702 kernel: [526927.392644] [ 5867] 1000 5867 1113 28 7 0 0 sh Jan 16 14:35:47 kvm-s63702 kernel: [526927.392646] [ 5868] 1000 5868 1331 109 8 0 0 godu Jan 16 14:35:47 kvm-s63702 kernel: [526927.392647] [ 5878] 1000 5878 2156 56 9 0 0 sigar_port Jan 16 14:35:47 kvm-s63702 kernel: [526927.392649] [ 5879] 1000 5879 1462 165 8 0 0 goport Jan 16 14:35:47 kvm-s63702 kernel: [526927.392650] [ 5884] 1000 5884 74324 1962 32 0 0 goxdcr Jan 16 14:35:47 kvm-s63702 kernel: [526927.392652] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 16 14:35:47 kvm-s63702 kernel: [526927.392653] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 16 14:35:47 kvm-s63702 kernel: [526927.392655] [31298] 1000 31298 1526 153 8 0 0 goport Jan 16 14:35:47 kvm-s63702 kernel: [526927.392657] [31303] 1000 31303 14001940 7350877 17926 0 0 indexer Jan 16 14:35:47 kvm-s63702 kernel: [526927.392658] Out of memory: Kill process 31303 (indexer) score 954 or sacrifice child Jan 16 14:35:47 kvm-s63702 kernel: [526927.392750] Killed process 31303 (indexer) total-vm:56007760kB, anon-rss:29403508kB, file-rss:0kB Jan 16 14:48:15 kvm-s63702 kernel: [527675.039866] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039871] indexer cpuset=/ mems_allowed=0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039887] CPU: 2 PID: 5230 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 16 14:48:15 kvm-s63702 kernel: [527675.039889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039890] 0000000000000000 ffff8801bede39d8 ffffffff817aed00 0000000000000000 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039893] ffff88064a4093a0 ffff8801bede3a78 ffffffff817a9c3f 0000000000000000 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039894] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039897] Call Trace: Jan 16 14:48:15 kvm-s63702 kernel: [527675.039937] [] dump_stack+0x45/0x57 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039940] [] dump_header+0x7f/0x1f1 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039957] [] oom_kill_process+0x22b/0x390 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039960] [] out_of_memory+0x4d2/0x520 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039963] [] __alloc_pages_nodemask+0x940/0xa60 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039974] [] alloc_pages_vma+0x97/0x150 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039979] [] handle_mm_fault+0xd13/0x1100 Jan 16 14:48:15 kvm-s63702 kernel: [527675.039994] [] ? ttwu_do_activate.constprop.94+0x5d/0x70 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040005] [] ? call_rwsem_down_read_failed+0x14/0x30 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040046] [] __do_page_fault+0x1c4/0x5b0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040052] [] ? __enqueue_entity+0x78/0x80 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040054] [] ? set_next_entity+0x9d/0xb0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040057] [] ? pick_next_task_fair+0x61b/0x880 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040068] [] ? __switch_to+0x15f/0x570 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040070] [] ? sched_clock_cpu+0x98/0xc0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040072] [] trace_do_page_fault+0x45/0x100 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040076] [] do_async_page_fault+0x1e/0xd0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040112] [] async_page_fault+0x28/0x30 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040114] Mem-Info: Jan 16 14:48:15 kvm-s63702 kernel: [527675.040118] Node 0 DMA per-cpu: Jan 16 14:48:15 kvm-s63702 kernel: [527675.040122] CPU 0: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040124] CPU 1: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040125] CPU 2: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040126] CPU 3: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040127] CPU 4: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040128] CPU 5: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040129] CPU 6: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040130] CPU 7: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040131] CPU 8: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040133] CPU 9: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040134] CPU 10: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040135] CPU 11: hi: 0, btch: 1 usd: 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040136] Node 0 DMA32 per-cpu: Jan 16 14:48:15 kvm-s63702 kernel: [527675.040137] CPU 0: hi: 186, btch: 31 usd: 50 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040138] CPU 1: hi: 186, btch: 31 usd: 59 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040139] CPU 2: hi: 186, btch: 31 usd: 41 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040140] CPU 3: hi: 186, btch: 31 usd: 38 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040141] CPU 4: hi: 186, btch: 31 usd: 80 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040142] CPU 5: hi: 186, btch: 31 usd: 60 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040143] CPU 6: hi: 186, btch: 31 usd: 44 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040144] CPU 7: hi: 186, btch: 31 usd: 57 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040145] CPU 8: hi: 186, btch: 31 usd: 40 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040146] CPU 9: hi: 186, btch: 31 usd: 36 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040147] CPU 10: hi: 186, btch: 31 usd: 43 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040147] CPU 11: hi: 186, btch: 31 usd: 10 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040148] Node 0 Normal per-cpu: Jan 16 14:48:15 kvm-s63702 kernel: [527675.040149] CPU 0: hi: 186, btch: 31 usd: 40 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040150] CPU 1: hi: 186, btch: 31 usd: 30 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040151] CPU 2: hi: 186, btch: 31 usd: 13 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040152] CPU 3: hi: 186, btch: 31 usd: 51 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040153] CPU 4: hi: 186, btch: 31 usd: 9 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040154] CPU 5: hi: 186, btch: 31 usd: 30 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040155] CPU 6: hi: 186, btch: 31 usd: 15 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040156] CPU 7: hi: 186, btch: 31 usd: 8 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040157] CPU 8: hi: 186, btch: 31 usd: 41 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040158] CPU 9: hi: 186, btch: 31 usd: 30 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040159] CPU 10: hi: 186, btch: 31 usd: 44 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040160] CPU 11: hi: 186, btch: 31 usd: 25 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040163] active_anon:7593034 inactive_anon:73 isolated_anon:0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040163] active_file:217 inactive_file:76 isolated_file:0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040163] unevictable:0 dirty:200 writeback:0 unstable:0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040163] free:47968 slab_reclaimable:7053 slab_unreclaimable:11517 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040163] mapped:128 shmem:94 pagetables:19922 bounce:0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040163] free_cma:0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040165] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 16 14:48:15 kvm-s63702 kernel: [527675.040172] lowmem_reserve[]: 0 2978 30143 30143 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040174] Node 0 DMA32 free:115140kB min:6668kB low:8332kB high:10000kB active_anon:2907132kB inactive_anon:40kB active_file:28kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:0kB dirty:8kB writeback:0kB mapped:0kB shmem:40kB slab_reclaimable:2328kB slab_unreclaimable:3976kB kernel_stack:6208kB pagetables:7956kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:47412 all_unreclaimable? yes Jan 16 14:48:15 kvm-s63702 kernel: [527675.040178] lowmem_reserve[]: 0 0 27165 27165 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040180] Node 0 Normal free:60840kB min:60876kB low:76092kB high:91312kB active_anon:27465004kB inactive_anon:252kB active_file:840kB inactive_file:524kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:0kB dirty:792kB writeback:0kB mapped:544kB shmem:336kB slab_reclaimable:25884kB slab_unreclaimable:42076kB kernel_stack:49776kB pagetables:71732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:176440 all_unreclaimable? yes Jan 16 14:48:15 kvm-s63702 kernel: [527675.040184] lowmem_reserve[]: 0 0 0 0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040187] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 16 14:48:15 kvm-s63702 kernel: [527675.040215] Node 0 DMA32: 44*4kB (UE) 41*8kB (UEM) 69*16kB (UEM) 78*32kB (UEM) 61*64kB (UEM) 32*128kB (UEM) 17*256kB (UE) 12*512kB (UEM) 4*1024kB (UE) 7*2048kB (UEM) 18*4096kB (MR) = 114760kB Jan 16 14:48:15 kvm-s63702 kernel: [527675.040223] Node 0 Normal: 177*4kB (UE) 134*8kB (UE) 184*16kB (EM) 209*32kB (UE) 151*64kB (UE) 74*128kB (EM) 75*256kB (UE) 11*512kB (U) 0*1024kB 0*2048kB 1*4096kB (R) = 59476kB Jan 16 14:48:15 kvm-s63702 kernel: [527675.040232] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 16 14:48:15 kvm-s63702 kernel: [527675.040233] 658 total pagecache pages Jan 16 14:48:15 kvm-s63702 kernel: [527675.040235] 0 pages in swap cache Jan 16 14:48:15 kvm-s63702 kernel: [527675.040236] Swap cache stats: add 0, delete 0, find 0/0 Jan 16 14:48:15 kvm-s63702 kernel: [527675.040237] Free swap = 0kB Jan 16 14:48:15 kvm-s63702 kernel: [527675.040238] Total swap = 0kB Jan 16 14:48:15 kvm-s63702 kernel: [527675.040239] 7864218 pages RAM Jan 16 14:48:15 kvm-s63702 kernel: [527675.040239] 0 pages HighMem/MovableOnly Jan 16 14:48:15 kvm-s63702 kernel: [527675.040240] 143432 pages reserved Jan 16 14:48:15 kvm-s63702 kernel: [527675.040241] 0 pages cma reserved Jan 16 14:48:15 kvm-s63702 kernel: [527675.040241] 0 pages hwpoisoned Jan 16 14:48:15 kvm-s63702 kernel: [527675.040242] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 16 14:48:15 kvm-s63702 kernel: [527675.040249] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 16 14:48:15 kvm-s63702 kernel: [527675.040251] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 16 14:48:15 kvm-s63702 kernel: [527675.040254] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 16 14:48:15 kvm-s63702 kernel: [527675.040255] [ 724] 0 724 2559 593 8 0 0 dhclient Jan 16 14:48:15 kvm-s63702 kernel: [527675.040257] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 16 14:48:15 kvm-s63702 kernel: [527675.040259] [ 932] 101 932 65020 247 29 0 0 rsyslogd Jan 16 14:48:15 kvm-s63702 kernel: [527675.040260] [ 961] 102 961 9809 96 23 0 0 dbus-daemon Jan 16 14:48:15 kvm-s63702 kernel: [527675.040262] [ 964] 0 964 3956 41 12 0 0 getty Jan 16 14:48:15 kvm-s63702 kernel: [527675.040264] [ 967] 0 967 3956 40 13 0 0 getty Jan 16 14:48:15 kvm-s63702 kernel: [527675.040265] [ 972] 0 972 3956 40 13 0 0 getty Jan 16 14:48:15 kvm-s63702 kernel: [527675.040267] [ 973] 0 973 3956 39 13 0 0 getty Jan 16 14:48:15 kvm-s63702 kernel: [527675.040268] [ 975] 0 975 3956 39 13 0 0 getty Jan 16 14:48:15 kvm-s63702 kernel: [527675.040270] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 16 14:48:15 kvm-s63702 kernel: [527675.040272] [ 998] 0 998 1093 37 8 0 0 acpid Jan 16 14:48:15 kvm-s63702 kernel: [527675.040273] [ 1016] 0 1016 5915 64 18 0 0 cron Jan 16 14:48:15 kvm-s63702 kernel: [527675.040275] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 16 14:48:15 kvm-s63702 kernel: [527675.040277] [ 1046] 0 1046 10864 91 27 0 0 systemd-logind Jan 16 14:48:15 kvm-s63702 kernel: [527675.040278] [ 1052] 0 1052 4799 86 14 0 0 irqbalance Jan 16 14:48:15 kvm-s63702 kernel: [527675.040280] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 16 14:48:15 kvm-s63702 kernel: [527675.040282] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 16 14:48:15 kvm-s63702 kernel: [527675.040284] [ 5396] 1000 5396 352268 7899 90 0 0 beam.smp Jan 16 14:48:15 kvm-s63702 kernel: [527675.040286] [ 5432] 1000 5432 2884 1494 10 0 0 gosecrets Jan 16 14:48:15 kvm-s63702 kernel: [527675.040287] [ 5436] 1000 5436 652886 151603 576 0 0 beam.smp Jan 16 14:48:15 kvm-s63702 kernel: [527675.040289] [ 5474] 1000 5474 1113 36 7 0 0 sh Jan 16 14:48:15 kvm-s63702 kernel: [527675.040291] [ 5476] 1000 5476 1084 37 7 0 0 memsup Jan 16 14:48:15 kvm-s63702 kernel: [527675.040292] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 16 14:48:15 kvm-s63702 kernel: [527675.040294] [ 5479] 1000 5479 1867 23 9 0 0 inet_gethost Jan 16 14:48:15 kvm-s63702 kernel: [527675.040295] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 16 14:48:15 kvm-s63702 kernel: [527675.040297] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 16 14:48:15 kvm-s63702 kernel: [527675.040298] [ 5571] 1000 5571 428648 5411 112 0 0 memcached Jan 16 14:48:15 kvm-s63702 kernel: [527675.040300] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 16 14:48:15 kvm-s63702 kernel: [527675.040304] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 16 14:48:15 kvm-s63702 kernel: [527675.040306] [ 5786] 1000 5786 291586 15293 135 0 0 beam.smp Jan 16 14:48:15 kvm-s63702 kernel: [527675.040307] [ 5854] 1000 5854 1113 27 5 0 0 sh Jan 16 14:48:15 kvm-s63702 kernel: [527675.040309] [ 5856] 1000 5856 1084 37 7 0 0 memsup Jan 16 14:48:15 kvm-s63702 kernel: [527675.040310] [ 5857] 1000 5857 1084 21 8 0 0 cpu_sup Jan 16 14:48:15 kvm-s63702 kernel: [527675.040312] [ 5863] 1000 5863 1331 110 8 0 0 godu Jan 16 14:48:15 kvm-s63702 kernel: [527675.040314] [ 5867] 1000 5867 1113 38 7 0 0 sh Jan 16 14:48:15 kvm-s63702 kernel: [527675.040315] [ 5868] 1000 5868 1331 109 8 0 0 godu Jan 16 14:48:15 kvm-s63702 kernel: [527675.040317] [ 5878] 1000 5878 2156 71 9 0 0 sigar_port Jan 16 14:48:15 kvm-s63702 kernel: [527675.040318] [ 5879] 1000 5879 1462 165 8 0 0 goport Jan 16 14:48:15 kvm-s63702 kernel: [527675.040320] [ 5884] 1000 5884 74324 2075 32 0 0 goxdcr Jan 16 14:48:15 kvm-s63702 kernel: [527675.040321] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 16 14:48:15 kvm-s63702 kernel: [527675.040323] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 16 14:48:15 kvm-s63702 kernel: [527675.040325] [ 3339] 1000 3339 1462 151 8 0 0 goport Jan 16 14:48:15 kvm-s63702 kernel: [527675.040327] [ 3344] 1000 3344 18185823 7295145 18394 0 0 indexer Jan 16 14:48:15 kvm-s63702 kernel: [527675.040328] Out of memory: Kill process 3344 (indexer) score 947 or sacrifice child Jan 16 14:48:15 kvm-s63702 kernel: [527675.040493] Killed process 3344 (indexer) total-vm:72743292kB, anon-rss:29180452kB, file-rss:128kB Jan 16 17:29:40 kvm-s63702 kernel: [537360.195259] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195265] indexer cpuset=/ mems_allowed=0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195281] CPU: 11 PID: 7519 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 16 17:29:40 kvm-s63702 kernel: [537360.195283] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195285] 0000000000000000 ffff88013ba479d8 ffffffff817aed00 0000000000000000 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195287] ffff880663e95850 ffff88013ba47a78 ffffffff817a9c3f 0000000000000000 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195289] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195291] Call Trace: Jan 16 17:29:40 kvm-s63702 kernel: [537360.195334] [] dump_stack+0x45/0x57 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195337] [] dump_header+0x7f/0x1f1 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195355] [] oom_kill_process+0x22b/0x390 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195358] [] out_of_memory+0x4d2/0x520 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195363] [] __alloc_pages_nodemask+0x940/0xa60 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195373] [] alloc_pages_vma+0x97/0x150 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195380] [] handle_mm_fault+0xd13/0x1100 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195392] [] ? x2apic_send_IPI_mask+0x13/0x20 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195400] [] ? try_to_wake_up+0x1f1/0x340 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195408] [] __do_page_fault+0x1c4/0x5b0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195414] [] ? __enqueue_entity+0x78/0x80 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195423] [] ? do_futex+0x10a/0x5d0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195426] [] ? pick_next_task_fair+0x61b/0x880 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195438] [] ? __switch_to+0x15f/0x570 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195440] [] ? sched_clock_cpu+0x98/0xc0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195442] [] trace_do_page_fault+0x45/0x100 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195446] [] do_async_page_fault+0x1e/0xd0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195456] [] async_page_fault+0x28/0x30 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195457] Mem-Info: Jan 16 17:29:40 kvm-s63702 kernel: [537360.195461] Node 0 DMA per-cpu: Jan 16 17:29:40 kvm-s63702 kernel: [537360.195465] CPU 0: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195466] CPU 1: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195468] CPU 2: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195469] CPU 3: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195470] CPU 4: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195471] CPU 5: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195472] CPU 6: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195473] CPU 7: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195475] CPU 8: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195476] CPU 9: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195477] CPU 10: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195480] CPU 11: hi: 0, btch: 1 usd: 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195481] Node 0 DMA32 per-cpu: Jan 16 17:29:40 kvm-s63702 kernel: [537360.195483] CPU 0: hi: 186, btch: 31 usd: 31 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195484] CPU 1: hi: 186, btch: 31 usd: 5 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195485] CPU 2: hi: 186, btch: 31 usd: 37 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195486] CPU 3: hi: 186, btch: 31 usd: 26 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195487] CPU 4: hi: 186, btch: 31 usd: 26 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195488] CPU 5: hi: 186, btch: 31 usd: 30 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195489] CPU 6: hi: 186, btch: 31 usd: 53 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195490] CPU 7: hi: 186, btch: 31 usd: 49 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195491] CPU 8: hi: 186, btch: 31 usd: 34 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195492] CPU 9: hi: 186, btch: 31 usd: 166 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195493] CPU 10: hi: 186, btch: 31 usd: 16 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195494] CPU 11: hi: 186, btch: 31 usd: 21 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195495] Node 0 Normal per-cpu: Jan 16 17:29:40 kvm-s63702 kernel: [537360.195496] CPU 0: hi: 186, btch: 31 usd: 26 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195497] CPU 1: hi: 186, btch: 31 usd: 55 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195498] CPU 2: hi: 186, btch: 31 usd: 50 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195499] CPU 3: hi: 186, btch: 31 usd: 25 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195500] CPU 4: hi: 186, btch: 31 usd: 32 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195501] CPU 5: hi: 186, btch: 31 usd: 42 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195502] CPU 6: hi: 186, btch: 31 usd: 6 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195503] CPU 7: hi: 186, btch: 31 usd: 27 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195504] CPU 8: hi: 186, btch: 31 usd: 49 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195505] CPU 9: hi: 186, btch: 31 usd: 71 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195505] CPU 10: hi: 186, btch: 31 usd: 34 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195506] CPU 11: hi: 186, btch: 31 usd: 33 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195519] active_anon:7596487 inactive_anon:73 isolated_anon:0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195519] active_file:304 inactive_file:198 isolated_file:0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195519] unevictable:0 dirty:205 writeback:0 unstable:0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195519] free:47999 slab_reclaimable:7418 slab_unreclaimable:8785 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195519] mapped:199 shmem:94 pagetables:19495 bounce:0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195519] free_cma:0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195522] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 16 17:29:40 kvm-s63702 kernel: [537360.195530] lowmem_reserve[]: 0 2978 30143 30143 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195532] Node 0 DMA32 free:115308kB min:6668kB low:8332kB high:10000kB active_anon:2910044kB inactive_anon:32kB active_file:100kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:0kB dirty:4kB writeback:0kB mapped:136kB shmem:36kB slab_reclaimable:2224kB slab_unreclaimable:2556kB kernel_stack:1520kB pagetables:8024kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1524 all_unreclaimable? yes Jan 16 17:29:40 kvm-s63702 kernel: [537360.195537] lowmem_reserve[]: 0 0 27165 27165 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195539] Node 0 Normal free:60796kB min:60876kB low:76092kB high:91312kB active_anon:27475904kB inactive_anon:260kB active_file:1116kB inactive_file:884kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:0kB dirty:816kB writeback:0kB mapped:660kB shmem:340kB slab_reclaimable:27448kB slab_unreclaimable:32568kB kernel_stack:21792kB pagetables:69956kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:334656 all_unreclaimable? yes Jan 16 17:29:40 kvm-s63702 kernel: [537360.195542] lowmem_reserve[]: 0 0 0 0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195544] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 16 17:29:40 kvm-s63702 kernel: [537360.195555] Node 0 DMA32: 77*4kB (UEM) 66*8kB (UEM) 114*16kB (UE) 91*32kB (UEM) 51*64kB (UE) 33*128kB (UE) 23*256kB (UE) 14*512kB (UEM) 5*1024kB (UEM) 5*2048kB (UEM) 18*4096kB (MR) = 115204kB Jan 16 17:29:40 kvm-s63702 kernel: [537360.195564] Node 0 Normal: 312*4kB (UEM) 224*8kB (UE) 324*16kB (UEM) 281*32kB (UE) 143*64kB (UE) 63*128kB (UEM) 80*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 59008kB Jan 16 17:29:40 kvm-s63702 kernel: [537360.195573] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 16 17:29:40 kvm-s63702 kernel: [537360.195574] 721 total pagecache pages Jan 16 17:29:40 kvm-s63702 kernel: [537360.195576] 0 pages in swap cache Jan 16 17:29:40 kvm-s63702 kernel: [537360.195579] Swap cache stats: add 0, delete 0, find 0/0 Jan 16 17:29:40 kvm-s63702 kernel: [537360.195580] Free swap = 0kB Jan 16 17:29:40 kvm-s63702 kernel: [537360.195581] Total swap = 0kB Jan 16 17:29:40 kvm-s63702 kernel: [537360.195582] 7864218 pages RAM Jan 16 17:29:40 kvm-s63702 kernel: [537360.195583] 0 pages HighMem/MovableOnly Jan 16 17:29:40 kvm-s63702 kernel: [537360.195584] 143432 pages reserved Jan 16 17:29:40 kvm-s63702 kernel: [537360.195585] 0 pages cma reserved Jan 16 17:29:40 kvm-s63702 kernel: [537360.195585] 0 pages hwpoisoned Jan 16 17:29:40 kvm-s63702 kernel: [537360.195586] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 16 17:29:40 kvm-s63702 kernel: [537360.195594] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 16 17:29:40 kvm-s63702 kernel: [537360.195596] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 16 17:29:40 kvm-s63702 kernel: [537360.195598] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 16 17:29:40 kvm-s63702 kernel: [537360.195600] [ 724] 0 724 2559 593 8 0 0 dhclient Jan 16 17:29:40 kvm-s63702 kernel: [537360.195602] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 16 17:29:40 kvm-s63702 kernel: [537360.195604] [ 932] 101 932 65020 247 29 0 0 rsyslogd Jan 16 17:29:40 kvm-s63702 kernel: [537360.195606] [ 961] 102 961 9809 109 23 0 0 dbus-daemon Jan 16 17:29:40 kvm-s63702 kernel: [537360.195607] [ 964] 0 964 3956 41 12 0 0 getty Jan 16 17:29:40 kvm-s63702 kernel: [537360.195609] [ 967] 0 967 3956 40 13 0 0 getty Jan 16 17:29:40 kvm-s63702 kernel: [537360.195611] [ 972] 0 972 3956 40 13 0 0 getty Jan 16 17:29:40 kvm-s63702 kernel: [537360.195612] [ 973] 0 973 3956 39 13 0 0 getty Jan 16 17:29:40 kvm-s63702 kernel: [537360.195614] [ 975] 0 975 3956 39 13 0 0 getty Jan 16 17:29:40 kvm-s63702 kernel: [537360.195616] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 16 17:29:40 kvm-s63702 kernel: [537360.195618] [ 998] 0 998 1093 37 8 0 0 acpid Jan 16 17:29:40 kvm-s63702 kernel: [537360.195619] [ 1016] 0 1016 5915 71 18 0 0 cron Jan 16 17:29:40 kvm-s63702 kernel: [537360.195621] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 16 17:29:40 kvm-s63702 kernel: [537360.195623] [ 1046] 0 1046 10864 105 27 0 0 systemd-logind Jan 16 17:29:40 kvm-s63702 kernel: [537360.195624] [ 1052] 0 1052 4799 82 14 0 0 irqbalance Jan 16 17:29:40 kvm-s63702 kernel: [537360.195626] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 16 17:29:40 kvm-s63702 kernel: [537360.195629] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 16 17:29:40 kvm-s63702 kernel: [537360.195631] [ 5396] 1000 5396 351904 8225 88 0 0 beam.smp Jan 16 17:29:40 kvm-s63702 kernel: [537360.195633] [ 5432] 1000 5432 2884 1468 10 0 0 gosecrets Jan 16 17:29:40 kvm-s63702 kernel: [537360.195634] [ 5436] 1000 5436 673450 162809 622 0 0 beam.smp Jan 16 17:29:40 kvm-s63702 kernel: [537360.195636] [ 5474] 1000 5474 1113 33 7 0 0 sh Jan 16 17:29:40 kvm-s63702 kernel: [537360.195638] [ 5476] 1000 5476 1084 34 7 0 0 memsup Jan 16 17:29:40 kvm-s63702 kernel: [537360.195639] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 16 17:29:40 kvm-s63702 kernel: [537360.195641] [ 5479] 1000 5479 1867 23 9 0 0 inet_gethost Jan 16 17:29:40 kvm-s63702 kernel: [537360.195643] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 16 17:29:40 kvm-s63702 kernel: [537360.195645] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 16 17:29:40 kvm-s63702 kernel: [537360.195646] [ 5571] 1000 5571 428648 5411 112 0 0 memcached Jan 16 17:29:40 kvm-s63702 kernel: [537360.195648] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 16 17:29:40 kvm-s63702 kernel: [537360.195649] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 16 17:29:40 kvm-s63702 kernel: [537360.195651] [ 5786] 1000 5786 292661 15344 133 0 0 beam.smp Jan 16 17:29:40 kvm-s63702 kernel: [537360.195653] [ 5854] 1000 5854 1113 31 5 0 0 sh Jan 16 17:29:40 kvm-s63702 kernel: [537360.195654] [ 5856] 1000 5856 1084 34 7 0 0 memsup Jan 16 17:29:40 kvm-s63702 kernel: [537360.195656] [ 5857] 1000 5857 1084 21 8 0 0 cpu_sup Jan 16 17:29:40 kvm-s63702 kernel: [537360.195658] [ 5863] 1000 5863 1331 110 8 0 0 godu Jan 16 17:29:40 kvm-s63702 kernel: [537360.195659] [ 5867] 1000 5867 1113 34 7 0 0 sh Jan 16 17:29:40 kvm-s63702 kernel: [537360.195661] [ 5868] 1000 5868 1331 109 8 0 0 godu Jan 16 17:29:40 kvm-s63702 kernel: [537360.195664] [ 5878] 1000 5878 2156 73 9 0 0 sigar_port Jan 16 17:29:40 kvm-s63702 kernel: [537360.195666] [ 5879] 1000 5879 1462 165 8 0 0 goport Jan 16 17:29:40 kvm-s63702 kernel: [537360.195668] [ 5884] 1000 5884 74324 2073 32 0 0 goxdcr Jan 16 17:29:40 kvm-s63702 kernel: [537360.195670] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 16 17:29:40 kvm-s63702 kernel: [537360.195672] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 16 17:29:40 kvm-s63702 kernel: [537360.195674] [ 6680] 1000 6680 1462 164 8 0 0 goport Jan 16 17:29:40 kvm-s63702 kernel: [537360.195675] [ 6685] 1000 6685 13768383 7355004 17924 0 0 indexer Jan 16 17:29:40 kvm-s63702 kernel: [537360.195677] Out of memory: Kill process 6685 (indexer) score 954 or sacrifice child Jan 16 17:29:40 kvm-s63702 kernel: [537360.195773] Killed process 6685 (indexer) total-vm:55073532kB, anon-rss:29419660kB, file-rss:356kB Jan 17 00:00:02 kvm-s63702 kernel: [560782.745186] Process accounting resumed Jan 17 04:13:00 kvm-s63702 kernel: [575960.816251] goport invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816261] goport cpuset=/ mems_allowed=0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816307] CPU: 5 PID: 14030 Comm: goport Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 17 04:13:00 kvm-s63702 kernel: [575960.816309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816312] 0000000000000000 ffff88079b363858 ffffffff817aed00 0000000000000000 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816315] ffff8806231c09d0 ffff88079b3638f8 ffffffff817a9c3f 0000000000000000 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816318] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816321] Call Trace: Jan 17 04:13:00 kvm-s63702 kernel: [575960.816390] [] dump_stack+0x45/0x57 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816394] [] dump_header+0x7f/0x1f1 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816419] [] oom_kill_process+0x22b/0x390 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816423] [] out_of_memory+0x4d2/0x520 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816427] [] __alloc_pages_nodemask+0x940/0xa60 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816447] [] alloc_pages_current+0x91/0x100 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816451] [] __page_cache_alloc+0xa7/0xc0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816453] [] filemap_fault+0x1a8/0x430 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816457] [] __do_fault+0x39/0x90 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816460] [] do_read_fault.isra.55+0x1d7/0x2f0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816478] [] ? hrtimer_cancel+0x22/0x30 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816481] [] handle_mm_fault+0x7e5/0x1100 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816496] [] __do_page_fault+0x1c4/0x5b0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816500] [] ? hrtimer_get_res+0x50/0x50 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816510] [] ? do_futex+0xde/0x5d0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816518] [] ? pick_next_task_fair+0x61b/0x880 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816531] [] ? __switch_to+0x15f/0x570 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816535] [] trace_do_page_fault+0x45/0x100 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816540] [] do_async_page_fault+0x1e/0xd0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816571] [] async_page_fault+0x28/0x30 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816574] Mem-Info: Jan 17 04:13:00 kvm-s63702 kernel: [575960.816579] Node 0 DMA per-cpu: Jan 17 04:13:00 kvm-s63702 kernel: [575960.816585] CPU 0: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816586] CPU 1: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816588] CPU 2: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816589] CPU 3: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816604] CPU 4: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816605] CPU 5: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816607] CPU 6: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816608] CPU 7: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816609] CPU 8: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816611] CPU 9: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816612] CPU 10: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816613] CPU 11: hi: 0, btch: 1 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816615] Node 0 DMA32 per-cpu: Jan 17 04:13:00 kvm-s63702 kernel: [575960.816617] CPU 0: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816618] CPU 1: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816620] CPU 2: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816621] CPU 3: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816623] CPU 4: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816624] CPU 5: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816625] CPU 6: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816627] CPU 7: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816628] CPU 8: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816630] CPU 9: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816636] CPU 10: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816638] CPU 11: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816639] Node 0 Normal per-cpu: Jan 17 04:13:00 kvm-s63702 kernel: [575960.816640] CPU 0: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816642] CPU 1: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816643] CPU 2: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816644] CPU 3: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816645] CPU 4: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816647] CPU 5: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816648] CPU 6: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816650] CPU 7: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816651] CPU 8: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816652] CPU 9: hi: 186, btch: 31 usd: 64 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816654] CPU 10: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816655] CPU 11: hi: 186, btch: 31 usd: 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816660] active_anon:7586933 inactive_anon:73 isolated_anon:0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816660] active_file:83 inactive_file:0 isolated_file:32 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816660] unevictable:1199 dirty:0 writeback:0 unstable:0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816660] free:47952 slab_reclaimable:8558 slab_unreclaimable:8824 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816660] mapped:800 shmem:103 pagetables:22435 bounce:0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816660] free_cma:0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816664] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 17 04:13:00 kvm-s63702 kernel: [575960.816675] lowmem_reserve[]: 0 2978 30143 30143 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816679] Node 0 DMA32 free:115200kB min:6668kB low:8332kB high:10000kB active_anon:2908760kB inactive_anon:32kB active_file:0kB inactive_file:0kB unevictable:848kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:848kB dirty:0kB writeback:0kB mapped:764kB shmem:32kB slab_reclaimable:2724kB slab_unreclaimable:3044kB kernel_stack:3088kB pagetables:7628kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 17 04:13:00 kvm-s63702 kernel: [575960.816685] lowmem_reserve[]: 0 0 27165 27165 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816688] Node 0 Normal free:60716kB min:60876kB low:76092kB high:91312kB active_anon:27438972kB inactive_anon:260kB active_file:332kB inactive_file:0kB unevictable:3948kB isolated(anon):0kB isolated(file):128kB present:28311552kB managed:27817664kB mlocked:3948kB dirty:0kB writeback:0kB mapped:2436kB shmem:380kB slab_reclaimable:31508kB slab_unreclaimable:32236kB kernel_stack:23408kB pagetables:82112kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2564 all_unreclaimable? yes Jan 17 04:13:00 kvm-s63702 kernel: [575960.816693] lowmem_reserve[]: 0 0 0 0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816696] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 17 04:13:00 kvm-s63702 kernel: [575960.816712] Node 0 DMA32: 208*4kB (UEM) 124*8kB (UEM) 168*16kB (UEM) 127*32kB (UEM) 78*64kB (UEM) 40*128kB (UE) 9*256kB (UE) 4*512kB (U) 4*1024kB (UE) 7*2048kB (UEM) 18*4096kB (MR) = 115200kB Jan 17 04:13:00 kvm-s63702 kernel: [575960.816725] Node 0 Normal: 621*4kB (UEM) 294*8kB (UEM) 419*16kB (UEM) 348*32kB (UEM) 220*64kB (UE) 110*128kB (UEM) 19*256kB (UEM) 2*512kB (EM) 0*1024kB 0*2048kB 1*4096kB (R) = 60820kB Jan 17 04:13:00 kvm-s63702 kernel: [575960.816738] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 17 04:13:00 kvm-s63702 kernel: [575960.816739] 1096 total pagecache pages Jan 17 04:13:00 kvm-s63702 kernel: [575960.816741] 0 pages in swap cache Jan 17 04:13:00 kvm-s63702 kernel: [575960.816743] Swap cache stats: add 0, delete 0, find 0/0 Jan 17 04:13:00 kvm-s63702 kernel: [575960.816744] Free swap = 0kB Jan 17 04:13:00 kvm-s63702 kernel: [575960.816745] Total swap = 0kB Jan 17 04:13:00 kvm-s63702 kernel: [575960.816746] 7864218 pages RAM Jan 17 04:13:00 kvm-s63702 kernel: [575960.816748] 0 pages HighMem/MovableOnly Jan 17 04:13:00 kvm-s63702 kernel: [575960.816749] 143432 pages reserved Jan 17 04:13:00 kvm-s63702 kernel: [575960.816750] 0 pages cma reserved Jan 17 04:13:00 kvm-s63702 kernel: [575960.816751] 0 pages hwpoisoned Jan 17 04:13:00 kvm-s63702 kernel: [575960.816752] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 17 04:13:00 kvm-s63702 kernel: [575960.816760] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 17 04:13:00 kvm-s63702 kernel: [575960.816763] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 17 04:13:00 kvm-s63702 kernel: [575960.816771] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 17 04:13:00 kvm-s63702 kernel: [575960.816774] [ 724] 0 724 2559 864 8 0 0 dhclient Jan 17 04:13:00 kvm-s63702 kernel: [575960.816776] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 17 04:13:00 kvm-s63702 kernel: [575960.816779] [ 932] 101 932 65020 247 29 0 0 rsyslogd Jan 17 04:13:00 kvm-s63702 kernel: [575960.816782] [ 961] 102 961 9809 334 23 0 0 dbus-daemon Jan 17 04:13:00 kvm-s63702 kernel: [575960.816785] [ 964] 0 964 3956 41 12 0 0 getty Jan 17 04:13:00 kvm-s63702 kernel: [575960.816787] [ 967] 0 967 3956 40 13 0 0 getty Jan 17 04:13:00 kvm-s63702 kernel: [575960.816790] [ 972] 0 972 3956 40 13 0 0 getty Jan 17 04:13:00 kvm-s63702 kernel: [575960.816792] [ 973] 0 973 3956 39 13 0 0 getty Jan 17 04:13:00 kvm-s63702 kernel: [575960.816794] [ 975] 0 975 3956 39 13 0 0 getty Jan 17 04:13:00 kvm-s63702 kernel: [575960.816797] [ 995] 0 995 15343 301 33 0 -1000 sshd Jan 17 04:13:00 kvm-s63702 kernel: [575960.816799] [ 998] 0 998 1093 37 8 0 0 acpid Jan 17 04:13:00 kvm-s63702 kernel: [575960.816801] [ 1016] 0 1016 5915 238 18 0 0 cron Jan 17 04:13:00 kvm-s63702 kernel: [575960.816804] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 17 04:13:00 kvm-s63702 kernel: [575960.816806] [ 1046] 0 1046 10864 408 27 0 0 systemd-logind Jan 17 04:13:00 kvm-s63702 kernel: [575960.816809] [ 1052] 0 1052 4799 283 14 0 0 irqbalance Jan 17 04:13:00 kvm-s63702 kernel: [575960.816811] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 17 04:13:00 kvm-s63702 kernel: [575960.816815] [ 5353] 1000 5353 1875 90 8 0 0 epmd Jan 17 04:13:00 kvm-s63702 kernel: [575960.816817] [ 5396] 1000 5396 353550 8308 90 0 0 beam.smp Jan 17 04:13:00 kvm-s63702 kernel: [575960.816820] [ 5432] 1000 5432 2884 1183 10 0 0 gosecrets Jan 17 04:13:00 kvm-s63702 kernel: [575960.816823] [ 5436] 1000 5436 715867 182841 671 0 0 beam.smp Jan 17 04:13:00 kvm-s63702 kernel: [575960.816825] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 17 04:13:00 kvm-s63702 kernel: [575960.816827] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 17 04:13:00 kvm-s63702 kernel: [575960.816829] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 17 04:13:00 kvm-s63702 kernel: [575960.816831] [ 5479] 1000 5479 1867 46 9 0 0 inet_gethost Jan 17 04:13:00 kvm-s63702 kernel: [575960.816833] [ 5480] 1000 5480 1867 80 9 0 0 inet_gethost Jan 17 04:13:00 kvm-s63702 kernel: [575960.816835] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 17 04:13:00 kvm-s63702 kernel: [575960.816837] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 17 04:13:00 kvm-s63702 kernel: [575960.816840] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 17 04:13:00 kvm-s63702 kernel: [575960.816842] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 17 04:13:00 kvm-s63702 kernel: [575960.816844] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 17 04:13:00 kvm-s63702 kernel: [575960.816846] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 17 04:13:00 kvm-s63702 kernel: [575960.816849] [11156] 1000 11156 1462 151 8 0 0 goport Jan 17 04:13:00 kvm-s63702 kernel: [575960.816852] [11161] 1000 11161 15348998 7324436 20784 0 0 indexer Jan 17 04:13:00 kvm-s63702 kernel: [575960.816855] [14025] 1000 14025 1462 163 8 0 0 goport Jan 17 04:13:00 kvm-s63702 kernel: [575960.816857] [14029] 1000 14029 74324 1919 29 0 0 goxdcr Jan 17 04:13:00 kvm-s63702 kernel: [575960.816859] [14040] 1000 14040 281522 20493 146 0 0 beam.smp Jan 17 04:13:00 kvm-s63702 kernel: [575960.816861] [14076] 1000 14076 1113 361 7 0 0 sh Jan 17 04:13:00 kvm-s63702 kernel: [575960.816864] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 17 04:13:00 kvm-s63702 kernel: [575960.816866] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 17 04:13:00 kvm-s63702 kernel: [575960.816868] [14085] 1000 14085 1331 101 8 0 0 godu Jan 17 04:13:00 kvm-s63702 kernel: [575960.816871] [14086] 1000 14086 1113 320 7 0 0 sh Jan 17 04:13:00 kvm-s63702 kernel: [575960.816873] [14087] 1000 14087 1331 100 8 0 0 godu Jan 17 04:13:00 kvm-s63702 kernel: [575960.816875] [14098] 1000 14098 2156 404 10 0 0 sigar_port Jan 17 04:13:00 kvm-s63702 kernel: [575960.816884] [20999] 0 20999 4270 1201 14 0 0 atop Jan 17 04:13:00 kvm-s63702 kernel: [575960.816887] Out of memory: Kill process 11161 (indexer) score 951 or sacrifice child Jan 17 04:13:00 kvm-s63702 kernel: [575960.817027] Killed process 11161 (indexer) total-vm:61395992kB, anon-rss:29297744kB, file-rss:0kB Jan 17 07:08:09 kvm-s63702 kernel: [586469.794692] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794698] indexer cpuset=/ mems_allowed=0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794714] CPU: 4 PID: 23970 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 17 07:08:09 kvm-s63702 kernel: [586469.794716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794717] 0000000000000000 ffff8805edda79d8 ffffffff817aed00 0000000000000000 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794720] ffff88039aed75c0 ffff8805edda7a78 ffffffff817a9c3f 0000000000000000 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794721] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794724] Call Trace: Jan 17 07:08:09 kvm-s63702 kernel: [586469.794762] [] dump_stack+0x45/0x57 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794765] [] dump_header+0x7f/0x1f1 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794782] [] oom_kill_process+0x22b/0x390 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794784] [] out_of_memory+0x4d2/0x520 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794788] [] __alloc_pages_nodemask+0x940/0xa60 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794798] [] alloc_pages_vma+0x97/0x150 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794803] [] handle_mm_fault+0xd13/0x1100 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794816] [] __do_page_fault+0x1c4/0x5b0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794825] [] ? __enqueue_entity+0x78/0x80 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794828] [] ? set_next_entity+0x9d/0xb0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794830] [] ? pick_next_task_fair+0x61b/0x880 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794840] [] ? __switch_to+0x15f/0x570 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794842] [] ? sched_clock_cpu+0x98/0xc0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794844] [] trace_do_page_fault+0x45/0x100 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794847] [] do_async_page_fault+0x1e/0xd0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794856] [] async_page_fault+0x28/0x30 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794857] Mem-Info: Jan 17 07:08:09 kvm-s63702 kernel: [586469.794861] Node 0 DMA per-cpu: Jan 17 07:08:09 kvm-s63702 kernel: [586469.794865] CPU 0: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794866] CPU 1: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794867] CPU 2: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794868] CPU 3: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794869] CPU 4: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794870] CPU 5: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794871] CPU 6: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794872] CPU 7: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794873] CPU 8: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794874] CPU 9: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794875] CPU 10: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794876] CPU 11: hi: 0, btch: 1 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794877] Node 0 DMA32 per-cpu: Jan 17 07:08:09 kvm-s63702 kernel: [586469.794878] CPU 0: hi: 186, btch: 31 usd: 25 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794879] CPU 1: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794880] CPU 2: hi: 186, btch: 31 usd: 30 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794881] CPU 3: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794882] CPU 4: hi: 186, btch: 31 usd: 28 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794883] CPU 5: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794884] CPU 6: hi: 186, btch: 31 usd: 9 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794885] CPU 7: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794886] CPU 8: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794887] CPU 9: hi: 186, btch: 31 usd: 28 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794887] CPU 10: hi: 186, btch: 31 usd: 30 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794888] CPU 11: hi: 186, btch: 31 usd: 29 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794889] Node 0 Normal per-cpu: Jan 17 07:08:09 kvm-s63702 kernel: [586469.794890] CPU 0: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794891] CPU 1: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794892] CPU 2: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794893] CPU 3: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794894] CPU 4: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794895] CPU 5: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794896] CPU 6: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794897] CPU 7: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794898] CPU 8: hi: 186, btch: 31 usd: 30 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794899] CPU 9: hi: 186, btch: 31 usd: 27 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794900] CPU 10: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794903] CPU 11: hi: 186, btch: 31 usd: 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794906] active_anon:7570523 inactive_anon:72 isolated_anon:0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794906] active_file:122 inactive_file:228 isolated_file:64 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794906] unevictable:1199 dirty:0 writeback:195 unstable:0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794906] free:48016 slab_reclaimable:6901 slab_unreclaimable:11401 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794906] mapped:922 shmem:112 pagetables:26347 bounce:0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794906] free_cma:0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794909] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 17 07:08:09 kvm-s63702 kernel: [586469.794915] lowmem_reserve[]: 0 2978 30143 30143 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794918] Node 0 DMA32 free:115236kB min:6668kB low:8332kB high:10000kB active_anon:2900976kB inactive_anon:32kB active_file:0kB inactive_file:4kB unevictable:848kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:848kB dirty:0kB writeback:0kB mapped:768kB shmem:36kB slab_reclaimable:2152kB slab_unreclaimable:3748kB kernel_stack:7600kB pagetables:10712kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? yes Jan 17 07:08:09 kvm-s63702 kernel: [586469.794921] lowmem_reserve[]: 0 0 27165 27165 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794923] Node 0 Normal free:60936kB min:60876kB low:76092kB high:91312kB active_anon:27381116kB inactive_anon:256kB active_file:488kB inactive_file:908kB unevictable:3948kB isolated(anon):0kB isolated(file):256kB present:28311552kB managed:27817664kB mlocked:3948kB dirty:0kB writeback:780kB mapped:2920kB shmem:412kB slab_reclaimable:25452kB slab_unreclaimable:41840kB kernel_stack:63136kB pagetables:94676kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9788 all_unreclaimable? yes Jan 17 07:08:09 kvm-s63702 kernel: [586469.794926] lowmem_reserve[]: 0 0 0 0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794928] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 17 07:08:09 kvm-s63702 kernel: [586469.794939] Node 0 DMA32: 144*4kB (UEM) 83*8kB (UEM) 122*16kB (UEM) 94*32kB (UE) 64*64kB (UEM) 41*128kB (UE) 27*256kB (UEM) 4*512kB (UE) 6*1024kB (EM) 7*2048kB (UEM) 17*4096kB (MR) = 114616kB Jan 17 07:08:09 kvm-s63702 kernel: [586469.794947] Node 0 Normal: 392*4kB (UEM) 271*8kB (UEM) 340*16kB (UEM) 245*32kB (UEM) 179*64kB (EM) 112*128kB (UEM) 51*256kB (UEM) 2*512kB (UE) 0*1024kB 0*2048kB 1*4096kB (R) = 60984kB Jan 17 07:08:09 kvm-s63702 kernel: [586469.794956] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 17 07:08:09 kvm-s63702 kernel: [586469.794957] 1493 total pagecache pages Jan 17 07:08:09 kvm-s63702 kernel: [586469.794959] 0 pages in swap cache Jan 17 07:08:09 kvm-s63702 kernel: [586469.794960] Swap cache stats: add 0, delete 0, find 0/0 Jan 17 07:08:09 kvm-s63702 kernel: [586469.794961] Free swap = 0kB Jan 17 07:08:09 kvm-s63702 kernel: [586469.794961] Total swap = 0kB Jan 17 07:08:09 kvm-s63702 kernel: [586469.794962] 7864218 pages RAM Jan 17 07:08:09 kvm-s63702 kernel: [586469.794963] 0 pages HighMem/MovableOnly Jan 17 07:08:09 kvm-s63702 kernel: [586469.794964] 143432 pages reserved Jan 17 07:08:09 kvm-s63702 kernel: [586469.794964] 0 pages cma reserved Jan 17 07:08:09 kvm-s63702 kernel: [586469.794965] 0 pages hwpoisoned Jan 17 07:08:09 kvm-s63702 kernel: [586469.794966] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 17 07:08:09 kvm-s63702 kernel: [586469.794984] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 17 07:08:09 kvm-s63702 kernel: [586469.794987] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 17 07:08:09 kvm-s63702 kernel: [586469.794990] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 17 07:08:09 kvm-s63702 kernel: [586469.794993] [ 724] 0 724 2559 833 8 0 0 dhclient Jan 17 07:08:09 kvm-s63702 kernel: [586469.794995] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 17 07:08:09 kvm-s63702 kernel: [586469.794997] [ 932] 101 932 65020 247 29 0 0 rsyslogd Jan 17 07:08:09 kvm-s63702 kernel: [586469.794999] [ 961] 102 961 9809 339 23 0 0 dbus-daemon Jan 17 07:08:09 kvm-s63702 kernel: [586469.795001] [ 964] 0 964 3956 41 12 0 0 getty Jan 17 07:08:09 kvm-s63702 kernel: [586469.795002] [ 967] 0 967 3956 40 13 0 0 getty Jan 17 07:08:09 kvm-s63702 kernel: [586469.795004] [ 972] 0 972 3956 40 13 0 0 getty Jan 17 07:08:09 kvm-s63702 kernel: [586469.795006] [ 973] 0 973 3956 39 13 0 0 getty Jan 17 07:08:09 kvm-s63702 kernel: [586469.795010] [ 975] 0 975 3956 39 13 0 0 getty Jan 17 07:08:09 kvm-s63702 kernel: [586469.795013] [ 995] 0 995 15343 301 33 0 -1000 sshd Jan 17 07:08:09 kvm-s63702 kernel: [586469.795015] [ 998] 0 998 1093 37 8 0 0 acpid Jan 17 07:08:09 kvm-s63702 kernel: [586469.795018] [ 1016] 0 1016 5915 238 18 0 0 cron Jan 17 07:08:09 kvm-s63702 kernel: [586469.795020] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 17 07:08:09 kvm-s63702 kernel: [586469.795022] [ 1046] 0 1046 10864 408 27 0 0 systemd-logind Jan 17 07:08:09 kvm-s63702 kernel: [586469.795024] [ 1052] 0 1052 4799 288 14 0 0 irqbalance Jan 17 07:08:09 kvm-s63702 kernel: [586469.795030] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 17 07:08:09 kvm-s63702 kernel: [586469.795035] [ 5353] 1000 5353 1875 78 8 0 0 epmd Jan 17 07:08:09 kvm-s63702 kernel: [586469.795037] [ 5396] 1000 5396 351281 7964 87 0 0 beam.smp Jan 17 07:08:09 kvm-s63702 kernel: [586469.795039] [ 5432] 1000 5432 2884 1274 10 0 0 gosecrets Jan 17 07:08:09 kvm-s63702 kernel: [586469.795041] [ 5436] 1000 5436 747715 215095 752 0 0 beam.smp Jan 17 07:08:09 kvm-s63702 kernel: [586469.795043] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 17 07:08:09 kvm-s63702 kernel: [586469.795045] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 17 07:08:09 kvm-s63702 kernel: [586469.795047] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 17 07:08:09 kvm-s63702 kernel: [586469.795049] [ 5479] 1000 5479 1867 35 9 0 0 inet_gethost Jan 17 07:08:09 kvm-s63702 kernel: [586469.795051] [ 5480] 1000 5480 1867 69 9 0 0 inet_gethost Jan 17 07:08:09 kvm-s63702 kernel: [586469.795052] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 17 07:08:09 kvm-s63702 kernel: [586469.795054] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 17 07:08:09 kvm-s63702 kernel: [586469.795056] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 17 07:08:09 kvm-s63702 kernel: [586469.795058] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 17 07:08:09 kvm-s63702 kernel: [586469.795060] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 17 07:08:09 kvm-s63702 kernel: [586469.795062] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 17 07:08:09 kvm-s63702 kernel: [586469.795066] [14025] 1000 14025 1462 163 8 0 0 goport Jan 17 07:08:09 kvm-s63702 kernel: [586469.795068] [14029] 1000 14029 74324 1869 29 0 0 goxdcr Jan 17 07:08:09 kvm-s63702 kernel: [586469.795070] [14040] 1000 14040 278900 18001 140 0 0 beam.smp Jan 17 07:08:09 kvm-s63702 kernel: [586469.795071] [14076] 1000 14076 1113 361 7 0 0 sh Jan 17 07:08:09 kvm-s63702 kernel: [586469.795073] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 17 07:08:09 kvm-s63702 kernel: [586469.795075] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 17 07:08:09 kvm-s63702 kernel: [586469.795077] [14085] 1000 14085 1331 101 8 0 0 godu Jan 17 07:08:09 kvm-s63702 kernel: [586469.795079] [14086] 1000 14086 1113 320 7 0 0 sh Jan 17 07:08:09 kvm-s63702 kernel: [586469.795081] [14087] 1000 14087 1331 100 8 0 0 godu Jan 17 07:08:09 kvm-s63702 kernel: [586469.795083] [14098] 1000 14098 2156 396 10 0 0 sigar_port Jan 17 07:08:09 kvm-s63702 kernel: [586469.795086] [20999] 0 20999 4270 1201 14 0 0 atop Jan 17 07:08:09 kvm-s63702 kernel: [586469.795089] [21725] 1000 21725 1526 154 8 0 0 goport Jan 17 07:08:09 kvm-s63702 kernel: [586469.795091] [21730] 1000 21730 21941747 7194957 24624 0 0 indexer Jan 17 07:08:09 kvm-s63702 kernel: [586469.795093] Out of memory: Kill process 21730 (indexer) score 935 or sacrifice child Jan 17 07:08:09 kvm-s63702 kernel: [586469.802263] Killed process 21730 (indexer) total-vm:87766988kB, anon-rss:28778160kB, file-rss:1668kB Jan 17 13:28:48 kvm-s63702 kernel: [609308.520627] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520634] indexer cpuset=/ mems_allowed=0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520655] CPU: 1 PID: 27148 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 17 13:28:48 kvm-s63702 kernel: [609308.520657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520659] 0000000000000000 ffff8807989f39d8 ffffffff817aed00 0000000000000000 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520676] ffff88079cf844b0 ffff8807989f3a78 ffffffff817a9c3f 0000000000000000 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520679] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520686] Call Trace: Jan 17 13:28:48 kvm-s63702 kernel: [609308.520738] [] dump_stack+0x45/0x57 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520742] [] dump_header+0x7f/0x1f1 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520766] [] oom_kill_process+0x22b/0x390 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520770] [] out_of_memory+0x4d2/0x520 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520775] [] __alloc_pages_nodemask+0x940/0xa60 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520789] [] alloc_pages_vma+0x97/0x150 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520796] [] handle_mm_fault+0xd13/0x1100 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520804] [] ? vmacache_update+0x40/0x40 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520820] [] __do_page_fault+0x1c4/0x5b0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520833] [] ? __enqueue_entity+0x78/0x80 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520841] [] ? set_next_entity+0x9d/0xb0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520844] [] ? pick_next_task_fair+0x61b/0x880 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520858] [] ? __switch_to+0x15f/0x570 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520861] [] ? sched_clock_cpu+0x98/0xc0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520864] [] trace_do_page_fault+0x45/0x100 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520868] [] do_async_page_fault+0x1e/0xd0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520881] [] async_page_fault+0x28/0x30 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520883] Mem-Info: Jan 17 13:28:48 kvm-s63702 kernel: [609308.520888] Node 0 DMA per-cpu: Jan 17 13:28:48 kvm-s63702 kernel: [609308.520893] CPU 0: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520895] CPU 1: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520897] CPU 2: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520898] CPU 3: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520900] CPU 4: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520901] CPU 5: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520903] CPU 6: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520904] CPU 7: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520906] CPU 8: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520907] CPU 9: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520908] CPU 10: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520910] CPU 11: hi: 0, btch: 1 usd: 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520911] Node 0 DMA32 per-cpu: Jan 17 13:28:48 kvm-s63702 kernel: [609308.520913] CPU 0: hi: 186, btch: 31 usd: 56 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520914] CPU 1: hi: 186, btch: 31 usd: 30 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520916] CPU 2: hi: 186, btch: 31 usd: 16 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520917] CPU 3: hi: 186, btch: 31 usd: 56 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520919] CPU 4: hi: 186, btch: 31 usd: 28 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520920] CPU 5: hi: 186, btch: 31 usd: 51 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520921] CPU 6: hi: 186, btch: 31 usd: 11 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520923] CPU 7: hi: 186, btch: 31 usd: 33 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520924] CPU 8: hi: 186, btch: 31 usd: 37 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520926] CPU 9: hi: 186, btch: 31 usd: 124 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520927] CPU 10: hi: 186, btch: 31 usd: 38 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520929] CPU 11: hi: 186, btch: 31 usd: 61 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520930] Node 0 Normal per-cpu: Jan 17 13:28:48 kvm-s63702 kernel: [609308.520932] CPU 0: hi: 186, btch: 31 usd: 41 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520933] CPU 1: hi: 186, btch: 31 usd: 25 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520934] CPU 2: hi: 186, btch: 31 usd: 17 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520937] CPU 3: hi: 186, btch: 31 usd: 30 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520939] CPU 4: hi: 186, btch: 31 usd: 52 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520940] CPU 5: hi: 186, btch: 31 usd: 36 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520942] CPU 6: hi: 186, btch: 31 usd: 7 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520943] CPU 7: hi: 186, btch: 31 usd: 56 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520945] CPU 8: hi: 186, btch: 31 usd: 32 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520946] CPU 9: hi: 186, btch: 31 usd: 40 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520947] CPU 10: hi: 186, btch: 31 usd: 5 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520949] CPU 11: hi: 186, btch: 31 usd: 90 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520953] active_anon:7586216 inactive_anon:72 isolated_anon:0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520953] active_file:172 inactive_file:0 isolated_file:0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520953] unevictable:1232 dirty:48 writeback:0 unstable:0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520953] free:47937 slab_reclaimable:9550 slab_unreclaimable:8884 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520953] mapped:818 shmem:126 pagetables:21786 bounce:0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520953] free_cma:0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.520957] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 17 13:28:48 kvm-s63702 kernel: [609308.520998] lowmem_reserve[]: 0 2978 30143 30143 Jan 17 13:28:48 kvm-s63702 kernel: [609308.521001] Node 0 DMA32 free:115116kB min:6668kB low:8332kB high:10000kB active_anon:2903288kB inactive_anon:32kB active_file:16kB inactive_file:0kB unevictable:848kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:848kB dirty:0kB writeback:0kB mapped:748kB shmem:44kB slab_reclaimable:3216kB slab_unreclaimable:3296kB kernel_stack:2560kB pagetables:8804kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7892 all_unreclaimable? yes Jan 17 13:28:48 kvm-s63702 kernel: [609308.521007] lowmem_reserve[]: 0 0 27165 27165 Jan 17 13:28:48 kvm-s63702 kernel: [609308.521010] Node 0 Normal free:60740kB min:60876kB low:76092kB high:91312kB active_anon:27441576kB inactive_anon:256kB active_file:672kB inactive_file:0kB unevictable:4080kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:4080kB dirty:192kB writeback:0kB mapped:2524kB shmem:460kB slab_reclaimable:34984kB slab_unreclaimable:32224kB kernel_stack:21904kB pagetables:78340kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:194976 all_unreclaimable? yes Jan 17 13:28:48 kvm-s63702 kernel: [609308.521016] lowmem_reserve[]: 0 0 0 0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.521019] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 17 13:28:48 kvm-s63702 kernel: [609308.521034] Node 0 DMA32: 90*4kB (UE) 73*8kB (EM) 161*16kB (UEM) 132*32kB (UEM) 97*64kB (UE) 51*128kB (UE) 22*256kB (UE) 7*512kB (UEM) 3*1024kB (UM) 6*2048kB (UM) 17*4096kB (MR) = 114688kB Jan 17 13:28:48 kvm-s63702 kernel: [609308.521046] Node 0 Normal: 323*4kB (UEM) 210*8kB (UE) 459*16kB (UEM) 367*32kB (UEM) 295*64kB (UEM) 99*128kB (UE) 7*256kB (E) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 59500kB Jan 17 13:28:48 kvm-s63702 kernel: [609308.521058] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 17 13:28:48 kvm-s63702 kernel: [609308.521060] 1013 total pagecache pages Jan 17 13:28:48 kvm-s63702 kernel: [609308.521062] 0 pages in swap cache Jan 17 13:28:48 kvm-s63702 kernel: [609308.521063] Swap cache stats: add 0, delete 0, find 0/0 Jan 17 13:28:48 kvm-s63702 kernel: [609308.521064] Free swap = 0kB Jan 17 13:28:48 kvm-s63702 kernel: [609308.521065] Total swap = 0kB Jan 17 13:28:48 kvm-s63702 kernel: [609308.521067] 7864218 pages RAM Jan 17 13:28:48 kvm-s63702 kernel: [609308.521068] 0 pages HighMem/MovableOnly Jan 17 13:28:48 kvm-s63702 kernel: [609308.521069] 143432 pages reserved Jan 17 13:28:48 kvm-s63702 kernel: [609308.521070] 0 pages cma reserved Jan 17 13:28:48 kvm-s63702 kernel: [609308.521071] 0 pages hwpoisoned Jan 17 13:28:48 kvm-s63702 kernel: [609308.521072] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 17 13:28:48 kvm-s63702 kernel: [609308.521088] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 17 13:28:48 kvm-s63702 kernel: [609308.521092] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 17 13:28:48 kvm-s63702 kernel: [609308.521096] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 17 13:28:48 kvm-s63702 kernel: [609308.521098] [ 724] 0 724 2559 833 8 0 0 dhclient Jan 17 13:28:48 kvm-s63702 kernel: [609308.521101] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 17 13:28:48 kvm-s63702 kernel: [609308.521104] [ 932] 101 932 65020 263 29 0 0 rsyslogd Jan 17 13:28:48 kvm-s63702 kernel: [609308.521107] [ 961] 102 961 9809 334 23 0 0 dbus-daemon Jan 17 13:28:48 kvm-s63702 kernel: [609308.521110] [ 964] 0 964 3956 41 12 0 0 getty Jan 17 13:28:48 kvm-s63702 kernel: [609308.521113] [ 967] 0 967 3956 40 13 0 0 getty Jan 17 13:28:48 kvm-s63702 kernel: [609308.521115] [ 972] 0 972 3956 40 13 0 0 getty Jan 17 13:28:48 kvm-s63702 kernel: [609308.521118] [ 973] 0 973 3956 39 13 0 0 getty Jan 17 13:28:48 kvm-s63702 kernel: [609308.521120] [ 975] 0 975 3956 39 13 0 0 getty Jan 17 13:28:48 kvm-s63702 kernel: [609308.521123] [ 995] 0 995 15343 301 33 0 -1000 sshd Jan 17 13:28:48 kvm-s63702 kernel: [609308.521126] [ 998] 0 998 1093 37 8 0 0 acpid Jan 17 13:28:48 kvm-s63702 kernel: [609308.521129] [ 1016] 0 1016 5915 238 18 0 0 cron Jan 17 13:28:48 kvm-s63702 kernel: [609308.521131] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 17 13:28:48 kvm-s63702 kernel: [609308.521133] [ 1046] 0 1046 10864 408 27 0 0 systemd-logind Jan 17 13:28:48 kvm-s63702 kernel: [609308.521136] [ 1052] 0 1052 4799 284 14 0 0 irqbalance Jan 17 13:28:48 kvm-s63702 kernel: [609308.521145] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 17 13:28:48 kvm-s63702 kernel: [609308.521151] [ 5353] 1000 5353 1875 66 8 0 0 epmd Jan 17 13:28:48 kvm-s63702 kernel: [609308.521154] [ 5396] 1000 5396 352306 8280 88 0 0 beam.smp Jan 17 13:28:48 kvm-s63702 kernel: [609308.521156] [ 5432] 1000 5432 2884 1228 10 0 0 gosecrets Jan 17 13:28:48 kvm-s63702 kernel: [609308.521159] [ 5436] 1000 5436 740997 201226 715 0 0 beam.smp Jan 17 13:28:48 kvm-s63702 kernel: [609308.521161] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 17 13:28:48 kvm-s63702 kernel: [609308.521164] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 17 13:28:48 kvm-s63702 kernel: [609308.521166] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 17 13:28:48 kvm-s63702 kernel: [609308.521169] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 17 13:28:48 kvm-s63702 kernel: [609308.521171] [ 5480] 1000 5480 1867 58 9 0 0 inet_gethost Jan 17 13:28:48 kvm-s63702 kernel: [609308.521174] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 17 13:28:48 kvm-s63702 kernel: [609308.521176] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 17 13:28:48 kvm-s63702 kernel: [609308.521179] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 17 13:28:48 kvm-s63702 kernel: [609308.521181] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 17 13:28:48 kvm-s63702 kernel: [609308.521184] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 17 13:28:48 kvm-s63702 kernel: [609308.521186] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 17 13:28:48 kvm-s63702 kernel: [609308.521190] [14025] 1000 14025 1462 167 8 0 0 goport Jan 17 13:28:48 kvm-s63702 kernel: [609308.521193] [14029] 1000 14029 74588 1812 29 0 0 goxdcr Jan 17 13:28:48 kvm-s63702 kernel: [609308.521195] [14040] 1000 14040 276045 17023 132 0 0 beam.smp Jan 17 13:28:48 kvm-s63702 kernel: [609308.521197] [14076] 1000 14076 1113 361 7 0 0 sh Jan 17 13:28:48 kvm-s63702 kernel: [609308.521200] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 17 13:28:48 kvm-s63702 kernel: [609308.521202] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 17 13:28:48 kvm-s63702 kernel: [609308.521204] [14085] 1000 14085 1331 101 8 0 0 godu Jan 17 13:28:48 kvm-s63702 kernel: [609308.521207] [14086] 1000 14086 1113 320 7 0 0 sh Jan 17 13:28:48 kvm-s63702 kernel: [609308.521209] [14087] 1000 14087 1331 100 8 0 0 godu Jan 17 13:28:48 kvm-s63702 kernel: [609308.521211] [14098] 1000 14098 2156 408 10 0 0 sigar_port Jan 17 13:28:48 kvm-s63702 kernel: [609308.521214] [20999] 0 20999 4303 1234 14 0 0 atop Jan 17 13:28:48 kvm-s63702 kernel: [609308.521217] [26319] 1000 26319 1526 154 8 0 0 goport Jan 17 13:28:48 kvm-s63702 kernel: [609308.521219] [26324] 1000 26324 14254335 7306097 20107 0 0 indexer Jan 17 13:28:48 kvm-s63702 kernel: [609308.521222] Out of memory: Kill process 27465 (indexer) score 948 or sacrifice child Jan 17 13:28:48 kvm-s63702 kernel: [609308.525848] Killed process 26324 (indexer) total-vm:57017340kB, anon-rss:29223764kB, file-rss:1352kB Jan 17 13:31:56 kvm-s63702 kernel: [609496.801145] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801150] indexer cpuset=/ mems_allowed=0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801166] CPU: 4 PID: 29135 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 17 13:31:57 kvm-s63702 kernel: [609496.801168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801169] 0000000000000000 ffff8805894879d8 ffffffff817aed00 0000000000000000 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801173] ffff88079adc13a0 ffff880589487a78 ffffffff817a9c3f 0000000000000000 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801175] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801178] Call Trace: Jan 17 13:31:57 kvm-s63702 kernel: [609496.801216] [] dump_stack+0x45/0x57 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801219] [] dump_header+0x7f/0x1f1 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801235] [] oom_kill_process+0x22b/0x390 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801238] [] out_of_memory+0x4d2/0x520 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801241] [] __alloc_pages_nodemask+0x940/0xa60 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801251] [] alloc_pages_vma+0x97/0x150 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801256] [] handle_mm_fault+0xd13/0x1100 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801268] [] ? x2apic_send_IPI_mask+0x13/0x20 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801276] [] ? try_to_wake_up+0x1f1/0x340 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801283] [] __do_page_fault+0x1c4/0x5b0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801290] [] ? __enqueue_entity+0x78/0x80 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801293] [] ? set_next_entity+0x9d/0xb0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801295] [] ? pick_next_task_fair+0x61b/0x880 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801305] [] ? __switch_to+0x15f/0x570 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801307] [] ? sched_clock_cpu+0x98/0xc0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801309] [] trace_do_page_fault+0x45/0x100 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801312] [] do_async_page_fault+0x1e/0xd0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801321] [] async_page_fault+0x28/0x30 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801323] Mem-Info: Jan 17 13:31:57 kvm-s63702 kernel: [609496.801326] Node 0 DMA per-cpu: Jan 17 13:31:57 kvm-s63702 kernel: [609496.801330] CPU 0: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801331] CPU 1: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801332] CPU 2: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801333] CPU 3: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801334] CPU 4: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801335] CPU 5: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801336] CPU 6: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801337] CPU 7: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801338] CPU 8: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801339] CPU 9: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801340] CPU 10: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801341] CPU 11: hi: 0, btch: 1 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801342] Node 0 DMA32 per-cpu: Jan 17 13:31:57 kvm-s63702 kernel: [609496.801343] CPU 0: hi: 186, btch: 31 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801344] CPU 1: hi: 186, btch: 31 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801345] CPU 2: hi: 186, btch: 31 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801346] CPU 3: hi: 186, btch: 31 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801347] CPU 4: hi: 186, btch: 31 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801347] CPU 5: hi: 186, btch: 31 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801349] CPU 6: hi: 186, btch: 31 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801349] CPU 7: hi: 186, btch: 31 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801350] CPU 8: hi: 186, btch: 31 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801351] CPU 9: hi: 186, btch: 31 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801352] CPU 10: hi: 186, btch: 31 usd: 99 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801353] CPU 11: hi: 186, btch: 31 usd: 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801354] Node 0 Normal per-cpu: Jan 17 13:31:57 kvm-s63702 kernel: [609496.801355] CPU 0: hi: 186, btch: 31 usd: 16 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801356] CPU 1: hi: 186, btch: 31 usd: 3 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801357] CPU 2: hi: 186, btch: 31 usd: 16 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801358] CPU 3: hi: 186, btch: 31 usd: 29 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801359] CPU 4: hi: 186, btch: 31 usd: 8 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801360] CPU 5: hi: 186, btch: 31 usd: 7 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801361] CPU 6: hi: 186, btch: 31 usd: 11 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801362] CPU 7: hi: 186, btch: 31 usd: 1 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801362] CPU 8: hi: 186, btch: 31 usd: 2 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801363] CPU 9: hi: 186, btch: 31 usd: 19 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801364] CPU 10: hi: 186, btch: 31 usd: 19 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801365] CPU 11: hi: 186, btch: 31 usd: 9 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801368] active_anon:7603516 inactive_anon:72 isolated_anon:0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801368] active_file:114 inactive_file:55 isolated_file:0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801368] unevictable:1232 dirty:0 writeback:4 unstable:0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801368] free:48526 slab_reclaimable:8989 slab_unreclaimable:8618 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801368] mapped:853 shmem:126 pagetables:17778 bounce:0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801368] free_cma:0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801371] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 17 13:31:57 kvm-s63702 kernel: [609496.801377] lowmem_reserve[]: 0 2978 30143 30143 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801380] Node 0 DMA32 free:115804kB min:6668kB low:8332kB high:10000kB active_anon:2913168kB inactive_anon:32kB active_file:0kB inactive_file:8kB unevictable:848kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:848kB dirty:0kB writeback:0kB mapped:764kB shmem:44kB slab_reclaimable:2940kB slab_unreclaimable:2812kB kernel_stack:2128kB pagetables:8700kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2600 all_unreclaimable? yes Jan 17 13:31:57 kvm-s63702 kernel: [609496.801383] lowmem_reserve[]: 0 0 27165 27165 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801385] Node 0 Normal free:62408kB min:60876kB low:76092kB high:91312kB active_anon:27500896kB inactive_anon:256kB active_file:456kB inactive_file:212kB unevictable:4080kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:4080kB dirty:0kB writeback:16kB mapped:2648kB shmem:460kB slab_reclaimable:33016kB slab_unreclaimable:31644kB kernel_stack:21184kB pagetables:62412kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:31440 all_unreclaimable? yes Jan 17 13:31:57 kvm-s63702 kernel: [609496.801392] lowmem_reserve[]: 0 0 0 0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801394] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 17 13:31:57 kvm-s63702 kernel: [609496.801404] Node 0 DMA32: 143*4kB (UEM) 90*8kB (UEM) 173*16kB (UEM) 165*32kB (UEM) 134*64kB (UEM) 67*128kB (UE) 27*256kB (UE) 7*512kB (UE) 3*1024kB (UM) 1*2048kB (M) 18*4096kB (UMR) = 115836kB Jan 17 13:31:57 kvm-s63702 kernel: [609496.801413] Node 0 Normal: 193*4kB (UE) 158*8kB (UEM) 387*16kB (UEM) 311*32kB (UEM) 274*64kB (UEM) 171*128kB (UEM) 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 61700kB Jan 17 13:31:57 kvm-s63702 kernel: [609496.801421] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 17 13:31:57 kvm-s63702 kernel: [609496.801422] 1156 total pagecache pages Jan 17 13:31:57 kvm-s63702 kernel: [609496.801423] 0 pages in swap cache Jan 17 13:31:57 kvm-s63702 kernel: [609496.801424] Swap cache stats: add 0, delete 0, find 0/0 Jan 17 13:31:57 kvm-s63702 kernel: [609496.801425] Free swap = 0kB Jan 17 13:31:57 kvm-s63702 kernel: [609496.801426] Total swap = 0kB Jan 17 13:31:57 kvm-s63702 kernel: [609496.801426] 7864218 pages RAM Jan 17 13:31:57 kvm-s63702 kernel: [609496.801427] 0 pages HighMem/MovableOnly Jan 17 13:31:57 kvm-s63702 kernel: [609496.801428] 143432 pages reserved Jan 17 13:31:57 kvm-s63702 kernel: [609496.801429] 0 pages cma reserved Jan 17 13:31:57 kvm-s63702 kernel: [609496.801429] 0 pages hwpoisoned Jan 17 13:31:57 kvm-s63702 kernel: [609496.801430] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 17 13:31:57 kvm-s63702 kernel: [609496.801442] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 17 13:31:57 kvm-s63702 kernel: [609496.801444] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 17 13:31:57 kvm-s63702 kernel: [609496.801447] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 17 13:31:57 kvm-s63702 kernel: [609496.801449] [ 724] 0 724 2559 833 8 0 0 dhclient Jan 17 13:31:57 kvm-s63702 kernel: [609496.801450] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 17 13:31:57 kvm-s63702 kernel: [609496.801452] [ 932] 101 932 65020 263 29 0 0 rsyslogd Jan 17 13:31:57 kvm-s63702 kernel: [609496.801454] [ 961] 102 961 9809 334 23 0 0 dbus-daemon Jan 17 13:31:57 kvm-s63702 kernel: [609496.801456] [ 964] 0 964 3956 41 12 0 0 getty Jan 17 13:31:57 kvm-s63702 kernel: [609496.801457] [ 967] 0 967 3956 40 13 0 0 getty Jan 17 13:31:57 kvm-s63702 kernel: [609496.801459] [ 972] 0 972 3956 40 13 0 0 getty Jan 17 13:31:57 kvm-s63702 kernel: [609496.801460] [ 973] 0 973 3956 39 13 0 0 getty Jan 17 13:31:57 kvm-s63702 kernel: [609496.801462] [ 975] 0 975 3956 39 13 0 0 getty Jan 17 13:31:57 kvm-s63702 kernel: [609496.801464] [ 995] 0 995 15343 301 33 0 -1000 sshd Jan 17 13:31:57 kvm-s63702 kernel: [609496.801465] [ 998] 0 998 1093 37 8 0 0 acpid Jan 17 13:31:57 kvm-s63702 kernel: [609496.801467] [ 1016] 0 1016 5915 228 18 0 0 cron Jan 17 13:31:57 kvm-s63702 kernel: [609496.801469] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 17 13:31:57 kvm-s63702 kernel: [609496.801471] [ 1046] 0 1046 10864 408 27 0 0 systemd-logind Jan 17 13:31:57 kvm-s63702 kernel: [609496.801472] [ 1052] 0 1052 4799 284 14 0 0 irqbalance Jan 17 13:31:57 kvm-s63702 kernel: [609496.801474] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 17 13:31:57 kvm-s63702 kernel: [609496.801477] [ 5353] 1000 5353 1875 54 8 0 0 epmd Jan 17 13:31:57 kvm-s63702 kernel: [609496.801479] [ 5396] 1000 5396 352160 8635 88 0 0 beam.smp Jan 17 13:31:57 kvm-s63702 kernel: [609496.801481] [ 5432] 1000 5432 2884 1292 10 0 0 gosecrets Jan 17 13:31:57 kvm-s63702 kernel: [609496.801482] [ 5436] 1000 5436 733334 197835 710 0 0 beam.smp Jan 17 13:31:57 kvm-s63702 kernel: [609496.801484] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 17 13:31:57 kvm-s63702 kernel: [609496.801486] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 17 13:31:57 kvm-s63702 kernel: [609496.801487] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 17 13:31:57 kvm-s63702 kernel: [609496.801492] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 17 13:31:57 kvm-s63702 kernel: [609496.801494] [ 5480] 1000 5480 1867 47 9 0 0 inet_gethost Jan 17 13:31:57 kvm-s63702 kernel: [609496.801496] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 17 13:31:57 kvm-s63702 kernel: [609496.801498] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 17 13:31:57 kvm-s63702 kernel: [609496.801500] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 17 13:31:57 kvm-s63702 kernel: [609496.801501] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 17 13:31:57 kvm-s63702 kernel: [609496.801503] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 17 13:31:57 kvm-s63702 kernel: [609496.801505] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 17 13:31:57 kvm-s63702 kernel: [609496.801507] [14025] 1000 14025 1462 167 8 0 0 goport Jan 17 13:31:57 kvm-s63702 kernel: [609496.801509] [14029] 1000 14029 74588 1995 29 0 0 goxdcr Jan 17 13:31:57 kvm-s63702 kernel: [609496.801510] [14040] 1000 14040 279120 18437 139 0 0 beam.smp Jan 17 13:31:57 kvm-s63702 kernel: [609496.801512] [14076] 1000 14076 1113 336 7 0 0 sh Jan 17 13:31:57 kvm-s63702 kernel: [609496.801513] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 17 13:31:57 kvm-s63702 kernel: [609496.801515] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 17 13:31:57 kvm-s63702 kernel: [609496.801516] [14085] 1000 14085 1331 101 8 0 0 godu Jan 17 13:31:57 kvm-s63702 kernel: [609496.801518] [14086] 1000 14086 1113 320 7 0 0 sh Jan 17 13:31:57 kvm-s63702 kernel: [609496.801520] [14087] 1000 14087 1331 100 8 0 0 godu Jan 17 13:31:57 kvm-s63702 kernel: [609496.801521] [14098] 1000 14098 2156 408 10 0 0 sigar_port Jan 17 13:31:57 kvm-s63702 kernel: [609496.801523] [20999] 0 20999 4303 1234 14 0 0 atop Jan 17 13:31:57 kvm-s63702 kernel: [609496.801525] [28316] 1000 28316 1526 152 8 0 0 goport Jan 17 13:31:57 kvm-s63702 kernel: [609496.801526] [28321] 1000 28321 13771322 7325804 16096 0 0 indexer Jan 17 13:31:57 kvm-s63702 kernel: [609496.801528] Out of memory: Kill process 28321 (indexer) score 950 or sacrifice child Jan 17 13:31:57 kvm-s63702 kernel: [609496.805340] Killed process 28321 (indexer) total-vm:55085288kB, anon-rss:29302120kB, file-rss:1096kB Jan 17 16:35:10 kvm-s63702 kernel: [620490.505964] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.505970] indexer cpuset=/ mems_allowed=0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.505987] CPU: 8 PID: 29786 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 17 16:35:10 kvm-s63702 kernel: [620490.505989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 17 16:35:10 kvm-s63702 kernel: [620490.505991] 0000000000000000 ffff88040054f9d8 ffffffff817aed00 0000000000000000 Jan 17 16:35:10 kvm-s63702 kernel: [620490.505993] ffff88061e8a44b0 ffff88040054fa78 ffffffff817a9c3f 0000000000000000 Jan 17 16:35:10 kvm-s63702 kernel: [620490.505995] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 17 16:35:10 kvm-s63702 kernel: [620490.505997] Call Trace: Jan 17 16:35:10 kvm-s63702 kernel: [620490.506035] [] dump_stack+0x45/0x57 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506038] [] dump_header+0x7f/0x1f1 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506055] [] oom_kill_process+0x22b/0x390 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506058] [] out_of_memory+0x4d2/0x520 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506061] [] __alloc_pages_nodemask+0x940/0xa60 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506071] [] alloc_pages_vma+0x97/0x150 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506077] [] handle_mm_fault+0xd13/0x1100 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506091] [] ? ttwu_do_activate.constprop.94+0x5d/0x70 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506093] [] ? try_to_wake_up+0x1f1/0x340 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506103] [] __do_page_fault+0x1c4/0x5b0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506107] [] ? __enqueue_entity+0x78/0x80 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506110] [] ? set_next_entity+0x9d/0xb0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506112] [] ? pick_next_task_fair+0x61b/0x880 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506122] [] ? __switch_to+0x15f/0x570 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506124] [] ? sched_clock_cpu+0x98/0xc0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506126] [] trace_do_page_fault+0x45/0x100 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506129] [] do_async_page_fault+0x1e/0xd0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506138] [] async_page_fault+0x28/0x30 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506140] Mem-Info: Jan 17 16:35:10 kvm-s63702 kernel: [620490.506144] Node 0 DMA per-cpu: Jan 17 16:35:10 kvm-s63702 kernel: [620490.506147] CPU 0: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506150] CPU 1: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506152] CPU 2: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506153] CPU 3: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506154] CPU 4: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506155] CPU 5: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506156] CPU 6: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506157] CPU 7: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506158] CPU 8: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506159] CPU 9: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506160] CPU 10: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506161] CPU 11: hi: 0, btch: 1 usd: 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506162] Node 0 DMA32 per-cpu: Jan 17 16:35:10 kvm-s63702 kernel: [620490.506163] CPU 0: hi: 186, btch: 31 usd: 18 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506164] CPU 1: hi: 186, btch: 31 usd: 31 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506165] CPU 2: hi: 186, btch: 31 usd: 45 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506166] CPU 3: hi: 186, btch: 31 usd: 166 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506167] CPU 4: hi: 186, btch: 31 usd: 48 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506168] CPU 5: hi: 186, btch: 31 usd: 16 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506169] CPU 6: hi: 186, btch: 31 usd: 38 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506170] CPU 7: hi: 186, btch: 31 usd: 38 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506171] CPU 8: hi: 186, btch: 31 usd: 32 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506172] CPU 9: hi: 186, btch: 31 usd: 51 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506172] CPU 10: hi: 186, btch: 31 usd: 16 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506173] CPU 11: hi: 186, btch: 31 usd: 15 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506174] Node 0 Normal per-cpu: Jan 17 16:35:10 kvm-s63702 kernel: [620490.506175] CPU 0: hi: 186, btch: 31 usd: 30 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506176] CPU 1: hi: 186, btch: 31 usd: 26 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506177] CPU 2: hi: 186, btch: 31 usd: 21 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506178] CPU 3: hi: 186, btch: 31 usd: 156 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506180] CPU 4: hi: 186, btch: 31 usd: 96 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506180] CPU 5: hi: 186, btch: 31 usd: 16 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506181] CPU 6: hi: 186, btch: 31 usd: 55 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506182] CPU 7: hi: 186, btch: 31 usd: 58 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506183] CPU 8: hi: 186, btch: 31 usd: 21 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506184] CPU 9: hi: 186, btch: 31 usd: 46 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506185] CPU 10: hi: 186, btch: 31 usd: 3 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506186] CPU 11: hi: 186, btch: 31 usd: 33 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506190] active_anon:7586860 inactive_anon:72 isolated_anon:0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506190] active_file:135 inactive_file:79 isolated_file:0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506190] unevictable:1199 dirty:39 writeback:0 unstable:0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506190] free:47887 slab_reclaimable:9664 slab_unreclaimable:9255 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506190] mapped:1019 shmem:133 pagetables:23111 bounce:0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506190] free_cma:0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506192] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 17 16:35:10 kvm-s63702 kernel: [620490.506200] lowmem_reserve[]: 0 2978 30143 30143 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506203] Node 0 DMA32 free:115100kB min:6668kB low:8332kB high:10000kB active_anon:2906456kB inactive_anon:32kB active_file:92kB inactive_file:100kB unevictable:848kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:848kB dirty:104kB writeback:0kB mapped:744kB shmem:48kB slab_reclaimable:3200kB slab_unreclaimable:3004kB kernel_stack:2480kB pagetables:8840kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9768 all_unreclaimable? yes Jan 17 16:35:10 kvm-s63702 kernel: [620490.506206] lowmem_reserve[]: 0 0 27165 27165 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506208] Node 0 Normal free:60556kB min:60876kB low:76092kB high:91312kB active_anon:27440984kB inactive_anon:256kB active_file:448kB inactive_file:216kB unevictable:3948kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:3948kB dirty:52kB writeback:0kB mapped:3332kB shmem:484kB slab_reclaimable:35456kB slab_unreclaimable:34000kB kernel_stack:20832kB pagetables:83604kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:85816 all_unreclaimable? yes Jan 17 16:35:10 kvm-s63702 kernel: [620490.506212] lowmem_reserve[]: 0 0 0 0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506213] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 17 16:35:10 kvm-s63702 kernel: [620490.506224] Node 0 DMA32: 58*4kB (UEM) 55*8kB (E) 139*16kB (UE) 128*32kB (UEM) 105*64kB (UEM) 54*128kB (UE) 16*256kB (UE) 8*512kB (UEM) 4*1024kB (U) 4*2048kB (M) 18*4096kB (UMR) = 114832kB Jan 17 16:35:10 kvm-s63702 kernel: [620490.506235] Node 0 Normal: 336*4kB (UE) 267*8kB (UEM) 517*16kB (UEM) 337*32kB (UEM) 261*64kB (UE) 98*128kB (UE) 15*256kB (U) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 59720kB Jan 17 16:35:10 kvm-s63702 kernel: [620490.506244] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 17 16:35:10 kvm-s63702 kernel: [620490.506245] 1284 total pagecache pages Jan 17 16:35:10 kvm-s63702 kernel: [620490.506246] 0 pages in swap cache Jan 17 16:35:10 kvm-s63702 kernel: [620490.506247] Swap cache stats: add 0, delete 0, find 0/0 Jan 17 16:35:10 kvm-s63702 kernel: [620490.506248] Free swap = 0kB Jan 17 16:35:10 kvm-s63702 kernel: [620490.506249] Total swap = 0kB Jan 17 16:35:10 kvm-s63702 kernel: [620490.506250] 7864218 pages RAM Jan 17 16:35:10 kvm-s63702 kernel: [620490.506250] 0 pages HighMem/MovableOnly Jan 17 16:35:10 kvm-s63702 kernel: [620490.506251] 143432 pages reserved Jan 17 16:35:10 kvm-s63702 kernel: [620490.506252] 0 pages cma reserved Jan 17 16:35:10 kvm-s63702 kernel: [620490.506253] 0 pages hwpoisoned Jan 17 16:35:10 kvm-s63702 kernel: [620490.506254] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 17 16:35:10 kvm-s63702 kernel: [620490.506263] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 17 16:35:10 kvm-s63702 kernel: [620490.506265] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 17 16:35:10 kvm-s63702 kernel: [620490.506267] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 17 16:35:10 kvm-s63702 kernel: [620490.506269] [ 724] 0 724 2559 848 8 0 0 dhclient Jan 17 16:35:10 kvm-s63702 kernel: [620490.506271] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 17 16:35:10 kvm-s63702 kernel: [620490.506273] [ 932] 101 932 65020 265 29 0 0 rsyslogd Jan 17 16:35:10 kvm-s63702 kernel: [620490.506275] [ 961] 102 961 9809 334 23 0 0 dbus-daemon Jan 17 16:35:10 kvm-s63702 kernel: [620490.506276] [ 964] 0 964 3956 41 12 0 0 getty Jan 17 16:35:10 kvm-s63702 kernel: [620490.506278] [ 967] 0 967 3956 40 13 0 0 getty Jan 17 16:35:10 kvm-s63702 kernel: [620490.506280] [ 972] 0 972 3956 40 13 0 0 getty Jan 17 16:35:10 kvm-s63702 kernel: [620490.506281] [ 973] 0 973 3956 39 13 0 0 getty Jan 17 16:35:10 kvm-s63702 kernel: [620490.506283] [ 975] 0 975 3956 39 13 0 0 getty Jan 17 16:35:10 kvm-s63702 kernel: [620490.506285] [ 995] 0 995 15343 301 33 0 -1000 sshd Jan 17 16:35:10 kvm-s63702 kernel: [620490.506287] [ 998] 0 998 1093 37 8 0 0 acpid Jan 17 16:35:10 kvm-s63702 kernel: [620490.506289] [ 1016] 0 1016 5915 238 18 0 0 cron Jan 17 16:35:10 kvm-s63702 kernel: [620490.506291] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 17 16:35:10 kvm-s63702 kernel: [620490.506292] [ 1046] 0 1046 10864 408 27 0 0 systemd-logind Jan 17 16:35:10 kvm-s63702 kernel: [620490.506294] [ 1052] 0 1052 4799 288 14 0 0 irqbalance Jan 17 16:35:10 kvm-s63702 kernel: [620490.506296] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 17 16:35:10 kvm-s63702 kernel: [620490.506299] [ 5353] 1000 5353 1875 42 8 0 0 epmd Jan 17 16:35:10 kvm-s63702 kernel: [620490.506301] [ 5396] 1000 5396 353294 8430 89 0 0 beam.smp Jan 17 16:35:10 kvm-s63702 kernel: [620490.506302] [ 5432] 1000 5432 2884 1252 10 0 0 gosecrets Jan 17 16:35:10 kvm-s63702 kernel: [620490.506304] [ 5436] 1000 5436 744122 206561 730 0 0 beam.smp Jan 17 16:35:10 kvm-s63702 kernel: [620490.506306] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 17 16:35:10 kvm-s63702 kernel: [620490.506307] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 17 16:35:10 kvm-s63702 kernel: [620490.506309] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 17 16:35:10 kvm-s63702 kernel: [620490.506311] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 17 16:35:10 kvm-s63702 kernel: [620490.506312] [ 5480] 1000 5480 1867 36 9 0 0 inet_gethost Jan 17 16:35:10 kvm-s63702 kernel: [620490.506314] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 17 16:35:10 kvm-s63702 kernel: [620490.506316] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 17 16:35:10 kvm-s63702 kernel: [620490.506318] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 17 16:35:10 kvm-s63702 kernel: [620490.506320] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 17 16:35:10 kvm-s63702 kernel: [620490.506321] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 17 16:35:10 kvm-s63702 kernel: [620490.506326] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 17 16:35:10 kvm-s63702 kernel: [620490.506328] [14025] 1000 14025 1462 167 8 0 0 goport Jan 17 16:35:10 kvm-s63702 kernel: [620490.506330] [14029] 1000 14029 74588 1821 29 0 0 goxdcr Jan 17 16:35:10 kvm-s63702 kernel: [620490.506332] [14040] 1000 14040 280357 17619 140 0 0 beam.smp Jan 17 16:35:10 kvm-s63702 kernel: [620490.506334] [14076] 1000 14076 1113 336 7 0 0 sh Jan 17 16:35:10 kvm-s63702 kernel: [620490.506335] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 17 16:35:10 kvm-s63702 kernel: [620490.506337] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 17 16:35:10 kvm-s63702 kernel: [620490.506338] [14085] 1000 14085 1331 101 8 0 0 godu Jan 17 16:35:10 kvm-s63702 kernel: [620490.506340] [14086] 1000 14086 1113 320 7 0 0 sh Jan 17 16:35:10 kvm-s63702 kernel: [620490.506342] [14087] 1000 14087 1331 100 8 0 0 godu Jan 17 16:35:10 kvm-s63702 kernel: [620490.506343] [14098] 1000 14098 2156 412 10 0 0 sigar_port Jan 17 16:35:10 kvm-s63702 kernel: [620490.506345] [20999] 0 20999 4270 1201 14 0 0 atop Jan 17 16:35:10 kvm-s63702 kernel: [620490.506347] [29388] 1000 29388 1462 157 8 0 0 goport Jan 17 16:35:10 kvm-s63702 kernel: [620490.506349] [29393] 1000 29393 16238239 7299591 21411 0 0 indexer Jan 17 16:35:10 kvm-s63702 kernel: [620490.506350] Out of memory: Kill process 30869 (indexer) score 948 or sacrifice child Jan 17 16:35:10 kvm-s63702 kernel: [620490.514217] Killed process 29393 (indexer) total-vm:64952956kB, anon-rss:29196816kB, file-rss:1812kB Jan 17 19:44:15 kvm-s63702 kernel: [631835.130732] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130740] indexer cpuset=/ mems_allowed=0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130756] CPU: 8 PID: 32421 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 17 19:44:15 kvm-s63702 kernel: [631835.130758] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130759] 0000000000000000 ffff8800bb21b9d8 ffffffff817aed00 0000000000000000 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130761] ffff880693b62740 ffff8800bb21ba78 ffffffff817a9c3f 0000000000000000 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130763] 0000000000000002 0000000000000001 ffffffff81c476c0 ffff8800bb21bac8 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130765] Call Trace: Jan 17 19:44:15 kvm-s63702 kernel: [631835.130807] [] dump_stack+0x45/0x57 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130810] [] dump_header+0x7f/0x1f1 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130827] [] oom_kill_process+0x22b/0x390 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130829] [] out_of_memory+0x4d2/0x520 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130833] [] __alloc_pages_nodemask+0x940/0xa60 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130842] [] alloc_pages_vma+0x97/0x150 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130848] [] handle_mm_fault+0xd13/0x1100 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130859] [] ? x2apic_send_IPI_mask+0x13/0x20 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130867] [] ? try_to_wake_up+0x1f1/0x340 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130874] [] __do_page_fault+0x1c4/0x5b0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130879] [] ? __enqueue_entity+0x78/0x80 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130881] [] ? set_next_entity+0x9d/0xb0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130884] [] ? pick_next_task_fair+0x61b/0x880 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130893] [] ? __switch_to+0x15f/0x570 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130896] [] ? sched_clock_cpu+0x98/0xc0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130897] [] trace_do_page_fault+0x45/0x100 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130901] [] do_async_page_fault+0x1e/0xd0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130910] [] async_page_fault+0x28/0x30 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130912] Mem-Info: Jan 17 19:44:15 kvm-s63702 kernel: [631835.130916] Node 0 DMA per-cpu: Jan 17 19:44:15 kvm-s63702 kernel: [631835.130919] CPU 0: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130921] CPU 1: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130922] CPU 2: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130923] CPU 3: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130923] CPU 4: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130925] CPU 5: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130925] CPU 6: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130927] CPU 7: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130928] CPU 8: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130928] CPU 9: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130930] CPU 10: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130931] CPU 11: hi: 0, btch: 1 usd: 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130931] Node 0 DMA32 per-cpu: Jan 17 19:44:15 kvm-s63702 kernel: [631835.130935] CPU 0: hi: 186, btch: 31 usd: 56 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130936] CPU 1: hi: 186, btch: 31 usd: 37 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130937] CPU 2: hi: 186, btch: 31 usd: 47 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130938] CPU 3: hi: 186, btch: 31 usd: 128 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130939] CPU 4: hi: 186, btch: 31 usd: 60 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130940] CPU 5: hi: 186, btch: 31 usd: 42 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130940] CPU 6: hi: 186, btch: 31 usd: 104 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130941] CPU 7: hi: 186, btch: 31 usd: 66 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130942] CPU 8: hi: 186, btch: 31 usd: 46 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130943] CPU 9: hi: 186, btch: 31 usd: 23 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130944] CPU 10: hi: 186, btch: 31 usd: 48 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130945] CPU 11: hi: 186, btch: 31 usd: 135 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130946] Node 0 Normal per-cpu: Jan 17 19:44:15 kvm-s63702 kernel: [631835.130947] CPU 0: hi: 186, btch: 31 usd: 52 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130948] CPU 1: hi: 186, btch: 31 usd: 28 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130949] CPU 2: hi: 186, btch: 31 usd: 6 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130949] CPU 3: hi: 186, btch: 31 usd: 105 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130950] CPU 4: hi: 186, btch: 31 usd: 14 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130951] CPU 5: hi: 186, btch: 31 usd: 52 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130952] CPU 6: hi: 186, btch: 31 usd: 7 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130953] CPU 7: hi: 186, btch: 31 usd: 177 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130954] CPU 8: hi: 186, btch: 31 usd: 22 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130955] CPU 9: hi: 186, btch: 31 usd: 9 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130956] CPU 10: hi: 186, btch: 31 usd: 11 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130957] CPU 11: hi: 186, btch: 31 usd: 114 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130960] active_anon:7588958 inactive_anon:72 isolated_anon:0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130960] active_file:666 inactive_file:644 isolated_file:0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130960] unevictable:1232 dirty:627 writeback:0 unstable:0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130960] free:47929 slab_reclaimable:6831 slab_unreclaimable:8783 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130960] mapped:1354 shmem:139 pagetables:22596 bounce:0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130960] free_cma:0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130962] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 17 19:44:15 kvm-s63702 kernel: [631835.130968] lowmem_reserve[]: 0 2978 30143 30143 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130971] Node 0 DMA32 free:115292kB min:6668kB low:8332kB high:10000kB active_anon:2905796kB inactive_anon:32kB active_file:192kB inactive_file:156kB unevictable:848kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:848kB dirty:172kB writeback:0kB mapped:876kB shmem:48kB slab_reclaimable:2208kB slab_unreclaimable:2756kB kernel_stack:2160kB pagetables:9076kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:73428 all_unreclaimable? yes Jan 17 19:44:15 kvm-s63702 kernel: [631835.130975] lowmem_reserve[]: 0 0 27165 27165 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130977] Node 0 Normal free:60532kB min:60876kB low:76092kB high:91312kB active_anon:27450036kB inactive_anon:256kB active_file:2472kB inactive_file:2420kB unevictable:4080kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:4080kB dirty:2336kB writeback:0kB mapped:4540kB shmem:508kB slab_reclaimable:25116kB slab_unreclaimable:32360kB kernel_stack:21328kB pagetables:81308kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:100652 all_unreclaimable? yes Jan 17 19:44:15 kvm-s63702 kernel: [631835.130980] lowmem_reserve[]: 0 0 0 0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.130982] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 17 19:44:15 kvm-s63702 kernel: [631835.130993] Node 0 DMA32: 109*4kB (UEM) 64*8kB (UEM) 108*16kB (EM) 99*32kB (E) 56*64kB (UEM) 52*128kB (UEM) 24*256kB (UEM) 11*512kB (UEM) 5*1024kB (UEM) 4*2048kB (UM) 18*4096kB (UMR) = 114900kB Jan 17 19:44:15 kvm-s63702 kernel: [631835.131002] Node 0 Normal: 500*4kB (UE) 328*8kB (UE) 363*16kB (UEM) 265*32kB (UEM) 173*64kB (UEM) 101*128kB (UEM) 42*256kB (EM) 4*512kB (E) 0*1024kB 0*2048kB 1*4096kB (R) = 59808kB Jan 17 19:44:15 kvm-s63702 kernel: [631835.131011] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 17 19:44:15 kvm-s63702 kernel: [631835.131012] 2347 total pagecache pages Jan 17 19:44:15 kvm-s63702 kernel: [631835.131014] 0 pages in swap cache Jan 17 19:44:15 kvm-s63702 kernel: [631835.131015] Swap cache stats: add 0, delete 0, find 0/0 Jan 17 19:44:15 kvm-s63702 kernel: [631835.131016] Free swap = 0kB Jan 17 19:44:15 kvm-s63702 kernel: [631835.131019] Total swap = 0kB Jan 17 19:44:15 kvm-s63702 kernel: [631835.131020] 7864218 pages RAM Jan 17 19:44:15 kvm-s63702 kernel: [631835.131021] 0 pages HighMem/MovableOnly Jan 17 19:44:15 kvm-s63702 kernel: [631835.131021] 143432 pages reserved Jan 17 19:44:15 kvm-s63702 kernel: [631835.131022] 0 pages cma reserved Jan 17 19:44:15 kvm-s63702 kernel: [631835.131023] 0 pages hwpoisoned Jan 17 19:44:15 kvm-s63702 kernel: [631835.131024] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 17 19:44:15 kvm-s63702 kernel: [631835.131033] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 17 19:44:15 kvm-s63702 kernel: [631835.131036] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 17 19:44:15 kvm-s63702 kernel: [631835.131038] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 17 19:44:15 kvm-s63702 kernel: [631835.131040] [ 724] 0 724 2559 848 8 0 0 dhclient Jan 17 19:44:15 kvm-s63702 kernel: [631835.131043] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 17 19:44:15 kvm-s63702 kernel: [631835.131045] [ 932] 101 932 65020 265 29 0 0 rsyslogd Jan 17 19:44:15 kvm-s63702 kernel: [631835.131047] [ 961] 102 961 9809 334 23 0 0 dbus-daemon Jan 17 19:44:15 kvm-s63702 kernel: [631835.131049] [ 964] 0 964 3956 41 12 0 0 getty Jan 17 19:44:15 kvm-s63702 kernel: [631835.131050] [ 967] 0 967 3956 40 13 0 0 getty Jan 17 19:44:15 kvm-s63702 kernel: [631835.131052] [ 972] 0 972 3956 40 13 0 0 getty Jan 17 19:44:15 kvm-s63702 kernel: [631835.131054] [ 973] 0 973 3956 39 13 0 0 getty Jan 17 19:44:15 kvm-s63702 kernel: [631835.131056] [ 975] 0 975 3956 39 13 0 0 getty Jan 17 19:44:15 kvm-s63702 kernel: [631835.131058] [ 995] 0 995 15343 301 33 0 -1000 sshd Jan 17 19:44:15 kvm-s63702 kernel: [631835.131060] [ 998] 0 998 1093 37 8 0 0 acpid Jan 17 19:44:15 kvm-s63702 kernel: [631835.131062] [ 1016] 0 1016 5915 228 18 0 0 cron Jan 17 19:44:15 kvm-s63702 kernel: [631835.131064] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 17 19:44:15 kvm-s63702 kernel: [631835.131066] [ 1046] 0 1046 10864 408 27 0 0 systemd-logind Jan 17 19:44:15 kvm-s63702 kernel: [631835.131068] [ 1052] 0 1052 4799 288 14 0 0 irqbalance Jan 17 19:44:15 kvm-s63702 kernel: [631835.131070] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 17 19:44:15 kvm-s63702 kernel: [631835.131072] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 17 19:44:15 kvm-s63702 kernel: [631835.131074] [ 5396] 1000 5396 352306 8527 88 0 0 beam.smp Jan 17 19:44:15 kvm-s63702 kernel: [631835.131076] [ 5432] 1000 5432 2884 1309 10 0 0 gosecrets Jan 17 19:44:15 kvm-s63702 kernel: [631835.131078] [ 5436] 1000 5436 724920 186903 672 0 0 beam.smp Jan 17 19:44:15 kvm-s63702 kernel: [631835.131079] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 17 19:44:15 kvm-s63702 kernel: [631835.131082] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 17 19:44:15 kvm-s63702 kernel: [631835.131083] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 17 19:44:15 kvm-s63702 kernel: [631835.131085] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 17 19:44:15 kvm-s63702 kernel: [631835.131087] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 17 19:44:15 kvm-s63702 kernel: [631835.131089] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 17 19:44:15 kvm-s63702 kernel: [631835.131091] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 17 19:44:15 kvm-s63702 kernel: [631835.131093] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 17 19:44:15 kvm-s63702 kernel: [631835.131095] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 17 19:44:15 kvm-s63702 kernel: [631835.131097] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 17 19:44:15 kvm-s63702 kernel: [631835.131099] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 17 19:44:15 kvm-s63702 kernel: [631835.131101] [14025] 1000 14025 1462 167 8 0 0 goport Jan 17 19:44:15 kvm-s63702 kernel: [631835.131103] [14029] 1000 14029 74588 1838 29 0 0 goxdcr Jan 17 19:44:15 kvm-s63702 kernel: [631835.131105] [14040] 1000 14040 276551 17223 137 0 0 beam.smp Jan 17 19:44:15 kvm-s63702 kernel: [631835.131107] [14076] 1000 14076 1113 361 7 0 0 sh Jan 17 19:44:15 kvm-s63702 kernel: [631835.131108] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 17 19:44:15 kvm-s63702 kernel: [631835.131113] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 17 19:44:15 kvm-s63702 kernel: [631835.131115] [14085] 1000 14085 1331 101 8 0 0 godu Jan 17 19:44:15 kvm-s63702 kernel: [631835.131117] [14086] 1000 14086 1113 320 7 0 0 sh Jan 17 19:44:15 kvm-s63702 kernel: [631835.131118] [14087] 1000 14087 1331 100 8 0 0 godu Jan 17 19:44:15 kvm-s63702 kernel: [631835.131120] [14098] 1000 14098 2156 413 10 0 0 sigar_port Jan 17 19:44:15 kvm-s63702 kernel: [631835.131122] [20999] 0 20999 4303 1234 14 0 0 atop Jan 17 19:44:15 kvm-s63702 kernel: [631835.131123] [32051] 1000 32051 1462 179 8 0 0 goport Jan 17 19:44:15 kvm-s63702 kernel: [631835.131125] [32055] 1000 32055 15684727 7322738 20953 0 0 indexer Jan 17 19:44:15 kvm-s63702 kernel: [631835.131127] Out of memory: Kill process 843 (indexer) score 951 or sacrifice child Jan 17 19:44:15 kvm-s63702 kernel: [631835.134070] Killed process 32055 (indexer) total-vm:62738908kB, anon-rss:29288556kB, file-rss:2660kB Jan 17 22:45:48 kvm-s63702 kernel: [642728.682894] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.682899] indexer cpuset=/ mems_allowed=0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.682917] CPU: 9 PID: 2962 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 17 22:45:48 kvm-s63702 kernel: [642728.682919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 17 22:45:48 kvm-s63702 kernel: [642728.682921] 0000000000000000 ffff8803fa77f9d8 ffffffff817aed00 0000000000000000 Jan 17 22:45:48 kvm-s63702 kernel: [642728.682923] ffff880654ac4e80 ffff8803fa77fa78 ffffffff817a9c3f ffff8803fa77fa98 Jan 17 22:45:48 kvm-s63702 kernel: [642728.682925] 0000000000000000 0000000000000000 0000000000000002 0000000000000001 Jan 17 22:45:48 kvm-s63702 kernel: [642728.682928] Call Trace: Jan 17 22:45:48 kvm-s63702 kernel: [642728.682974] [] dump_stack+0x45/0x57 Jan 17 22:45:48 kvm-s63702 kernel: [642728.682977] [] dump_header+0x7f/0x1f1 Jan 17 22:45:48 kvm-s63702 kernel: [642728.682994] [] oom_kill_process+0x22b/0x390 Jan 17 22:45:48 kvm-s63702 kernel: [642728.682997] [] out_of_memory+0x4d2/0x520 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683000] [] __alloc_pages_nodemask+0x940/0xa60 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683010] [] alloc_pages_vma+0x97/0x150 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683015] [] handle_mm_fault+0xd13/0x1100 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683031] [] ? ttwu_do_activate.constprop.94+0x5d/0x70 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683034] [] ? try_to_wake_up+0x1f1/0x340 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683048] [] __do_page_fault+0x1c4/0x5b0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683054] [] ? __enqueue_entity+0x78/0x80 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683061] [] ? do_futex+0x10a/0x5d0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683064] [] ? pick_next_task_fair+0x61b/0x880 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683074] [] ? __switch_to+0x15f/0x570 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683077] [] ? sched_clock_cpu+0x98/0xc0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683078] [] trace_do_page_fault+0x45/0x100 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683082] [] do_async_page_fault+0x1e/0xd0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683091] [] async_page_fault+0x28/0x30 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683093] Mem-Info: Jan 17 22:45:48 kvm-s63702 kernel: [642728.683096] Node 0 DMA per-cpu: Jan 17 22:45:48 kvm-s63702 kernel: [642728.683100] CPU 0: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683101] CPU 1: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683102] CPU 2: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683103] CPU 3: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683104] CPU 4: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683105] CPU 5: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683106] CPU 6: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683107] CPU 7: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683108] CPU 8: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683110] CPU 9: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683111] CPU 10: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683113] CPU 11: hi: 0, btch: 1 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683114] Node 0 DMA32 per-cpu: Jan 17 22:45:48 kvm-s63702 kernel: [642728.683116] CPU 0: hi: 186, btch: 31 usd: 1 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683118] CPU 1: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683119] CPU 2: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683121] CPU 3: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683122] CPU 4: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683124] CPU 5: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683125] CPU 6: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683127] CPU 7: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683128] CPU 8: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683130] CPU 9: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683131] CPU 10: hi: 186, btch: 31 usd: 30 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683136] CPU 11: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683137] Node 0 Normal per-cpu: Jan 17 22:45:48 kvm-s63702 kernel: [642728.683139] CPU 0: hi: 186, btch: 31 usd: 20 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683140] CPU 1: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683140] CPU 2: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683141] CPU 3: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683142] CPU 4: hi: 186, btch: 31 usd: 18 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683143] CPU 5: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683144] CPU 6: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683145] CPU 7: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683146] CPU 8: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683146] CPU 9: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683147] CPU 10: hi: 186, btch: 31 usd: 30 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683148] CPU 11: hi: 186, btch: 31 usd: 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683152] active_anon:7590687 inactive_anon:72 isolated_anon:0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683152] active_file:130 inactive_file:0 isolated_file:0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683152] unevictable:1199 dirty:11 writeback:72 unstable:0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683152] free:48726 slab_reclaimable:6560 slab_unreclaimable:8718 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683152] mapped:840 shmem:146 pagetables:22469 bounce:0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683152] free_cma:0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683154] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 17 22:45:48 kvm-s63702 kernel: [642728.683161] lowmem_reserve[]: 0 2978 30143 30143 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683163] Node 0 DMA32 free:117256kB min:6668kB low:8332kB high:10000kB active_anon:2904264kB inactive_anon:32kB active_file:96kB inactive_file:0kB unevictable:848kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:848kB dirty:0kB writeback:0kB mapped:792kB shmem:56kB slab_reclaimable:2276kB slab_unreclaimable:3144kB kernel_stack:2176kB pagetables:9168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:544 all_unreclaimable? no Jan 17 22:45:48 kvm-s63702 kernel: [642728.683167] lowmem_reserve[]: 0 0 27165 27165 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683169] Node 0 Normal free:61756kB min:60876kB low:76092kB high:91312kB active_anon:27458484kB inactive_anon:256kB active_file:424kB inactive_file:0kB unevictable:3948kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:3948kB dirty:108kB writeback:288kB mapped:2568kB shmem:528kB slab_reclaimable:23964kB slab_unreclaimable:31712kB kernel_stack:21152kB pagetables:80708kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2668 all_unreclaimable? yes Jan 17 22:45:48 kvm-s63702 kernel: [642728.683173] lowmem_reserve[]: 0 0 0 0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683175] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 17 22:45:48 kvm-s63702 kernel: [642728.683185] Node 0 DMA32: 444*4kB (UEM) 109*8kB (UEM) 127*16kB (UEM) 103*32kB (UEM) 61*64kB (UEM) 38*128kB (UEM) 21*256kB (UE) 11*512kB (UE) 4*1024kB (UEM) 6*2048kB (UM) 18*4096kB (UMR) = 117864kB Jan 17 22:45:48 kvm-s63702 kernel: [642728.683193] Node 0 Normal: 528*4kB (UEM) 293*8kB (UEM) 293*16kB (UEM) 170*32kB (UEM) 116*64kB (UEM) 120*128kB (UEM) 43*256kB (UEM) 13*512kB (UE) 3*1024kB (E) 0*2048kB 1*4096kB (R) = 62200kB Jan 17 22:45:48 kvm-s63702 kernel: [642728.683206] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 17 22:45:48 kvm-s63702 kernel: [642728.683208] 1563 total pagecache pages Jan 17 22:45:48 kvm-s63702 kernel: [642728.683210] 0 pages in swap cache Jan 17 22:45:48 kvm-s63702 kernel: [642728.683211] Swap cache stats: add 0, delete 0, find 0/0 Jan 17 22:45:48 kvm-s63702 kernel: [642728.683213] Free swap = 0kB Jan 17 22:45:48 kvm-s63702 kernel: [642728.683214] Total swap = 0kB Jan 17 22:45:48 kvm-s63702 kernel: [642728.683215] 7864218 pages RAM Jan 17 22:45:48 kvm-s63702 kernel: [642728.683217] 0 pages HighMem/MovableOnly Jan 17 22:45:48 kvm-s63702 kernel: [642728.683218] 143432 pages reserved Jan 17 22:45:48 kvm-s63702 kernel: [642728.683219] 0 pages cma reserved Jan 17 22:45:48 kvm-s63702 kernel: [642728.683220] 0 pages hwpoisoned Jan 17 22:45:48 kvm-s63702 kernel: [642728.683221] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 17 22:45:48 kvm-s63702 kernel: [642728.683228] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 17 22:45:48 kvm-s63702 kernel: [642728.683230] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 17 22:45:48 kvm-s63702 kernel: [642728.683232] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 17 22:45:48 kvm-s63702 kernel: [642728.683237] [ 724] 0 724 2559 833 8 0 0 dhclient Jan 17 22:45:48 kvm-s63702 kernel: [642728.683239] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 17 22:45:48 kvm-s63702 kernel: [642728.683241] [ 932] 101 932 65020 265 29 0 0 rsyslogd Jan 17 22:45:48 kvm-s63702 kernel: [642728.683243] [ 961] 102 961 9809 334 23 0 0 dbus-daemon Jan 17 22:45:48 kvm-s63702 kernel: [642728.683245] [ 964] 0 964 3956 41 12 0 0 getty Jan 17 22:45:48 kvm-s63702 kernel: [642728.683246] [ 967] 0 967 3956 40 13 0 0 getty Jan 17 22:45:48 kvm-s63702 kernel: [642728.683248] [ 972] 0 972 3956 40 13 0 0 getty Jan 17 22:45:48 kvm-s63702 kernel: [642728.683249] [ 973] 0 973 3956 39 13 0 0 getty Jan 17 22:45:48 kvm-s63702 kernel: [642728.683251] [ 975] 0 975 3956 39 13 0 0 getty Jan 17 22:45:48 kvm-s63702 kernel: [642728.683252] [ 995] 0 995 15343 301 33 0 -1000 sshd Jan 17 22:45:48 kvm-s63702 kernel: [642728.683254] [ 998] 0 998 1093 37 8 0 0 acpid Jan 17 22:45:48 kvm-s63702 kernel: [642728.683256] [ 1016] 0 1016 5915 238 18 0 0 cron Jan 17 22:45:48 kvm-s63702 kernel: [642728.683257] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 17 22:45:48 kvm-s63702 kernel: [642728.683259] [ 1046] 0 1046 10864 408 27 0 0 systemd-logind Jan 17 22:45:48 kvm-s63702 kernel: [642728.683260] [ 1052] 0 1052 4799 287 14 0 0 irqbalance Jan 17 22:45:48 kvm-s63702 kernel: [642728.683262] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 17 22:45:48 kvm-s63702 kernel: [642728.683264] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 17 22:45:48 kvm-s63702 kernel: [642728.683266] [ 5396] 1000 5396 353403 8536 90 0 0 beam.smp Jan 17 22:45:48 kvm-s63702 kernel: [642728.683268] [ 5432] 1000 5432 2884 1305 10 0 0 gosecrets Jan 17 22:45:48 kvm-s63702 kernel: [642728.683270] [ 5436] 1000 5436 740986 202219 734 0 0 beam.smp Jan 17 22:45:48 kvm-s63702 kernel: [642728.683272] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 17 22:45:48 kvm-s63702 kernel: [642728.683274] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 17 22:45:48 kvm-s63702 kernel: [642728.683276] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 17 22:45:48 kvm-s63702 kernel: [642728.683277] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 17 22:45:48 kvm-s63702 kernel: [642728.683279] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 17 22:45:48 kvm-s63702 kernel: [642728.683281] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 17 22:45:48 kvm-s63702 kernel: [642728.683282] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 17 22:45:48 kvm-s63702 kernel: [642728.683284] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 17 22:45:48 kvm-s63702 kernel: [642728.683287] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 17 22:45:48 kvm-s63702 kernel: [642728.683289] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 17 22:45:48 kvm-s63702 kernel: [642728.683291] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 17 22:45:48 kvm-s63702 kernel: [642728.683295] [14025] 1000 14025 1462 167 8 0 0 goport Jan 17 22:45:48 kvm-s63702 kernel: [642728.683297] [14029] 1000 14029 74588 1970 29 0 0 goxdcr Jan 17 22:45:48 kvm-s63702 kernel: [642728.683299] [14040] 1000 14040 279344 19620 144 0 0 beam.smp Jan 17 22:45:48 kvm-s63702 kernel: [642728.683302] [14076] 1000 14076 1113 361 7 0 0 sh Jan 17 22:45:48 kvm-s63702 kernel: [642728.683304] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 17 22:45:48 kvm-s63702 kernel: [642728.683306] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 17 22:45:48 kvm-s63702 kernel: [642728.683309] [14085] 1000 14085 1331 101 8 0 0 godu Jan 17 22:45:48 kvm-s63702 kernel: [642728.683311] [14086] 1000 14086 1113 320 7 0 0 sh Jan 17 22:45:48 kvm-s63702 kernel: [642728.683313] [14087] 1000 14087 1331 100 8 0 0 godu Jan 17 22:45:48 kvm-s63702 kernel: [642728.683314] [14098] 1000 14098 2156 418 10 0 0 sigar_port Jan 17 22:45:48 kvm-s63702 kernel: [642728.683316] [20999] 0 20999 4270 1201 14 0 0 atop Jan 17 22:45:48 kvm-s63702 kernel: [642728.683317] [ 2239] 1000 2239 1462 163 8 0 0 goport Jan 17 22:45:48 kvm-s63702 kernel: [642728.683319] [ 2245] 1000 2245 15625491 7305068 20746 0 0 indexer Jan 17 22:45:48 kvm-s63702 kernel: [642728.683324] Out of memory: Kill process 2245 (indexer) score 948 or sacrifice child Jan 17 22:45:48 kvm-s63702 kernel: [642728.686563] Killed process 2245 (indexer) total-vm:62501964kB, anon-rss:29218472kB, file-rss:1800kB Jan 17 23:35:40 kvm-s63702 kernel: [645720.845625] TCP: request_sock_TCP: Possible SYN flooding on port 9101. Sending cookies. Check SNMP counters. Jan 18 00:00:02 kvm-s63702 kernel: [647182.999397] Process accounting resumed Jan 18 01:12:20 kvm-s63702 kernel: [651520.184263] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184270] indexer cpuset=/ mems_allowed=0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184291] CPU: 4 PID: 3981 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 18 01:12:20 kvm-s63702 kernel: [651520.184293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184295] 0000000000000000 ffff88013bafb9d8 ffffffff817aed00 0000000000000000 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184298] ffff8800bb20ce80 ffff88013bafba78 ffffffff817a9c3f 0000000000000000 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184301] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184305] Call Trace: Jan 18 01:12:20 kvm-s63702 kernel: [651520.184358] [] dump_stack+0x45/0x57 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184362] [] dump_header+0x7f/0x1f1 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184385] [] oom_kill_process+0x22b/0x390 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184388] [] out_of_memory+0x4d2/0x520 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184393] [] __alloc_pages_nodemask+0x940/0xa60 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184405] [] alloc_pages_vma+0x97/0x150 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184413] [] handle_mm_fault+0xd13/0x1100 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184430] [] ? ttwu_do_activate.constprop.94+0x5d/0x70 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184432] [] ? try_to_wake_up+0x1f1/0x340 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184446] [] __do_page_fault+0x1c4/0x5b0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184453] [] ? __enqueue_entity+0x78/0x80 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184455] [] ? set_next_entity+0x9d/0xb0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184459] [] ? pick_next_task_fair+0x61b/0x880 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184473] [] ? __switch_to+0x15f/0x570 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184476] [] ? sched_clock_cpu+0x98/0xc0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184479] [] trace_do_page_fault+0x45/0x100 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184483] [] do_async_page_fault+0x1e/0xd0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184496] [] async_page_fault+0x28/0x30 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184498] Mem-Info: Jan 18 01:12:20 kvm-s63702 kernel: [651520.184504] Node 0 DMA per-cpu: Jan 18 01:12:20 kvm-s63702 kernel: [651520.184509] CPU 0: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184511] CPU 1: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184513] CPU 2: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184514] CPU 3: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184516] CPU 4: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184517] CPU 5: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184519] CPU 6: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184520] CPU 7: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184522] CPU 8: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184523] CPU 9: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184525] CPU 10: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184526] CPU 11: hi: 0, btch: 1 usd: 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184527] Node 0 DMA32 per-cpu: Jan 18 01:12:20 kvm-s63702 kernel: [651520.184529] CPU 0: hi: 186, btch: 31 usd: 15 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184531] CPU 1: hi: 186, btch: 31 usd: 60 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184532] CPU 2: hi: 186, btch: 31 usd: 39 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184534] CPU 3: hi: 186, btch: 31 usd: 32 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184535] CPU 4: hi: 186, btch: 31 usd: 26 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184536] CPU 5: hi: 186, btch: 31 usd: 19 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184538] CPU 6: hi: 186, btch: 31 usd: 156 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184539] CPU 7: hi: 186, btch: 31 usd: 9 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184541] CPU 8: hi: 186, btch: 31 usd: 64 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184542] CPU 9: hi: 186, btch: 31 usd: 20 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184543] CPU 10: hi: 186, btch: 31 usd: 32 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184545] CPU 11: hi: 186, btch: 31 usd: 19 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184546] Node 0 Normal per-cpu: Jan 18 01:12:20 kvm-s63702 kernel: [651520.184548] CPU 0: hi: 186, btch: 31 usd: 35 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184550] CPU 1: hi: 186, btch: 31 usd: 30 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184551] CPU 2: hi: 186, btch: 31 usd: 17 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184553] CPU 3: hi: 186, btch: 31 usd: 15 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184554] CPU 4: hi: 186, btch: 31 usd: 58 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184556] CPU 5: hi: 186, btch: 31 usd: 60 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184557] CPU 6: hi: 186, btch: 31 usd: 152 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184564] CPU 7: hi: 186, btch: 31 usd: 38 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184566] CPU 8: hi: 186, btch: 31 usd: 1 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184567] CPU 9: hi: 186, btch: 31 usd: 23 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184569] CPU 10: hi: 186, btch: 31 usd: 40 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184570] CPU 11: hi: 186, btch: 31 usd: 3 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184575] active_anon:7589814 inactive_anon:73 isolated_anon:0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184575] active_file:553 inactive_file:379 isolated_file:0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184575] unevictable:1233 dirty:616 writeback:16 unstable:0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184575] free:47943 slab_reclaimable:7242 slab_unreclaimable:8993 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184575] mapped:1281 shmem:99 pagetables:20256 bounce:0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184575] free_cma:0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184579] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 18 01:12:20 kvm-s63702 kernel: [651520.184590] lowmem_reserve[]: 0 2978 30143 30143 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184593] Node 0 DMA32 free:115304kB min:6668kB low:8332kB high:10000kB active_anon:2906824kB inactive_anon:32kB active_file:140kB inactive_file:0kB unevictable:884kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:884kB dirty:244kB writeback:0kB mapped:828kB shmem:32kB slab_reclaimable:2468kB slab_unreclaimable:3216kB kernel_stack:2240kB pagetables:7676kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:151700 all_unreclaimable? yes Jan 18 01:12:20 kvm-s63702 kernel: [651520.184599] lowmem_reserve[]: 0 0 27165 27165 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184602] Node 0 Normal free:60576kB min:60876kB low:76092kB high:91312kB active_anon:27452432kB inactive_anon:260kB active_file:2072kB inactive_file:1520kB unevictable:4048kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:4048kB dirty:2220kB writeback:64kB mapped:4296kB shmem:364kB slab_reclaimable:26500kB slab_unreclaimable:32740kB kernel_stack:21072kB pagetables:73348kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:25164 all_unreclaimable? yes Jan 18 01:12:20 kvm-s63702 kernel: [651520.184608] lowmem_reserve[]: 0 0 0 0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184611] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 18 01:12:20 kvm-s63702 kernel: [651520.184627] Node 0 DMA32: 86*4kB (UE) 69*8kB (UEM) 137*16kB (UEM) 108*32kB (EM) 65*64kB (UE) 37*128kB (UE) 15*256kB (UE) 15*512kB (UEM) 4*1024kB (UEM) 5*2048kB (UM) 18*4096kB (UMR) = 115024kB Jan 18 01:12:20 kvm-s63702 kernel: [651520.184639] Node 0 Normal: 314*4kB (UEM) 240*8kB (UE) 330*16kB (UEM) 254*32kB (UEM) 222*64kB (UE) 119*128kB (UEM) 32*256kB (UEM) 1*512kB (E) 0*1024kB 0*2048kB 1*4096kB (R) = 58824kB Jan 18 01:12:20 kvm-s63702 kernel: [651520.184653] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 18 01:12:20 kvm-s63702 kernel: [651520.184654] 2441 total pagecache pages Jan 18 01:12:20 kvm-s63702 kernel: [651520.184656] 0 pages in swap cache Jan 18 01:12:20 kvm-s63702 kernel: [651520.184658] Swap cache stats: add 0, delete 0, find 0/0 Jan 18 01:12:20 kvm-s63702 kernel: [651520.184659] Free swap = 0kB Jan 18 01:12:20 kvm-s63702 kernel: [651520.184660] Total swap = 0kB Jan 18 01:12:20 kvm-s63702 kernel: [651520.184661] 7864218 pages RAM Jan 18 01:12:20 kvm-s63702 kernel: [651520.184662] 0 pages HighMem/MovableOnly Jan 18 01:12:20 kvm-s63702 kernel: [651520.184663] 143432 pages reserved Jan 18 01:12:20 kvm-s63702 kernel: [651520.184665] 0 pages cma reserved Jan 18 01:12:20 kvm-s63702 kernel: [651520.184666] 0 pages hwpoisoned Jan 18 01:12:20 kvm-s63702 kernel: [651520.184667] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 18 01:12:20 kvm-s63702 kernel: [651520.184679] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 18 01:12:20 kvm-s63702 kernel: [651520.184682] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 18 01:12:20 kvm-s63702 kernel: [651520.184685] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 18 01:12:20 kvm-s63702 kernel: [651520.184688] [ 724] 0 724 2559 833 8 0 0 dhclient Jan 18 01:12:20 kvm-s63702 kernel: [651520.184690] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 18 01:12:20 kvm-s63702 kernel: [651520.184693] [ 932] 101 932 65020 265 29 0 0 rsyslogd Jan 18 01:12:20 kvm-s63702 kernel: [651520.184695] [ 961] 102 961 9809 339 23 0 0 dbus-daemon Jan 18 01:12:20 kvm-s63702 kernel: [651520.184698] [ 964] 0 964 3956 41 12 0 0 getty Jan 18 01:12:20 kvm-s63702 kernel: [651520.184706] [ 967] 0 967 3956 40 13 0 0 getty Jan 18 01:12:20 kvm-s63702 kernel: [651520.184709] [ 972] 0 972 3956 40 13 0 0 getty Jan 18 01:12:20 kvm-s63702 kernel: [651520.184711] [ 973] 0 973 3956 39 13 0 0 getty Jan 18 01:12:20 kvm-s63702 kernel: [651520.184713] [ 975] 0 975 3956 39 13 0 0 getty Jan 18 01:12:20 kvm-s63702 kernel: [651520.184714] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 18 01:12:20 kvm-s63702 kernel: [651520.184716] [ 998] 0 998 1093 37 8 0 0 acpid Jan 18 01:12:20 kvm-s63702 kernel: [651520.184718] [ 1016] 0 1016 5915 238 18 0 0 cron Jan 18 01:12:20 kvm-s63702 kernel: [651520.184719] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 18 01:12:20 kvm-s63702 kernel: [651520.184721] [ 1046] 0 1046 10864 413 27 0 0 systemd-logind Jan 18 01:12:20 kvm-s63702 kernel: [651520.184722] [ 1052] 0 1052 4799 287 14 0 0 irqbalance Jan 18 01:12:20 kvm-s63702 kernel: [651520.184724] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 18 01:12:20 kvm-s63702 kernel: [651520.184727] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 18 01:12:20 kvm-s63702 kernel: [651520.184728] [ 5396] 1000 5396 351793 8479 87 0 0 beam.smp Jan 18 01:12:20 kvm-s63702 kernel: [651520.184730] [ 5432] 1000 5432 2884 1245 10 0 0 gosecrets Jan 18 01:12:20 kvm-s63702 kernel: [651520.184732] [ 5436] 1000 5436 759674 229928 760 0 0 beam.smp Jan 18 01:12:20 kvm-s63702 kernel: [651520.184734] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 18 01:12:20 kvm-s63702 kernel: [651520.184735] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 18 01:12:20 kvm-s63702 kernel: [651520.184737] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 18 01:12:20 kvm-s63702 kernel: [651520.184738] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 18 01:12:20 kvm-s63702 kernel: [651520.184740] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 18 01:12:20 kvm-s63702 kernel: [651520.184742] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 18 01:12:20 kvm-s63702 kernel: [651520.184743] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 18 01:12:20 kvm-s63702 kernel: [651520.184745] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 18 01:12:20 kvm-s63702 kernel: [651520.184746] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 18 01:12:20 kvm-s63702 kernel: [651520.184748] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 18 01:12:20 kvm-s63702 kernel: [651520.184749] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 18 01:12:20 kvm-s63702 kernel: [651520.184752] [14025] 1000 14025 1462 167 8 0 0 goport Jan 18 01:12:20 kvm-s63702 kernel: [651520.184753] [14029] 1000 14029 74588 1975 29 0 0 goxdcr Jan 18 01:12:20 kvm-s63702 kernel: [651520.184755] [14040] 1000 14040 280139 20341 143 0 0 beam.smp Jan 18 01:12:20 kvm-s63702 kernel: [651520.184756] [14076] 1000 14076 1113 336 7 0 0 sh Jan 18 01:12:20 kvm-s63702 kernel: [651520.184758] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 18 01:12:20 kvm-s63702 kernel: [651520.184759] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 18 01:12:20 kvm-s63702 kernel: [651520.184761] [14085] 1000 14085 1331 101 8 0 0 godu Jan 18 01:12:20 kvm-s63702 kernel: [651520.184762] [14086] 1000 14086 1113 320 7 0 0 sh Jan 18 01:12:20 kvm-s63702 kernel: [651520.184764] [14087] 1000 14087 1331 100 8 0 0 godu Jan 18 01:12:20 kvm-s63702 kernel: [651520.184765] [14098] 1000 14098 2156 418 10 0 0 sigar_port Jan 18 01:12:20 kvm-s63702 kernel: [651520.184767] [ 3634] 1000 3634 1726 160 8 0 0 goport Jan 18 01:12:20 kvm-s63702 kernel: [651520.184769] [ 3639] 1000 3639 14242176 7275690 18521 0 0 indexer Jan 18 01:12:20 kvm-s63702 kernel: [651520.184770] [ 6503] 0 6503 4304 1235 15 0 0 atop Jan 18 01:12:20 kvm-s63702 kernel: [651520.184772] Out of memory: Kill process 3639 (indexer) score 944 or sacrifice child Jan 18 01:12:20 kvm-s63702 kernel: [651520.188748] Killed process 3639 (indexer) total-vm:56968704kB, anon-rss:29100708kB, file-rss:2580kB Jan 18 04:28:06 kvm-s63702 kernel: [663266.467754] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467762] indexer cpuset=/ mems_allowed=0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467782] CPU: 5 PID: 8483 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 18 04:28:07 kvm-s63702 kernel: [663266.467787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467789] 0000000000000000 ffff8807988b39d8 ffffffff817aed00 0000000000000000 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467793] ffff880687d74e80 ffff8807988b3a78 ffffffff817a9c3f 0000000000000000 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467796] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467799] Call Trace: Jan 18 04:28:07 kvm-s63702 kernel: [663266.467848] [] dump_stack+0x45/0x57 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467851] [] dump_header+0x7f/0x1f1 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467869] [] oom_kill_process+0x22b/0x390 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467871] [] out_of_memory+0x4d2/0x520 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467875] [] __alloc_pages_nodemask+0x940/0xa60 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467884] [] alloc_pages_vma+0x97/0x150 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467890] [] handle_mm_fault+0xd13/0x1100 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467904] [] ? x2apic_send_IPI_mask+0x13/0x20 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467915] [] ? try_to_wake_up+0x1f1/0x340 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467925] [] __do_page_fault+0x1c4/0x5b0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467932] [] ? __enqueue_entity+0x78/0x80 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467942] [] ? do_futex+0x10a/0x5d0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467946] [] ? pick_next_task_fair+0x61b/0x880 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467959] [] ? __switch_to+0x15f/0x570 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467963] [] ? sched_clock_cpu+0x98/0xc0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467966] [] trace_do_page_fault+0x45/0x100 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467970] [] do_async_page_fault+0x1e/0xd0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467982] [] async_page_fault+0x28/0x30 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467984] Mem-Info: Jan 18 04:28:07 kvm-s63702 kernel: [663266.467989] Node 0 DMA per-cpu: Jan 18 04:28:07 kvm-s63702 kernel: [663266.467995] CPU 0: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467997] CPU 1: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.467999] CPU 2: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468000] CPU 3: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468025] CPU 4: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468026] CPU 5: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468027] CPU 6: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468028] CPU 7: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468029] CPU 8: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468030] CPU 9: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468031] CPU 10: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468032] CPU 11: hi: 0, btch: 1 usd: 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468033] Node 0 DMA32 per-cpu: Jan 18 04:28:07 kvm-s63702 kernel: [663266.468035] CPU 0: hi: 186, btch: 31 usd: 47 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468036] CPU 1: hi: 186, btch: 31 usd: 32 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468037] CPU 2: hi: 186, btch: 31 usd: 15 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468038] CPU 3: hi: 186, btch: 31 usd: 74 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468039] CPU 4: hi: 186, btch: 31 usd: 39 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468043] CPU 5: hi: 186, btch: 31 usd: 25 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468044] CPU 6: hi: 186, btch: 31 usd: 26 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468045] CPU 7: hi: 186, btch: 31 usd: 28 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468046] CPU 8: hi: 186, btch: 31 usd: 35 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468047] CPU 9: hi: 186, btch: 31 usd: 39 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468051] CPU 10: hi: 186, btch: 31 usd: 40 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468055] CPU 11: hi: 186, btch: 31 usd: 31 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468059] Node 0 Normal per-cpu: Jan 18 04:28:07 kvm-s63702 kernel: [663266.468065] CPU 0: hi: 186, btch: 31 usd: 175 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468069] CPU 1: hi: 186, btch: 31 usd: 24 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468074] CPU 2: hi: 186, btch: 31 usd: 23 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468079] CPU 3: hi: 186, btch: 31 usd: 22 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468099] CPU 4: hi: 186, btch: 31 usd: 58 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468104] CPU 5: hi: 186, btch: 31 usd: 47 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468109] CPU 6: hi: 186, btch: 31 usd: 31 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468113] CPU 7: hi: 186, btch: 31 usd: 32 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468118] CPU 8: hi: 186, btch: 31 usd: 18 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468123] CPU 9: hi: 186, btch: 31 usd: 51 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468126] CPU 10: hi: 186, btch: 31 usd: 29 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468128] CPU 11: hi: 186, btch: 31 usd: 53 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468133] active_anon:7590588 inactive_anon:73 isolated_anon:0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468133] active_file:0 inactive_file:24 isolated_file:32 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468133] unevictable:1232 dirty:1 writeback:61 unstable:0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468133] free:47922 slab_reclaimable:6819 slab_unreclaimable:8687 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468133] mapped:864 shmem:106 pagetables:22388 bounce:0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468133] free_cma:0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468137] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 18 04:28:07 kvm-s63702 kernel: [663266.468153] lowmem_reserve[]: 0 2978 30143 30143 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468156] Node 0 DMA32 free:115220kB min:6668kB low:8332kB high:10000kB active_anon:2906380kB inactive_anon:32kB active_file:56kB inactive_file:0kB unevictable:760kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:760kB dirty:0kB writeback:0kB mapped:832kB shmem:32kB slab_reclaimable:2216kB slab_unreclaimable:2648kB kernel_stack:2288kB pagetables:10576kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:189556 all_unreclaimable? yes Jan 18 04:28:07 kvm-s63702 kernel: [663266.468161] lowmem_reserve[]: 0 0 27165 27165 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468164] Node 0 Normal free:60576kB min:60876kB low:76092kB high:91312kB active_anon:27455972kB inactive_anon:260kB active_file:0kB inactive_file:404kB unevictable:4168kB isolated(anon):0kB isolated(file):128kB present:28311552kB managed:27817664kB mlocked:4168kB dirty:4kB writeback:244kB mapped:2624kB shmem:392kB slab_reclaimable:25060kB slab_unreclaimable:32084kB kernel_stack:21168kB pagetables:78976kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:19888 all_unreclaimable? yes Jan 18 04:28:07 kvm-s63702 kernel: [663266.468170] lowmem_reserve[]: 0 0 0 0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468173] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 18 04:28:07 kvm-s63702 kernel: [663266.468190] Node 0 DMA32: 79*4kB (E) 54*8kB (EM) 114*16kB (UEM) 96*32kB (UEM) 58*64kB (UEM) 37*128kB (UE) 23*256kB (UEM) 13*512kB (UEM) 2*1024kB (UE) 6*2048kB (UM) 18*4096kB (UMR) = 114700kB Jan 18 04:28:07 kvm-s63702 kernel: [663266.468203] Node 0 Normal: 402*4kB (UEM) 283*8kB (UEM) 289*16kB (UEM) 188*32kB (UEM) 141*64kB (UE) 133*128kB (UE) 42*256kB (UEM) 7*512kB (E) 0*1024kB 0*2048kB 1*4096kB (R) = 58992kB Jan 18 04:28:07 kvm-s63702 kernel: [663266.468217] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 18 04:28:07 kvm-s63702 kernel: [663266.468219] 1484 total pagecache pages Jan 18 04:28:07 kvm-s63702 kernel: [663266.468221] 0 pages in swap cache Jan 18 04:28:07 kvm-s63702 kernel: [663266.468223] Swap cache stats: add 0, delete 0, find 0/0 Jan 18 04:28:07 kvm-s63702 kernel: [663266.468224] Free swap = 0kB Jan 18 04:28:07 kvm-s63702 kernel: [663266.468225] Total swap = 0kB Jan 18 04:28:07 kvm-s63702 kernel: [663266.468226] 7864218 pages RAM Jan 18 04:28:07 kvm-s63702 kernel: [663266.468227] 0 pages HighMem/MovableOnly Jan 18 04:28:07 kvm-s63702 kernel: [663266.468228] 143432 pages reserved Jan 18 04:28:07 kvm-s63702 kernel: [663266.468230] 0 pages cma reserved Jan 18 04:28:07 kvm-s63702 kernel: [663266.468231] 0 pages hwpoisoned Jan 18 04:28:07 kvm-s63702 kernel: [663266.468232] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 18 04:28:07 kvm-s63702 kernel: [663266.468243] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 18 04:28:07 kvm-s63702 kernel: [663266.468247] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 18 04:28:07 kvm-s63702 kernel: [663266.468250] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 18 04:28:07 kvm-s63702 kernel: [663266.468253] [ 724] 0 724 2559 821 8 0 0 dhclient Jan 18 04:28:07 kvm-s63702 kernel: [663266.468256] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 18 04:28:07 kvm-s63702 kernel: [663266.468258] [ 932] 101 932 65020 339 29 0 0 rsyslogd Jan 18 04:28:07 kvm-s63702 kernel: [663266.468261] [ 961] 102 961 9809 337 23 0 0 dbus-daemon Jan 18 04:28:07 kvm-s63702 kernel: [663266.468264] [ 964] 0 964 3956 41 12 0 0 getty Jan 18 04:28:07 kvm-s63702 kernel: [663266.468266] [ 967] 0 967 3956 40 13 0 0 getty Jan 18 04:28:07 kvm-s63702 kernel: [663266.468268] [ 972] 0 972 3956 40 13 0 0 getty Jan 18 04:28:07 kvm-s63702 kernel: [663266.468270] [ 973] 0 973 3956 39 13 0 0 getty Jan 18 04:28:07 kvm-s63702 kernel: [663266.468272] [ 975] 0 975 3956 39 13 0 0 getty Jan 18 04:28:07 kvm-s63702 kernel: [663266.468274] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 18 04:28:07 kvm-s63702 kernel: [663266.468277] [ 998] 0 998 1093 37 8 0 0 acpid Jan 18 04:28:07 kvm-s63702 kernel: [663266.468280] [ 1016] 0 1016 5915 228 18 0 0 cron Jan 18 04:28:07 kvm-s63702 kernel: [663266.468287] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 18 04:28:07 kvm-s63702 kernel: [663266.468291] [ 1046] 0 1046 10864 411 27 0 0 systemd-logind Jan 18 04:28:07 kvm-s63702 kernel: [663266.468293] [ 1052] 0 1052 4799 286 14 0 0 irqbalance Jan 18 04:28:07 kvm-s63702 kernel: [663266.468296] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 18 04:28:07 kvm-s63702 kernel: [663266.468300] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 18 04:28:07 kvm-s63702 kernel: [663266.468303] [ 5396] 1000 5396 353550 8593 90 0 0 beam.smp Jan 18 04:28:07 kvm-s63702 kernel: [663266.468305] [ 5432] 1000 5432 2884 1221 10 0 0 gosecrets Jan 18 04:28:07 kvm-s63702 kernel: [663266.468308] [ 5436] 1000 5436 741467 210677 701 0 0 beam.smp Jan 18 04:28:07 kvm-s63702 kernel: [663266.468310] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 18 04:28:07 kvm-s63702 kernel: [663266.468313] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 18 04:28:07 kvm-s63702 kernel: [663266.468315] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 18 04:28:07 kvm-s63702 kernel: [663266.468317] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 18 04:28:07 kvm-s63702 kernel: [663266.468320] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 18 04:28:07 kvm-s63702 kernel: [663266.468322] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 18 04:28:07 kvm-s63702 kernel: [663266.468324] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 18 04:28:07 kvm-s63702 kernel: [663266.468326] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 18 04:28:07 kvm-s63702 kernel: [663266.468328] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 18 04:28:07 kvm-s63702 kernel: [663266.468329] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 18 04:28:07 kvm-s63702 kernel: [663266.468331] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 18 04:28:07 kvm-s63702 kernel: [663266.468334] [14025] 1000 14025 1462 167 8 0 0 goport Jan 18 04:28:07 kvm-s63702 kernel: [663266.468335] [14029] 1000 14029 74588 1919 29 0 0 goxdcr Jan 18 04:28:07 kvm-s63702 kernel: [663266.468337] [14040] 1000 14040 279017 18319 141 0 0 beam.smp Jan 18 04:28:07 kvm-s63702 kernel: [663266.468339] [14076] 1000 14076 1113 361 7 0 0 sh Jan 18 04:28:07 kvm-s63702 kernel: [663266.468340] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 18 04:28:07 kvm-s63702 kernel: [663266.468341] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 18 04:28:07 kvm-s63702 kernel: [663266.468343] [14085] 1000 14085 1331 101 8 0 0 godu Jan 18 04:28:07 kvm-s63702 kernel: [663266.468345] [14086] 1000 14086 1113 320 7 0 0 sh Jan 18 04:28:07 kvm-s63702 kernel: [663266.468346] [14087] 1000 14087 1331 100 8 0 0 godu Jan 18 04:28:07 kvm-s63702 kernel: [663266.468348] [14098] 1000 14098 2156 416 10 0 0 sigar_port Jan 18 04:28:07 kvm-s63702 kernel: [663266.468350] [ 6503] 0 6503 4303 1234 15 0 0 atop Jan 18 04:28:07 kvm-s63702 kernel: [663266.468351] [ 7977] 1000 7977 1462 152 8 0 0 goport Jan 18 04:28:07 kvm-s63702 kernel: [663266.468353] [ 7982] 1000 7982 15531036 7300993 20714 0 0 indexer Jan 18 04:28:07 kvm-s63702 kernel: [663266.468355] Out of memory: Kill process 7982 (indexer) score 948 or sacrifice child Jan 18 04:28:07 kvm-s63702 kernel: [663266.472463] Killed process 7982 (indexer) total-vm:62124144kB, anon-rss:29202900kB, file-rss:1072kB Jan 18 07:33:57 kvm-s63702 kernel: [674417.184051] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184058] indexer cpuset=/ mems_allowed=0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184083] CPU: 2 PID: 10399 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 18 07:33:57 kvm-s63702 kernel: [674417.184085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184087] 0000000000000000 ffff88079ba3f9d8 ffffffff817aed00 0000000000000000 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184091] ffff8800369e13a0 ffff88079ba3fa78 ffffffff817a9c3f 0000000000000000 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184093] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184096] Call Trace: Jan 18 07:33:57 kvm-s63702 kernel: [674417.184157] [] dump_stack+0x45/0x57 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184162] [] dump_header+0x7f/0x1f1 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184190] [] oom_kill_process+0x22b/0x390 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184194] [] out_of_memory+0x4d2/0x520 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184199] [] __alloc_pages_nodemask+0x940/0xa60 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184213] [] alloc_pages_vma+0x97/0x150 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184219] [] handle_mm_fault+0xd13/0x1100 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184240] [] ? ttwu_do_activate.constprop.94+0x5d/0x70 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184243] [] ? try_to_wake_up+0x1f1/0x340 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184253] [] __do_page_fault+0x1c4/0x5b0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184258] [] ? __enqueue_entity+0x78/0x80 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184261] [] ? set_next_entity+0x9d/0xb0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184263] [] ? pick_next_task_fair+0x61b/0x880 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184274] [] ? __switch_to+0x15f/0x570 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184276] [] ? sched_clock_cpu+0x98/0xc0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184278] [] trace_do_page_fault+0x45/0x100 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184282] [] do_async_page_fault+0x1e/0xd0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184291] [] async_page_fault+0x28/0x30 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184293] Mem-Info: Jan 18 07:33:57 kvm-s63702 kernel: [674417.184297] Node 0 DMA per-cpu: Jan 18 07:33:57 kvm-s63702 kernel: [674417.184301] CPU 0: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184302] CPU 1: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184303] CPU 2: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184304] CPU 3: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184305] CPU 4: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184306] CPU 5: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184307] CPU 6: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184308] CPU 7: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184309] CPU 8: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184310] CPU 9: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184311] CPU 10: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184311] CPU 11: hi: 0, btch: 1 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184312] Node 0 DMA32 per-cpu: Jan 18 07:33:57 kvm-s63702 kernel: [674417.184314] CPU 0: hi: 186, btch: 31 usd: 16 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184315] CPU 1: hi: 186, btch: 31 usd: 30 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184316] CPU 2: hi: 186, btch: 31 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184316] CPU 3: hi: 186, btch: 31 usd: 30 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184317] CPU 4: hi: 186, btch: 31 usd: 30 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184318] CPU 5: hi: 186, btch: 31 usd: 30 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184319] CPU 6: hi: 186, btch: 31 usd: 30 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184320] CPU 7: hi: 186, btch: 31 usd: 28 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184321] CPU 8: hi: 186, btch: 31 usd: 23 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184322] CPU 9: hi: 186, btch: 31 usd: 30 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184323] CPU 10: hi: 186, btch: 31 usd: 21 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184324] CPU 11: hi: 186, btch: 31 usd: 17 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184324] Node 0 Normal per-cpu: Jan 18 07:33:57 kvm-s63702 kernel: [674417.184325] CPU 0: hi: 186, btch: 31 usd: 65 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184326] CPU 1: hi: 186, btch: 31 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184327] CPU 2: hi: 186, btch: 31 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184328] CPU 3: hi: 186, btch: 31 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184329] CPU 4: hi: 186, btch: 31 usd: 2 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184330] CPU 5: hi: 186, btch: 31 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184331] CPU 6: hi: 186, btch: 31 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184332] CPU 7: hi: 186, btch: 31 usd: 46 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184333] CPU 8: hi: 186, btch: 31 usd: 9 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184333] CPU 9: hi: 186, btch: 31 usd: 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184334] CPU 10: hi: 186, btch: 31 usd: 19 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184335] CPU 11: hi: 186, btch: 31 usd: 30 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184338] active_anon:7588598 inactive_anon:72 isolated_anon:0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184338] active_file:188 inactive_file:37 isolated_file:0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184338] unevictable:1231 dirty:330 writeback:0 unstable:0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184338] free:48069 slab_reclaimable:6711 slab_unreclaimable:9061 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184338] mapped:865 shmem:115 pagetables:23341 bounce:0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184338] free_cma:0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184342] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 18 07:33:57 kvm-s63702 kernel: [674417.184348] lowmem_reserve[]: 0 2978 30143 30143 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184350] Node 0 DMA32 free:115560kB min:6668kB low:8332kB high:10000kB active_anon:2906496kB inactive_anon:32kB active_file:0kB inactive_file:0kB unevictable:756kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:756kB dirty:100kB writeback:0kB mapped:360kB shmem:32kB slab_reclaimable:1972kB slab_unreclaimable:3604kB kernel_stack:1984kB pagetables:9140kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4352 all_unreclaimable? yes Jan 18 07:33:57 kvm-s63702 kernel: [674417.184356] lowmem_reserve[]: 0 0 27165 27165 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184358] Node 0 Normal free:60824kB min:60876kB low:76092kB high:91312kB active_anon:27447896kB inactive_anon:256kB active_file:844kB inactive_file:232kB unevictable:4168kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:4168kB dirty:1220kB writeback:0kB mapped:3100kB shmem:428kB slab_reclaimable:24872kB slab_unreclaimable:32624kB kernel_stack:22064kB pagetables:84224kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:13916 all_unreclaimable? yes Jan 18 07:33:57 kvm-s63702 kernel: [674417.184363] lowmem_reserve[]: 0 0 0 0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184365] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 18 07:33:57 kvm-s63702 kernel: [674417.184375] Node 0 DMA32: 108*4kB (UEM) 76*8kB (UE) 112*16kB (UEM) 95*32kB (UEM) 53*64kB (UEM) 31*128kB (UEM) 18*256kB (UE) 14*512kB (UE) 4*1024kB (UE) 6*2048kB (UM) 18*4096kB (UMR) = 115120kB Jan 18 07:33:57 kvm-s63702 kernel: [674417.184383] Node 0 Normal: 636*4kB (UEM) 364*8kB (UEM) 384*16kB (UEM) 227*32kB (UEM) 159*64kB (UEM) 97*128kB (UE) 45*256kB (UEM) 9*512kB (EM) 0*1024kB 0*2048kB 1*4096kB (R) = 61680kB Jan 18 07:33:57 kvm-s63702 kernel: [674417.184392] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 18 07:33:57 kvm-s63702 kernel: [674417.184393] 1818 total pagecache pages Jan 18 07:33:57 kvm-s63702 kernel: [674417.184395] 0 pages in swap cache Jan 18 07:33:57 kvm-s63702 kernel: [674417.184396] Swap cache stats: add 0, delete 0, find 0/0 Jan 18 07:33:57 kvm-s63702 kernel: [674417.184397] Free swap = 0kB Jan 18 07:33:57 kvm-s63702 kernel: [674417.184398] Total swap = 0kB Jan 18 07:33:57 kvm-s63702 kernel: [674417.184399] 7864218 pages RAM Jan 18 07:33:57 kvm-s63702 kernel: [674417.184400] 0 pages HighMem/MovableOnly Jan 18 07:33:57 kvm-s63702 kernel: [674417.184400] 143432 pages reserved Jan 18 07:33:57 kvm-s63702 kernel: [674417.184401] 0 pages cma reserved Jan 18 07:33:57 kvm-s63702 kernel: [674417.184402] 0 pages hwpoisoned Jan 18 07:33:57 kvm-s63702 kernel: [674417.184402] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 18 07:33:57 kvm-s63702 kernel: [674417.184418] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 18 07:33:57 kvm-s63702 kernel: [674417.184421] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 18 07:33:57 kvm-s63702 kernel: [674417.184424] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 18 07:33:57 kvm-s63702 kernel: [674417.184426] [ 724] 0 724 2559 865 8 0 0 dhclient Jan 18 07:33:57 kvm-s63702 kernel: [674417.184428] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 18 07:33:57 kvm-s63702 kernel: [674417.184430] [ 932] 101 932 65020 266 29 0 0 rsyslogd Jan 18 07:33:57 kvm-s63702 kernel: [674417.184432] [ 961] 102 961 9809 339 23 0 0 dbus-daemon Jan 18 07:33:57 kvm-s63702 kernel: [674417.184434] [ 964] 0 964 3956 41 12 0 0 getty Jan 18 07:33:57 kvm-s63702 kernel: [674417.184436] [ 967] 0 967 3956 40 13 0 0 getty Jan 18 07:33:57 kvm-s63702 kernel: [674417.184438] [ 972] 0 972 3956 40 13 0 0 getty Jan 18 07:33:57 kvm-s63702 kernel: [674417.184440] [ 973] 0 973 3956 39 13 0 0 getty Jan 18 07:33:57 kvm-s63702 kernel: [674417.184442] [ 975] 0 975 3956 39 13 0 0 getty Jan 18 07:33:57 kvm-s63702 kernel: [674417.184443] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 18 07:33:57 kvm-s63702 kernel: [674417.184445] [ 998] 0 998 1093 37 8 0 0 acpid Jan 18 07:33:57 kvm-s63702 kernel: [674417.184447] [ 1016] 0 1016 5915 238 18 0 0 cron Jan 18 07:33:57 kvm-s63702 kernel: [674417.184449] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 18 07:33:57 kvm-s63702 kernel: [674417.184451] [ 1046] 0 1046 10864 411 27 0 0 systemd-logind Jan 18 07:33:57 kvm-s63702 kernel: [674417.184453] [ 1052] 0 1052 4799 288 14 0 0 irqbalance Jan 18 07:33:57 kvm-s63702 kernel: [674417.184455] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 18 07:33:57 kvm-s63702 kernel: [674417.184459] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 18 07:33:57 kvm-s63702 kernel: [674417.184461] [ 5396] 1000 5396 352222 8814 93 0 0 beam.smp Jan 18 07:33:57 kvm-s63702 kernel: [674417.184462] [ 5432] 1000 5432 2884 1183 10 0 0 gosecrets Jan 18 07:33:57 kvm-s63702 kernel: [674417.184467] [ 5436] 1000 5436 718232 181989 683 0 0 beam.smp Jan 18 07:33:57 kvm-s63702 kernel: [674417.184469] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 18 07:33:57 kvm-s63702 kernel: [674417.184472] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 18 07:33:57 kvm-s63702 kernel: [674417.184474] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 18 07:33:57 kvm-s63702 kernel: [674417.184477] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 18 07:33:57 kvm-s63702 kernel: [674417.184480] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 18 07:33:57 kvm-s63702 kernel: [674417.184482] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 18 07:33:57 kvm-s63702 kernel: [674417.184485] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 18 07:33:57 kvm-s63702 kernel: [674417.184488] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 18 07:33:57 kvm-s63702 kernel: [674417.184490] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 18 07:33:57 kvm-s63702 kernel: [674417.184493] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 18 07:33:57 kvm-s63702 kernel: [674417.184495] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 18 07:33:57 kvm-s63702 kernel: [674417.184500] [14025] 1000 14025 1462 169 8 0 0 goport Jan 18 07:33:57 kvm-s63702 kernel: [674417.184502] [14029] 1000 14029 74588 1986 29 0 0 goxdcr Jan 18 07:33:57 kvm-s63702 kernel: [674417.184505] [14040] 1000 14040 278409 18332 140 0 0 beam.smp Jan 18 07:33:57 kvm-s63702 kernel: [674417.184507] [14076] 1000 14076 1113 361 7 0 0 sh Jan 18 07:33:57 kvm-s63702 kernel: [674417.184509] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 18 07:33:57 kvm-s63702 kernel: [674417.184512] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 18 07:33:57 kvm-s63702 kernel: [674417.184514] [14085] 1000 14085 1331 101 8 0 0 godu Jan 18 07:33:57 kvm-s63702 kernel: [674417.184517] [14086] 1000 14086 1113 320 7 0 0 sh Jan 18 07:33:57 kvm-s63702 kernel: [674417.184519] [14087] 1000 14087 1331 100 8 0 0 godu Jan 18 07:33:57 kvm-s63702 kernel: [674417.184520] [14098] 1000 14098 2156 418 10 0 0 sigar_port Jan 18 07:33:57 kvm-s63702 kernel: [674417.184522] [ 6503] 0 6503 4302 1233 15 0 0 atop Jan 18 07:33:57 kvm-s63702 kernel: [674417.184524] [ 9398] 1000 9398 1462 151 8 0 0 goport Jan 18 07:33:57 kvm-s63702 kernel: [674417.184526] [ 9403] 1000 9403 16259977 7322354 21681 0 0 indexer Jan 18 07:33:57 kvm-s63702 kernel: [674417.184528] Out of memory: Kill process 9403 (indexer) score 951 or sacrifice child Jan 18 07:33:57 kvm-s63702 kernel: [674417.187835] Killed process 9403 (indexer) total-vm:65039908kB, anon-rss:29288172kB, file-rss:1772kB Jan 18 10:47:58 kvm-s63702 kernel: [686058.239817] indexer invoked oom-killer: gfp_mask=0x2084d0, order=0, oom_score_adj=0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239825] indexer cpuset=/ mems_allowed=0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239847] CPU: 10 PID: 11238 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 18 10:47:58 kvm-s63702 kernel: [686058.239849] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239851] 0000000000000000 ffff880370b7b998 ffffffff817aed00 0000000000000000 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239855] ffff88005ccdebf0 ffff880370b7ba38 ffffffff817a9c3f ffff880370b7ba78 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239858] 0000000000000002 0000000000000000 0000000000000002 ffff880370b7ba88 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239861] Call Trace: Jan 18 10:47:58 kvm-s63702 kernel: [686058.239912] [] dump_stack+0x45/0x57 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239916] [] dump_header+0x7f/0x1f1 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239937] [] oom_kill_process+0x22b/0x390 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239940] [] out_of_memory+0x4d2/0x520 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239943] [] __alloc_pages_nodemask+0x940/0xa60 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239955] [] ? dequeue_task+0x5a/0x80 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239969] [] alloc_pages_current+0x91/0x100 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239981] [] pte_alloc_one+0x1b/0x50 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239984] [] __pte_alloc+0x27/0x180 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239987] [] ? schedule+0x29/0x70 Jan 18 10:47:58 kvm-s63702 kernel: [686058.239989] [] handle_mm_fault+0x1048/0x1100 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240010] [] ? call_rwsem_down_read_failed+0x14/0x30 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240040] [] __do_page_fault+0x1c4/0x5b0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240049] [] ? rwsem_wake+0x4a/0x60 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240052] [] ? call_rwsem_wake+0x17/0x30 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240057] [] ? up_write+0x25/0x30 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240061] [] ? vm_mmap_pgoff+0x99/0xc0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240063] [] trace_do_page_fault+0x45/0x100 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240070] [] do_async_page_fault+0x1e/0xd0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240079] [] async_page_fault+0x28/0x30 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240084] Mem-Info: Jan 18 10:47:58 kvm-s63702 kernel: [686058.240090] Node 0 DMA per-cpu: Jan 18 10:47:58 kvm-s63702 kernel: [686058.240097] CPU 0: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240101] CPU 1: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240104] CPU 2: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240108] CPU 3: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240113] CPU 4: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240117] CPU 5: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240122] CPU 6: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240125] CPU 7: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240129] CPU 8: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240131] CPU 9: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240132] CPU 10: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240134] CPU 11: hi: 0, btch: 1 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240134] Node 0 DMA32 per-cpu: Jan 18 10:47:58 kvm-s63702 kernel: [686058.240136] CPU 0: hi: 186, btch: 31 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240137] CPU 1: hi: 186, btch: 31 usd: 2 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240138] CPU 2: hi: 186, btch: 31 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240139] CPU 3: hi: 186, btch: 31 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240140] CPU 4: hi: 186, btch: 31 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240140] CPU 5: hi: 186, btch: 31 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240141] CPU 6: hi: 186, btch: 31 usd: 30 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240142] CPU 7: hi: 186, btch: 31 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240143] CPU 8: hi: 186, btch: 31 usd: 30 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240144] CPU 9: hi: 186, btch: 31 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240145] CPU 10: hi: 186, btch: 31 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240146] CPU 11: hi: 186, btch: 31 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240147] Node 0 Normal per-cpu: Jan 18 10:47:58 kvm-s63702 kernel: [686058.240148] CPU 0: hi: 186, btch: 31 usd: 4 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240149] CPU 1: hi: 186, btch: 31 usd: 16 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240150] CPU 2: hi: 186, btch: 31 usd: 42 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240151] CPU 3: hi: 186, btch: 31 usd: 9 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240152] CPU 4: hi: 186, btch: 31 usd: 77 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240153] CPU 5: hi: 186, btch: 31 usd: 33 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240154] CPU 6: hi: 186, btch: 31 usd: 12 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240155] CPU 7: hi: 186, btch: 31 usd: 60 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240155] CPU 8: hi: 186, btch: 31 usd: 50 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240156] CPU 9: hi: 186, btch: 31 usd: 1 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240157] CPU 10: hi: 186, btch: 31 usd: 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240158] CPU 11: hi: 186, btch: 31 usd: 41 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240161] active_anon:7591303 inactive_anon:72 isolated_anon:0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240161] active_file:340 inactive_file:286 isolated_file:0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240161] unevictable:1199 dirty:0 writeback:223 unstable:0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240161] free:48134 slab_reclaimable:6809 slab_unreclaimable:8765 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240161] mapped:1062 shmem:122 pagetables:22367 bounce:0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240161] free_cma:0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240164] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 18 10:47:58 kvm-s63702 kernel: [686058.240170] lowmem_reserve[]: 0 2978 30143 30143 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240172] Node 0 DMA32 free:115652kB min:6668kB low:8332kB high:10000kB active_anon:2909432kB inactive_anon:32kB active_file:144kB inactive_file:188kB unevictable:756kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:756kB dirty:0kB writeback:156kB mapped:924kB shmem:36kB slab_reclaimable:2124kB slab_unreclaimable:2816kB kernel_stack:2432kB pagetables:8260kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2100 all_unreclaimable? yes Jan 18 10:47:58 kvm-s63702 kernel: [686058.240176] lowmem_reserve[]: 0 0 27165 27165 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240178] Node 0 Normal free:60992kB min:60876kB low:76092kB high:91312kB active_anon:27455780kB inactive_anon:256kB active_file:1216kB inactive_file:956kB unevictable:4040kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:4040kB dirty:0kB writeback:736kB mapped:3324kB shmem:452kB slab_reclaimable:25112kB slab_unreclaimable:32228kB kernel_stack:20944kB pagetables:81208kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:15700 all_unreclaimable? yes Jan 18 10:47:58 kvm-s63702 kernel: [686058.240184] lowmem_reserve[]: 0 0 0 0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240186] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 18 10:47:58 kvm-s63702 kernel: [686058.240196] Node 0 DMA32: 123*4kB (UEM) 80*8kB (UEM) 116*16kB (UEM) 109*32kB (UEM) 84*64kB (UEM) 29*128kB (UE) 20*256kB (UEM) 14*512kB (UE) 4*1024kB (UE) 5*2048kB (UM) 18*4096kB (UMR) = 115916kB Jan 18 10:47:58 kvm-s63702 kernel: [686058.240205] Node 0 Normal: 365*4kB (EM) 298*8kB (UEM) 374*16kB (UEM) 238*32kB (UEM) 186*64kB (UEM) 90*128kB (E) 37*256kB (E) 12*512kB (UEM) 0*1024kB 0*2048kB 1*4096kB (R) = 60580kB Jan 18 10:47:58 kvm-s63702 kernel: [686058.240215] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 18 10:47:58 kvm-s63702 kernel: [686058.240216] 1925 total pagecache pages Jan 18 10:47:58 kvm-s63702 kernel: [686058.240217] 0 pages in swap cache Jan 18 10:47:58 kvm-s63702 kernel: [686058.240218] Swap cache stats: add 0, delete 0, find 0/0 Jan 18 10:47:58 kvm-s63702 kernel: [686058.240219] Free swap = 0kB Jan 18 10:47:58 kvm-s63702 kernel: [686058.240220] Total swap = 0kB Jan 18 10:47:58 kvm-s63702 kernel: [686058.240221] 7864218 pages RAM Jan 18 10:47:58 kvm-s63702 kernel: [686058.240221] 0 pages HighMem/MovableOnly Jan 18 10:47:58 kvm-s63702 kernel: [686058.240222] 143432 pages reserved Jan 18 10:47:58 kvm-s63702 kernel: [686058.240223] 0 pages cma reserved Jan 18 10:47:58 kvm-s63702 kernel: [686058.240224] 0 pages hwpoisoned Jan 18 10:47:58 kvm-s63702 kernel: [686058.240224] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 18 10:47:58 kvm-s63702 kernel: [686058.240238] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 18 10:47:58 kvm-s63702 kernel: [686058.240241] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 18 10:47:58 kvm-s63702 kernel: [686058.240244] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 18 10:47:58 kvm-s63702 kernel: [686058.240246] [ 724] 0 724 2559 865 8 0 0 dhclient Jan 18 10:47:58 kvm-s63702 kernel: [686058.240248] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 18 10:47:58 kvm-s63702 kernel: [686058.240251] [ 932] 101 932 65020 266 29 0 0 rsyslogd Jan 18 10:47:58 kvm-s63702 kernel: [686058.240253] [ 961] 102 961 9809 339 23 0 0 dbus-daemon Jan 18 10:47:58 kvm-s63702 kernel: [686058.240255] [ 964] 0 964 3956 41 12 0 0 getty Jan 18 10:47:58 kvm-s63702 kernel: [686058.240257] [ 967] 0 967 3956 40 13 0 0 getty Jan 18 10:47:58 kvm-s63702 kernel: [686058.240259] [ 972] 0 972 3956 40 13 0 0 getty Jan 18 10:47:58 kvm-s63702 kernel: [686058.240260] [ 973] 0 973 3956 39 13 0 0 getty Jan 18 10:47:58 kvm-s63702 kernel: [686058.240262] [ 975] 0 975 3956 39 13 0 0 getty Jan 18 10:47:58 kvm-s63702 kernel: [686058.240264] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 18 10:47:58 kvm-s63702 kernel: [686058.240266] [ 998] 0 998 1093 37 8 0 0 acpid Jan 18 10:47:58 kvm-s63702 kernel: [686058.240268] [ 1016] 0 1016 5915 238 18 0 0 cron Jan 18 10:47:58 kvm-s63702 kernel: [686058.240271] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 18 10:47:58 kvm-s63702 kernel: [686058.240273] [ 1046] 0 1046 10864 411 27 0 0 systemd-logind Jan 18 10:47:58 kvm-s63702 kernel: [686058.240274] [ 1052] 0 1052 4799 288 14 0 0 irqbalance Jan 18 10:47:58 kvm-s63702 kernel: [686058.240277] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 18 10:47:58 kvm-s63702 kernel: [686058.240281] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 18 10:47:58 kvm-s63702 kernel: [686058.240284] [ 5396] 1000 5396 351940 8387 89 0 0 beam.smp Jan 18 10:47:58 kvm-s63702 kernel: [686058.240286] [ 5432] 1000 5432 2884 1206 10 0 0 gosecrets Jan 18 10:47:58 kvm-s63702 kernel: [686058.240289] [ 5436] 1000 5436 725028 190795 698 0 0 beam.smp Jan 18 10:47:58 kvm-s63702 kernel: [686058.240292] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 18 10:47:58 kvm-s63702 kernel: [686058.240294] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 18 10:47:58 kvm-s63702 kernel: [686058.240297] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 18 10:47:58 kvm-s63702 kernel: [686058.240299] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 18 10:47:58 kvm-s63702 kernel: [686058.240302] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 18 10:47:58 kvm-s63702 kernel: [686058.240309] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 18 10:47:58 kvm-s63702 kernel: [686058.240312] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 18 10:47:58 kvm-s63702 kernel: [686058.240315] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 18 10:47:58 kvm-s63702 kernel: [686058.240318] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 18 10:47:58 kvm-s63702 kernel: [686058.240321] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 18 10:47:58 kvm-s63702 kernel: [686058.240323] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 18 10:47:58 kvm-s63702 kernel: [686058.240327] [14025] 1000 14025 1462 169 8 0 0 goport Jan 18 10:47:58 kvm-s63702 kernel: [686058.240330] [14029] 1000 14029 74588 1961 29 0 0 goxdcr Jan 18 10:47:58 kvm-s63702 kernel: [686058.240333] [14040] 1000 14040 278696 17468 136 0 0 beam.smp Jan 18 10:47:58 kvm-s63702 kernel: [686058.240335] [14076] 1000 14076 1113 361 7 0 0 sh Jan 18 10:47:58 kvm-s63702 kernel: [686058.240338] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 18 10:47:58 kvm-s63702 kernel: [686058.240340] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 18 10:47:58 kvm-s63702 kernel: [686058.240342] [14085] 1000 14085 1331 101 8 0 0 godu Jan 18 10:47:58 kvm-s63702 kernel: [686058.240345] [14086] 1000 14086 1113 320 7 0 0 sh Jan 18 10:47:58 kvm-s63702 kernel: [686058.240348] [14087] 1000 14087 1331 100 8 0 0 godu Jan 18 10:47:58 kvm-s63702 kernel: [686058.240350] [14098] 1000 14098 2156 418 10 0 0 sigar_port Jan 18 10:47:58 kvm-s63702 kernel: [686058.240353] [ 6503] 0 6503 4270 1201 15 0 0 atop Jan 18 10:47:58 kvm-s63702 kernel: [686058.240355] [11102] 1000 11102 1462 159 8 0 0 goport Jan 18 10:47:58 kvm-s63702 kernel: [686058.240357] [11107] 1000 11107 15463058 7319029 20786 0 0 indexer Jan 18 10:47:58 kvm-s63702 kernel: [686058.240358] Out of memory: Kill process 12451 (indexer) score 950 or sacrifice child Jan 18 10:47:58 kvm-s63702 kernel: [686058.246980] Killed process 11107 (indexer) total-vm:61852232kB, anon-rss:29274644kB, file-rss:1472kB Jan 18 13:46:05 kvm-s63702 kernel: [696745.437208] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437218] indexer cpuset=/ mems_allowed=0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437240] CPU: 4 PID: 15237 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 18 13:46:05 kvm-s63702 kernel: [696745.437242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437248] 0000000000000000 ffff8806232839d8 ffffffff817aed00 0000000000000000 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437251] ffff88060b22e220 ffff880623283a78 ffffffff817a9c3f 0000000000000000 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437254] 0000000000000000 ffff880623283ab8 0000000000000000 0000000000000000 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437257] Call Trace: Jan 18 13:46:05 kvm-s63702 kernel: [696745.437311] [] dump_stack+0x45/0x57 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437316] [] dump_header+0x7f/0x1f1 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437339] [] oom_kill_process+0x22b/0x390 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437343] [] out_of_memory+0x4d2/0x520 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437347] [] __alloc_pages_nodemask+0x940/0xa60 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437362] [] alloc_pages_vma+0x97/0x150 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437370] [] handle_mm_fault+0xd13/0x1100 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437387] [] __do_page_fault+0x1c4/0x5b0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437401] [] ? __enqueue_entity+0x78/0x80 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437412] [] ? do_futex+0x10a/0x5d0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437416] [] ? pick_next_task_fair+0x61b/0x880 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437429] [] ? __switch_to+0x15f/0x570 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437432] [] ? sched_clock_cpu+0x98/0xc0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437435] [] trace_do_page_fault+0x45/0x100 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437440] [] do_async_page_fault+0x1e/0xd0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437452] [] async_page_fault+0x28/0x30 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437455] Mem-Info: Jan 18 13:46:05 kvm-s63702 kernel: [696745.437460] Node 0 DMA per-cpu: Jan 18 13:46:05 kvm-s63702 kernel: [696745.437469] CPU 0: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437472] CPU 1: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437473] CPU 2: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437475] CPU 3: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437477] CPU 4: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437478] CPU 5: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437480] CPU 6: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437485] CPU 7: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437487] CPU 8: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437489] CPU 9: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437490] CPU 10: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437492] CPU 11: hi: 0, btch: 1 usd: 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437493] Node 0 DMA32 per-cpu: Jan 18 13:46:05 kvm-s63702 kernel: [696745.437496] CPU 0: hi: 186, btch: 31 usd: 33 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437497] CPU 1: hi: 186, btch: 31 usd: 19 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437499] CPU 2: hi: 186, btch: 31 usd: 22 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437500] CPU 3: hi: 186, btch: 31 usd: 52 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437502] CPU 4: hi: 186, btch: 31 usd: 36 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437503] CPU 5: hi: 186, btch: 31 usd: 23 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437505] CPU 6: hi: 186, btch: 31 usd: 15 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437506] CPU 7: hi: 186, btch: 31 usd: 98 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437508] CPU 8: hi: 186, btch: 31 usd: 49 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437509] CPU 9: hi: 186, btch: 31 usd: 40 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437511] CPU 10: hi: 186, btch: 31 usd: 54 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437512] CPU 11: hi: 186, btch: 31 usd: 30 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437513] Node 0 Normal per-cpu: Jan 18 13:46:05 kvm-s63702 kernel: [696745.437515] CPU 0: hi: 186, btch: 31 usd: 31 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437517] CPU 1: hi: 186, btch: 31 usd: 27 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437518] CPU 2: hi: 186, btch: 31 usd: 22 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437520] CPU 3: hi: 186, btch: 31 usd: 172 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437522] CPU 4: hi: 186, btch: 31 usd: 57 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437523] CPU 5: hi: 186, btch: 31 usd: 22 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437524] CPU 6: hi: 186, btch: 31 usd: 35 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437526] CPU 7: hi: 186, btch: 31 usd: 136 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437527] CPU 8: hi: 186, btch: 31 usd: 37 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437528] CPU 9: hi: 186, btch: 31 usd: 35 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437530] CPU 10: hi: 186, btch: 31 usd: 39 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437531] CPU 11: hi: 186, btch: 31 usd: 31 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437535] active_anon:7588290 inactive_anon:73 isolated_anon:0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437535] active_file:199 inactive_file:0 isolated_file:0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437535] unevictable:1387 dirty:98 writeback:0 unstable:0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437535] free:48145 slab_reclaimable:6774 slab_unreclaimable:8910 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437535] mapped:918 shmem:141 pagetables:23198 bounce:0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437535] free_cma:0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437539] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 18 13:46:05 kvm-s63702 kernel: [696745.437550] lowmem_reserve[]: 0 2978 30143 30143 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437553] Node 0 DMA32 free:115240kB min:6668kB low:8332kB high:10000kB active_anon:2907468kB inactive_anon:32kB active_file:0kB inactive_file:0kB unevictable:1104kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:1104kB dirty:0kB writeback:0kB mapped:676kB shmem:44kB slab_reclaimable:1984kB slab_unreclaimable:2828kB kernel_stack:1952kB pagetables:8120kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4908 all_unreclaimable? yes Jan 18 13:46:05 kvm-s63702 kernel: [696745.437558] lowmem_reserve[]: 0 0 27165 27165 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437561] Node 0 Normal free:61448kB min:60876kB low:76092kB high:91312kB active_anon:27445692kB inactive_anon:260kB active_file:840kB inactive_file:0kB unevictable:4444kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:4444kB dirty:392kB writeback:0kB mapped:2996kB shmem:520kB slab_reclaimable:25112kB slab_unreclaimable:32796kB kernel_stack:21344kB pagetables:84672kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6496 all_unreclaimable? yes Jan 18 13:46:05 kvm-s63702 kernel: [696745.437566] lowmem_reserve[]: 0 0 0 0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437569] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 18 13:46:05 kvm-s63702 kernel: [696745.437587] Node 0 DMA32: 69*4kB (UE) 67*8kB (UE) 95*16kB (UEM) 79*32kB (EM) 45*64kB (EM) 31*128kB (UE) 22*256kB (UE) 16*512kB (UEM) 5*1024kB (UEM) 5*2048kB (UM) 18*4096kB (UMR) = 114620kB Jan 18 13:46:05 kvm-s63702 kernel: [696745.437600] Node 0 Normal: 388*4kB (UEM) 287*8kB (EM) 314*16kB (UEM) 274*32kB (UEM) 168*64kB (UEM) 119*128kB (EM) 36*256kB (UE) 8*512kB (EM) 0*1024kB 0*2048kB 1*4096kB (R) = 61032kB Jan 18 13:46:05 kvm-s63702 kernel: [696745.437617] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 18 13:46:05 kvm-s63702 kernel: [696745.437618] 1495 total pagecache pages Jan 18 13:46:05 kvm-s63702 kernel: [696745.437621] 0 pages in swap cache Jan 18 13:46:05 kvm-s63702 kernel: [696745.437622] Swap cache stats: add 0, delete 0, find 0/0 Jan 18 13:46:05 kvm-s63702 kernel: [696745.437623] Free swap = 0kB Jan 18 13:46:05 kvm-s63702 kernel: [696745.437624] Total swap = 0kB Jan 18 13:46:05 kvm-s63702 kernel: [696745.437626] 7864218 pages RAM Jan 18 13:46:05 kvm-s63702 kernel: [696745.437627] 0 pages HighMem/MovableOnly Jan 18 13:46:05 kvm-s63702 kernel: [696745.437628] 143432 pages reserved Jan 18 13:46:05 kvm-s63702 kernel: [696745.437629] 0 pages cma reserved Jan 18 13:46:05 kvm-s63702 kernel: [696745.437630] 0 pages hwpoisoned Jan 18 13:46:05 kvm-s63702 kernel: [696745.437631] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 18 13:46:05 kvm-s63702 kernel: [696745.437653] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 18 13:46:05 kvm-s63702 kernel: [696745.437657] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 18 13:46:05 kvm-s63702 kernel: [696745.437661] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 18 13:46:05 kvm-s63702 kernel: [696745.437664] [ 724] 0 724 2559 865 8 0 0 dhclient Jan 18 13:46:05 kvm-s63702 kernel: [696745.437667] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 18 13:46:05 kvm-s63702 kernel: [696745.437670] [ 932] 101 932 65020 266 29 0 0 rsyslogd Jan 18 13:46:05 kvm-s63702 kernel: [696745.437673] [ 961] 102 961 9809 335 23 0 0 dbus-daemon Jan 18 13:46:05 kvm-s63702 kernel: [696745.437676] [ 964] 0 964 3956 41 12 0 0 getty Jan 18 13:46:05 kvm-s63702 kernel: [696745.437679] [ 967] 0 967 3956 40 13 0 0 getty Jan 18 13:46:05 kvm-s63702 kernel: [696745.437682] [ 972] 0 972 3956 40 13 0 0 getty Jan 18 13:46:05 kvm-s63702 kernel: [696745.437684] [ 973] 0 973 3956 39 13 0 0 getty Jan 18 13:46:05 kvm-s63702 kernel: [696745.437687] [ 975] 0 975 3956 39 13 0 0 getty Jan 18 13:46:05 kvm-s63702 kernel: [696745.437691] [ 995] 0 995 15343 269 33 0 -1000 sshd Jan 18 13:46:05 kvm-s63702 kernel: [696745.437694] [ 998] 0 998 1093 37 8 0 0 acpid Jan 18 13:46:05 kvm-s63702 kernel: [696745.437698] [ 1016] 0 1016 5915 238 18 0 0 cron Jan 18 13:46:05 kvm-s63702 kernel: [696745.437701] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 18 13:46:05 kvm-s63702 kernel: [696745.437704] [ 1046] 0 1046 10864 409 27 0 0 systemd-logind Jan 18 13:46:05 kvm-s63702 kernel: [696745.437707] [ 1052] 0 1052 4799 284 14 0 0 irqbalance Jan 18 13:46:05 kvm-s63702 kernel: [696745.437710] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 18 13:46:05 kvm-s63702 kernel: [696745.437717] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 18 13:46:05 kvm-s63702 kernel: [696745.437720] [ 5396] 1000 5396 352242 8158 91 0 0 beam.smp Jan 18 13:46:05 kvm-s63702 kernel: [696745.437723] [ 5432] 1000 5432 2884 1190 10 0 0 gosecrets Jan 18 13:46:05 kvm-s63702 kernel: [696745.437726] [ 5436] 1000 5436 765535 234746 788 0 0 beam.smp Jan 18 13:46:05 kvm-s63702 kernel: [696745.437729] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 18 13:46:05 kvm-s63702 kernel: [696745.437732] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 18 13:46:05 kvm-s63702 kernel: [696745.437735] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 18 13:46:05 kvm-s63702 kernel: [696745.437738] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 18 13:46:05 kvm-s63702 kernel: [696745.437741] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 18 13:46:05 kvm-s63702 kernel: [696745.437743] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 18 13:46:05 kvm-s63702 kernel: [696745.437746] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 18 13:46:05 kvm-s63702 kernel: [696745.437749] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 18 13:46:05 kvm-s63702 kernel: [696745.437753] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 18 13:46:05 kvm-s63702 kernel: [696745.437756] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 18 13:46:05 kvm-s63702 kernel: [696745.437758] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 18 13:46:05 kvm-s63702 kernel: [696745.437763] [14025] 1000 14025 1462 169 8 0 0 goport Jan 18 13:46:05 kvm-s63702 kernel: [696745.437766] [14029] 1000 14029 74588 2015 29 0 0 goxdcr Jan 18 13:46:05 kvm-s63702 kernel: [696745.437775] [14040] 1000 14040 278159 18643 142 0 0 beam.smp Jan 18 13:46:05 kvm-s63702 kernel: [696745.437778] [14076] 1000 14076 1113 361 7 0 0 sh Jan 18 13:46:05 kvm-s63702 kernel: [696745.437781] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 18 13:46:05 kvm-s63702 kernel: [696745.437784] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 18 13:46:05 kvm-s63702 kernel: [696745.437786] [14085] 1000 14085 1331 101 8 0 0 godu Jan 18 13:46:05 kvm-s63702 kernel: [696745.437789] [14086] 1000 14086 1113 320 7 0 0 sh Jan 18 13:46:05 kvm-s63702 kernel: [696745.437792] [14087] 1000 14087 1331 100 8 0 0 godu Jan 18 13:46:05 kvm-s63702 kernel: [696745.437795] [14098] 1000 14098 2156 415 10 0 0 sigar_port Jan 18 13:46:05 kvm-s63702 kernel: [696745.437798] [ 6503] 0 6503 4458 1389 15 0 0 atop Jan 18 13:46:05 kvm-s63702 kernel: [696745.437801] [14264] 1000 14264 1526 163 9 0 0 goport Jan 18 13:46:05 kvm-s63702 kernel: [696745.437804] [14269] 1000 14269 16141310 7274598 21428 0 0 indexer Jan 18 13:46:05 kvm-s63702 kernel: [696745.437806] Out of memory: Kill process 14269 (indexer) score 944 or sacrifice child Jan 18 13:46:05 kvm-s63702 kernel: [696745.442381] Killed process 14269 (indexer) total-vm:64565240kB, anon-rss:29098872kB, file-rss:0kB Jan 18 17:01:16 kvm-s63702 kernel: [708456.318990] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.318996] indexer cpuset=/ mems_allowed=0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319019] CPU: 4 PID: 18745 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 18 17:01:16 kvm-s63702 kernel: [708456.319021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319023] 0000000000000000 ffff88079c08f9d8 ffffffff817aed00 0000000000000000 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319027] ffff8804c8f12740 ffff88079c08fa78 ffffffff817a9c3f 0000000000000000 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319030] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319033] Call Trace: Jan 18 17:01:16 kvm-s63702 kernel: [708456.319089] [] dump_stack+0x45/0x57 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319093] [] dump_header+0x7f/0x1f1 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319111] [] oom_kill_process+0x22b/0x390 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319113] [] out_of_memory+0x4d2/0x520 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319117] [] __alloc_pages_nodemask+0x940/0xa60 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319127] [] alloc_pages_vma+0x97/0x150 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319132] [] handle_mm_fault+0xd13/0x1100 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319145] [] __do_page_fault+0x1c4/0x5b0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319155] [] ? __enqueue_entity+0x78/0x80 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319158] [] ? set_next_entity+0x9d/0xb0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319160] [] ? pick_next_task_fair+0x61b/0x880 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319171] [] ? __switch_to+0x15f/0x570 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319174] [] ? sched_clock_cpu+0x98/0xc0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319177] [] trace_do_page_fault+0x45/0x100 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319182] [] do_async_page_fault+0x1e/0xd0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319194] [] async_page_fault+0x28/0x30 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319196] Mem-Info: Jan 18 17:01:16 kvm-s63702 kernel: [708456.319202] Node 0 DMA per-cpu: Jan 18 17:01:16 kvm-s63702 kernel: [708456.319209] CPU 0: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319210] CPU 1: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319212] CPU 2: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319213] CPU 3: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319214] CPU 4: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319215] CPU 5: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319216] CPU 6: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319217] CPU 7: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319218] CPU 8: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319219] CPU 9: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319220] CPU 10: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319222] CPU 11: hi: 0, btch: 1 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319223] Node 0 DMA32 per-cpu: Jan 18 17:01:16 kvm-s63702 kernel: [708456.319225] CPU 0: hi: 186, btch: 31 usd: 20 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319226] CPU 1: hi: 186, btch: 31 usd: 12 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319228] CPU 2: hi: 186, btch: 31 usd: 23 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319229] CPU 3: hi: 186, btch: 31 usd: 30 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319231] CPU 4: hi: 186, btch: 31 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319232] CPU 5: hi: 186, btch: 31 usd: 21 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319234] CPU 6: hi: 186, btch: 31 usd: 26 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319235] CPU 7: hi: 186, btch: 31 usd: 25 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319241] CPU 8: hi: 186, btch: 31 usd: 20 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319243] CPU 9: hi: 186, btch: 31 usd: 4 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319244] CPU 10: hi: 186, btch: 31 usd: 27 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319245] CPU 11: hi: 186, btch: 31 usd: 18 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319246] Node 0 Normal per-cpu: Jan 18 17:01:16 kvm-s63702 kernel: [708456.319247] CPU 0: hi: 186, btch: 31 usd: 9 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319248] CPU 1: hi: 186, btch: 31 usd: 9 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319250] CPU 2: hi: 186, btch: 31 usd: 25 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319251] CPU 3: hi: 186, btch: 31 usd: 8 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319253] CPU 4: hi: 186, btch: 31 usd: 30 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319254] CPU 5: hi: 186, btch: 31 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319256] CPU 6: hi: 186, btch: 31 usd: 12 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319257] CPU 7: hi: 186, btch: 31 usd: 28 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319259] CPU 8: hi: 186, btch: 31 usd: 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319261] CPU 9: hi: 186, btch: 31 usd: 30 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319262] CPU 10: hi: 186, btch: 31 usd: 14 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319264] CPU 11: hi: 186, btch: 31 usd: 23 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319269] active_anon:7587923 inactive_anon:73 isolated_anon:0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319269] active_file:39 inactive_file:68 isolated_file:0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319269] unevictable:1394 dirty:10 writeback:0 unstable:0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319269] free:48098 slab_reclaimable:7550 slab_unreclaimable:8666 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319269] mapped:846 shmem:161 pagetables:21951 bounce:0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319269] free_cma:0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319274] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 18 17:01:16 kvm-s63702 kernel: [708456.319280] lowmem_reserve[]: 0 2978 30143 30143 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319284] Node 0 DMA32 free:115768kB min:6668kB low:8332kB high:10000kB active_anon:2908420kB inactive_anon:32kB active_file:16kB inactive_file:0kB unevictable:756kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:756kB dirty:0kB writeback:0kB mapped:848kB shmem:52kB slab_reclaimable:2532kB slab_unreclaimable:3036kB kernel_stack:2320kB pagetables:8156kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7804 all_unreclaimable? yes Jan 18 17:01:16 kvm-s63702 kernel: [708456.319291] lowmem_reserve[]: 0 0 27165 27165 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319294] Node 0 Normal free:60732kB min:60876kB low:76092kB high:91312kB active_anon:27443272kB inactive_anon:260kB active_file:140kB inactive_file:400kB unevictable:4820kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:4820kB dirty:40kB writeback:0kB mapped:2536kB shmem:592kB slab_reclaimable:27668kB slab_unreclaimable:31612kB kernel_stack:21232kB pagetables:79648kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:22524 all_unreclaimable? yes Jan 18 17:01:16 kvm-s63702 kernel: [708456.319300] lowmem_reserve[]: 0 0 0 0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319303] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 18 17:01:16 kvm-s63702 kernel: [708456.319315] Node 0 DMA32: 103*4kB (UEM) 87*8kB (UE) 135*16kB (UEM) 119*32kB (UEM) 91*64kB (UEM) 40*128kB (UEM) 15*256kB (UE) 10*512kB (UEM) 4*1024kB (UE) 5*2048kB (UM) 18*4096kB (UMR) = 115044kB Jan 18 17:01:16 kvm-s63702 kernel: [708456.319324] Node 0 Normal: 342*4kB (UEM) 225*8kB (UEM) 349*16kB (UE) 259*32kB (UE) 196*64kB (EM) 131*128kB (EM) 35*256kB (E) 2*512kB (E) 0*1024kB 0*2048kB 1*4096kB (R) = 60432kB Jan 18 17:01:16 kvm-s63702 kernel: [708456.319334] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 18 17:01:16 kvm-s63702 kernel: [708456.319335] 1220 total pagecache pages Jan 18 17:01:16 kvm-s63702 kernel: [708456.319337] 0 pages in swap cache Jan 18 17:01:16 kvm-s63702 kernel: [708456.319338] Swap cache stats: add 0, delete 0, find 0/0 Jan 18 17:01:16 kvm-s63702 kernel: [708456.319339] Free swap = 0kB Jan 18 17:01:16 kvm-s63702 kernel: [708456.319340] Total swap = 0kB Jan 18 17:01:16 kvm-s63702 kernel: [708456.319340] 7864218 pages RAM Jan 18 17:01:16 kvm-s63702 kernel: [708456.319341] 0 pages HighMem/MovableOnly Jan 18 17:01:16 kvm-s63702 kernel: [708456.319342] 143432 pages reserved Jan 18 17:01:16 kvm-s63702 kernel: [708456.319343] 0 pages cma reserved Jan 18 17:01:16 kvm-s63702 kernel: [708456.319343] 0 pages hwpoisoned Jan 18 17:01:16 kvm-s63702 kernel: [708456.319344] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 18 17:01:16 kvm-s63702 kernel: [708456.319363] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 18 17:01:16 kvm-s63702 kernel: [708456.319367] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 18 17:01:16 kvm-s63702 kernel: [708456.319375] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 18 17:01:16 kvm-s63702 kernel: [708456.319379] [ 724] 0 724 2559 699 8 0 0 dhclient Jan 18 17:01:16 kvm-s63702 kernel: [708456.319382] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 18 17:01:16 kvm-s63702 kernel: [708456.319385] [ 932] 101 932 65020 266 29 0 0 rsyslogd Jan 18 17:01:16 kvm-s63702 kernel: [708456.319389] [ 961] 102 961 9809 335 23 0 0 dbus-daemon Jan 18 17:01:16 kvm-s63702 kernel: [708456.319392] [ 964] 0 964 3956 41 12 0 0 getty Jan 18 17:01:16 kvm-s63702 kernel: [708456.319394] [ 967] 0 967 3956 40 13 0 0 getty Jan 18 17:01:16 kvm-s63702 kernel: [708456.319397] [ 972] 0 972 3956 40 13 0 0 getty Jan 18 17:01:16 kvm-s63702 kernel: [708456.319400] [ 973] 0 973 3956 39 13 0 0 getty Jan 18 17:01:16 kvm-s63702 kernel: [708456.319402] [ 975] 0 975 3956 39 13 0 0 getty Jan 18 17:01:16 kvm-s63702 kernel: [708456.319405] [ 995] 0 995 15343 170 33 0 -1000 sshd Jan 18 17:01:16 kvm-s63702 kernel: [708456.319408] [ 998] 0 998 1093 37 8 0 0 acpid Jan 18 17:01:16 kvm-s63702 kernel: [708456.319411] [ 1016] 0 1016 5915 228 18 0 0 cron Jan 18 17:01:16 kvm-s63702 kernel: [708456.319413] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 18 17:01:16 kvm-s63702 kernel: [708456.319416] [ 1046] 0 1046 10864 409 27 0 0 systemd-logind Jan 18 17:01:16 kvm-s63702 kernel: [708456.319419] [ 1052] 0 1052 4799 284 14 0 0 irqbalance Jan 18 17:01:16 kvm-s63702 kernel: [708456.319422] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 18 17:01:16 kvm-s63702 kernel: [708456.319428] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 18 17:01:16 kvm-s63702 kernel: [708456.319431] [ 5396] 1000 5396 351793 8187 88 0 0 beam.smp Jan 18 17:01:16 kvm-s63702 kernel: [708456.319433] [ 5432] 1000 5432 2884 1020 10 0 0 gosecrets Jan 18 17:01:16 kvm-s63702 kernel: [708456.319437] [ 5436] 1000 5436 740323 206405 732 0 0 beam.smp Jan 18 17:01:16 kvm-s63702 kernel: [708456.319439] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 18 17:01:16 kvm-s63702 kernel: [708456.319442] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 18 17:01:16 kvm-s63702 kernel: [708456.319445] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 18 17:01:16 kvm-s63702 kernel: [708456.319448] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 18 17:01:16 kvm-s63702 kernel: [708456.319451] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 18 17:01:16 kvm-s63702 kernel: [708456.319452] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 18 17:01:16 kvm-s63702 kernel: [708456.319454] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 18 17:01:16 kvm-s63702 kernel: [708456.319456] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 18 17:01:16 kvm-s63702 kernel: [708456.319458] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 18 17:01:16 kvm-s63702 kernel: [708456.319459] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 18 17:01:16 kvm-s63702 kernel: [708456.319461] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 18 17:01:16 kvm-s63702 kernel: [708456.319464] [14025] 1000 14025 1462 169 8 0 0 goport Jan 18 17:01:16 kvm-s63702 kernel: [708456.319466] [14029] 1000 14029 74588 1890 29 0 0 goxdcr Jan 18 17:01:16 kvm-s63702 kernel: [708456.319468] [14040] 1000 14040 282029 21652 153 0 0 beam.smp Jan 18 17:01:16 kvm-s63702 kernel: [708456.319469] [14076] 1000 14076 1113 336 7 0 0 sh Jan 18 17:01:16 kvm-s63702 kernel: [708456.319471] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 18 17:01:16 kvm-s63702 kernel: [708456.319472] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 18 17:01:16 kvm-s63702 kernel: [708456.319474] [14085] 1000 14085 1331 101 8 0 0 godu Jan 18 17:01:16 kvm-s63702 kernel: [708456.319475] [14086] 1000 14086 1113 320 7 0 0 sh Jan 18 17:01:16 kvm-s63702 kernel: [708456.319477] [14087] 1000 14087 1331 100 8 0 0 godu Jan 18 17:01:16 kvm-s63702 kernel: [708456.319479] [14098] 1000 14098 2156 418 10 0 0 sigar_port Jan 18 17:01:16 kvm-s63702 kernel: [708456.319482] [ 6503] 0 6503 4465 1396 15 0 0 atop Jan 18 17:01:16 kvm-s63702 kernel: [708456.319490] [17961] 1000 17961 1526 163 8 0 0 goport Jan 18 17:01:16 kvm-s63702 kernel: [708456.319493] [17966] 1000 17966 15227471 7298890 20233 0 0 indexer Jan 18 17:01:16 kvm-s63702 kernel: [708456.319495] Out of memory: Kill process 20760 (indexer) score 947 or sacrifice child Jan 18 17:01:16 kvm-s63702 kernel: [708456.323941] Killed process 17966 (indexer) total-vm:60909884kB, anon-rss:29194880kB, file-rss:1168kB Jan 18 20:08:04 kvm-s63702 kernel: [719664.026840] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026846] indexer cpuset=/ mems_allowed=0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026861] CPU: 4 PID: 22228 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 18 20:08:04 kvm-s63702 kernel: [719664.026863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026864] 0000000000000000 ffff88060b19b9d8 ffffffff817aed00 0000000000000000 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026867] ffff880623064e80 ffff88060b19ba78 ffffffff817a9c3f 0000000000000000 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026869] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026871] Call Trace: Jan 18 20:08:04 kvm-s63702 kernel: [719664.026910] [] dump_stack+0x45/0x57 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026913] [] dump_header+0x7f/0x1f1 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026929] [] oom_kill_process+0x22b/0x390 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026932] [] out_of_memory+0x4d2/0x520 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026935] [] __alloc_pages_nodemask+0x940/0xa60 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026944] [] alloc_pages_vma+0x97/0x150 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026950] [] handle_mm_fault+0xd13/0x1100 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026961] [] ? x2apic_send_IPI_mask+0x13/0x20 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026969] [] ? try_to_wake_up+0x1f1/0x340 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026985] [] ? call_rwsem_down_read_failed+0x14/0x30 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026992] [] __do_page_fault+0x1c4/0x5b0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026997] [] ? __enqueue_entity+0x78/0x80 Jan 18 20:08:04 kvm-s63702 kernel: [719664.026999] [] ? set_next_entity+0x9d/0xb0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027002] [] ? pick_next_task_fair+0x61b/0x880 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027011] [] ? __switch_to+0x15f/0x570 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027013] [] ? sched_clock_cpu+0x98/0xc0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027015] [] trace_do_page_fault+0x45/0x100 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027018] [] do_async_page_fault+0x1e/0xd0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027027] [] async_page_fault+0x28/0x30 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027028] Mem-Info: Jan 18 20:08:04 kvm-s63702 kernel: [719664.027032] Node 0 DMA per-cpu: Jan 18 20:08:04 kvm-s63702 kernel: [719664.027036] CPU 0: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027037] CPU 1: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027039] CPU 2: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027040] CPU 3: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027041] CPU 4: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027042] CPU 5: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027043] CPU 6: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027044] CPU 7: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027045] CPU 8: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027046] CPU 9: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027047] CPU 10: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027048] CPU 11: hi: 0, btch: 1 usd: 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027048] Node 0 DMA32 per-cpu: Jan 18 20:08:04 kvm-s63702 kernel: [719664.027050] CPU 0: hi: 186, btch: 31 usd: 21 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027051] CPU 1: hi: 186, btch: 31 usd: 28 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027052] CPU 2: hi: 186, btch: 31 usd: 19 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027053] CPU 3: hi: 186, btch: 31 usd: 59 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027054] CPU 4: hi: 186, btch: 31 usd: 27 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027055] CPU 5: hi: 186, btch: 31 usd: 31 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027055] CPU 6: hi: 186, btch: 31 usd: 39 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027056] CPU 7: hi: 186, btch: 31 usd: 158 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027057] CPU 8: hi: 186, btch: 31 usd: 38 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027058] CPU 9: hi: 186, btch: 31 usd: 29 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027059] CPU 10: hi: 186, btch: 31 usd: 35 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027060] CPU 11: hi: 186, btch: 31 usd: 51 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027061] Node 0 Normal per-cpu: Jan 18 20:08:04 kvm-s63702 kernel: [719664.027062] CPU 0: hi: 186, btch: 31 usd: 32 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027063] CPU 1: hi: 186, btch: 31 usd: 22 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027064] CPU 2: hi: 186, btch: 31 usd: 38 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027065] CPU 3: hi: 186, btch: 31 usd: 48 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027065] CPU 4: hi: 186, btch: 31 usd: 11 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027066] CPU 5: hi: 186, btch: 31 usd: 55 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027069] CPU 6: hi: 186, btch: 31 usd: 34 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027070] CPU 7: hi: 186, btch: 31 usd: 165 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027071] CPU 8: hi: 186, btch: 31 usd: 47 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027072] CPU 9: hi: 186, btch: 31 usd: 4 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027073] CPU 10: hi: 186, btch: 31 usd: 42 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027074] CPU 11: hi: 186, btch: 31 usd: 51 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027077] active_anon:7586025 inactive_anon:73 isolated_anon:0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027077] active_file:459 inactive_file:368 isolated_file:0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027077] unevictable:1394 dirty:460 writeback:0 unstable:0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027077] free:47976 slab_reclaimable:7205 slab_unreclaimable:9010 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027077] mapped:1050 shmem:181 pagetables:22434 bounce:0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027077] free_cma:0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027080] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 18 20:08:04 kvm-s63702 kernel: [719664.027086] lowmem_reserve[]: 0 2978 30143 30143 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027089] Node 0 DMA32 free:115328kB min:6668kB low:8332kB high:10000kB active_anon:2905356kB inactive_anon:32kB active_file:0kB inactive_file:0kB unevictable:756kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:756kB dirty:124kB writeback:0kB mapped:864kB shmem:56kB slab_reclaimable:2412kB slab_unreclaimable:3272kB kernel_stack:1520kB pagetables:8364kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:190884 all_unreclaimable? yes Jan 18 20:08:04 kvm-s63702 kernel: [719664.027092] lowmem_reserve[]: 0 0 27165 27165 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027095] Node 0 Normal free:60684kB min:60876kB low:76092kB high:91312kB active_anon:27438744kB inactive_anon:260kB active_file:1900kB inactive_file:1508kB unevictable:4820kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:4820kB dirty:1716kB writeback:0kB mapped:3336kB shmem:668kB slab_reclaimable:26408kB slab_unreclaimable:32752kB kernel_stack:21760kB pagetables:81372kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:168420 all_unreclaimable? yes Jan 18 20:08:04 kvm-s63702 kernel: [719664.027098] lowmem_reserve[]: 0 0 0 0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027100] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 18 20:08:04 kvm-s63702 kernel: [719664.027111] Node 0 DMA32: 71*4kB (UEM) 56*8kB (UE) 123*16kB (EM) 104*32kB (E) 84*64kB (UE) 37*128kB (UEM) 14*256kB (UE) 8*512kB (UE) 5*1024kB (UEM) 6*2048kB (UM) 18*4096kB (UMR) = 114956kB Jan 18 20:08:04 kvm-s63702 kernel: [719664.027120] Node 0 Normal: 288*4kB (UEM) 232*8kB (UEM) 314*16kB (UEM) 226*32kB (UEM) 191*64kB (EM) 120*128kB (UEM) 48*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 59232kB Jan 18 20:08:04 kvm-s63702 kernel: [719664.027129] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 18 20:08:04 kvm-s63702 kernel: [719664.027130] 2114 total pagecache pages Jan 18 20:08:04 kvm-s63702 kernel: [719664.027132] 0 pages in swap cache Jan 18 20:08:04 kvm-s63702 kernel: [719664.027133] Swap cache stats: add 0, delete 0, find 0/0 Jan 18 20:08:04 kvm-s63702 kernel: [719664.027134] Free swap = 0kB Jan 18 20:08:04 kvm-s63702 kernel: [719664.027135] Total swap = 0kB Jan 18 20:08:04 kvm-s63702 kernel: [719664.027135] 7864218 pages RAM Jan 18 20:08:04 kvm-s63702 kernel: [719664.027136] 0 pages HighMem/MovableOnly Jan 18 20:08:04 kvm-s63702 kernel: [719664.027137] 143432 pages reserved Jan 18 20:08:04 kvm-s63702 kernel: [719664.027138] 0 pages cma reserved Jan 18 20:08:04 kvm-s63702 kernel: [719664.027139] 0 pages hwpoisoned Jan 18 20:08:04 kvm-s63702 kernel: [719664.027139] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 18 20:08:04 kvm-s63702 kernel: [719664.027146] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 18 20:08:04 kvm-s63702 kernel: [719664.027148] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 18 20:08:04 kvm-s63702 kernel: [719664.027150] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 18 20:08:04 kvm-s63702 kernel: [719664.027152] [ 724] 0 724 2559 865 8 0 0 dhclient Jan 18 20:08:04 kvm-s63702 kernel: [719664.027154] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 18 20:08:04 kvm-s63702 kernel: [719664.027156] [ 932] 101 932 65020 267 29 0 0 rsyslogd Jan 18 20:08:04 kvm-s63702 kernel: [719664.027158] [ 961] 102 961 9809 339 23 0 0 dbus-daemon Jan 18 20:08:04 kvm-s63702 kernel: [719664.027160] [ 964] 0 964 3956 41 12 0 0 getty Jan 18 20:08:04 kvm-s63702 kernel: [719664.027164] [ 967] 0 967 3956 40 13 0 0 getty Jan 18 20:08:04 kvm-s63702 kernel: [719664.027166] [ 972] 0 972 3956 40 13 0 0 getty Jan 18 20:08:04 kvm-s63702 kernel: [719664.027168] [ 973] 0 973 3956 39 13 0 0 getty Jan 18 20:08:04 kvm-s63702 kernel: [719664.027169] [ 975] 0 975 3956 39 13 0 0 getty Jan 18 20:08:04 kvm-s63702 kernel: [719664.027171] [ 995] 0 995 15343 285 33 0 -1000 sshd Jan 18 20:08:04 kvm-s63702 kernel: [719664.027173] [ 998] 0 998 1093 37 8 0 0 acpid Jan 18 20:08:04 kvm-s63702 kernel: [719664.027174] [ 1016] 0 1016 5915 238 18 0 0 cron Jan 18 20:08:04 kvm-s63702 kernel: [719664.027176] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 18 20:08:04 kvm-s63702 kernel: [719664.027178] [ 1046] 0 1046 10864 413 27 0 0 systemd-logind Jan 18 20:08:04 kvm-s63702 kernel: [719664.027179] [ 1052] 0 1052 4799 288 14 0 0 irqbalance Jan 18 20:08:04 kvm-s63702 kernel: [719664.027180] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 18 20:08:04 kvm-s63702 kernel: [719664.027183] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 18 20:08:04 kvm-s63702 kernel: [719664.027184] [ 5396] 1000 5396 353101 8520 89 0 0 beam.smp Jan 18 20:08:04 kvm-s63702 kernel: [719664.027186] [ 5432] 1000 5432 2948 1090 10 0 0 gosecrets Jan 18 20:08:04 kvm-s63702 kernel: [719664.027187] [ 5436] 1000 5436 756707 224801 771 0 0 beam.smp Jan 18 20:08:04 kvm-s63702 kernel: [719664.027189] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 18 20:08:04 kvm-s63702 kernel: [719664.027190] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 18 20:08:04 kvm-s63702 kernel: [719664.027192] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 18 20:08:04 kvm-s63702 kernel: [719664.027193] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 18 20:08:04 kvm-s63702 kernel: [719664.027195] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 18 20:08:04 kvm-s63702 kernel: [719664.027197] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 18 20:08:04 kvm-s63702 kernel: [719664.027198] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 18 20:08:04 kvm-s63702 kernel: [719664.027200] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 18 20:08:04 kvm-s63702 kernel: [719664.027201] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 18 20:08:04 kvm-s63702 kernel: [719664.027202] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 18 20:08:04 kvm-s63702 kernel: [719664.027204] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 18 20:08:04 kvm-s63702 kernel: [719664.027206] [14025] 1000 14025 1462 169 8 0 0 goport Jan 18 20:08:04 kvm-s63702 kernel: [719664.027208] [14029] 1000 14029 74588 2164 29 0 0 goxdcr Jan 18 20:08:04 kvm-s63702 kernel: [719664.027209] [14040] 1000 14040 277166 17577 137 0 0 beam.smp Jan 18 20:08:04 kvm-s63702 kernel: [719664.027211] [14076] 1000 14076 1113 361 7 0 0 sh Jan 18 20:08:04 kvm-s63702 kernel: [719664.027212] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 18 20:08:04 kvm-s63702 kernel: [719664.027213] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 18 20:08:04 kvm-s63702 kernel: [719664.027215] [14085] 1000 14085 1331 101 8 0 0 godu Jan 18 20:08:04 kvm-s63702 kernel: [719664.027216] [14086] 1000 14086 1113 320 7 0 0 sh Jan 18 20:08:04 kvm-s63702 kernel: [719664.027218] [14087] 1000 14087 1331 100 8 0 0 godu Jan 18 20:08:04 kvm-s63702 kernel: [719664.027219] [14098] 1000 14098 2156 422 10 0 0 sigar_port Jan 18 20:08:04 kvm-s63702 kernel: [719664.027221] [ 6503] 0 6503 4465 1396 15 0 0 atop Jan 18 20:08:04 kvm-s63702 kernel: [719664.027223] [22184] 1000 22184 1526 163 8 0 0 goport Jan 18 20:08:04 kvm-s63702 kernel: [719664.027224] [22189] 1000 22189 15427399 7281254 20692 0 0 indexer Jan 18 20:08:04 kvm-s63702 kernel: [719664.027226] Out of memory: Kill process 22189 (indexer) score 945 or sacrifice child Jan 18 20:08:04 kvm-s63702 kernel: [719664.030419] Killed process 22189 (indexer) total-vm:61709596kB, anon-rss:29123636kB, file-rss:1800kB Jan 18 20:10:43 kvm-s63702 kernel: [719823.750927] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.750938] indexer cpuset=/ mems_allowed=0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.750958] CPU: 1 PID: 25242 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 18 20:10:43 kvm-s63702 kernel: [719823.750961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 18 20:10:43 kvm-s63702 kernel: [719823.750965] 0000000000000000 ffff8806549479d8 ffffffff817aed00 0000000000000000 Jan 18 20:10:43 kvm-s63702 kernel: [719823.750970] ffff88061ea3bae0 ffff880654947a78 ffffffff817a9c3f ffff880654947a98 Jan 18 20:10:43 kvm-s63702 kernel: [719823.750973] 0000000000000000 0000000000000014 0000000000000002 ffff880654947ac8 Jan 18 20:10:43 kvm-s63702 kernel: [719823.750979] Call Trace: Jan 18 20:10:43 kvm-s63702 kernel: [719823.751030] [] dump_stack+0x45/0x57 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751035] [] dump_header+0x7f/0x1f1 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751058] [] oom_kill_process+0x22b/0x390 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751078] [] ? has_capability_noaudit+0x1e/0x30 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751081] [] out_of_memory+0x4d2/0x520 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751085] [] __alloc_pages_nodemask+0x940/0xa60 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751099] [] alloc_pages_vma+0x97/0x150 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751106] [] handle_mm_fault+0xd13/0x1100 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751117] [] ? x2apic_send_IPI_mask+0x13/0x20 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751123] [] ? try_to_wake_up+0x1f1/0x340 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751133] [] __do_page_fault+0x1c4/0x5b0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751140] [] ? __enqueue_entity+0x78/0x80 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751146] [] ? pvclock_clocksource_read+0x3f/0xe0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751149] [] trace_do_page_fault+0x45/0x100 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751152] [] do_async_page_fault+0x1e/0xd0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751163] [] async_page_fault+0x28/0x30 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751165] Mem-Info: Jan 18 20:10:43 kvm-s63702 kernel: [719823.751169] Node 0 DMA per-cpu: Jan 18 20:10:43 kvm-s63702 kernel: [719823.751175] CPU 0: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751176] CPU 1: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751177] CPU 2: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751178] CPU 3: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751179] CPU 4: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751180] CPU 5: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751181] CPU 6: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751182] CPU 7: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751183] CPU 8: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751184] CPU 9: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751185] CPU 10: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751186] CPU 11: hi: 0, btch: 1 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751187] Node 0 DMA32 per-cpu: Jan 18 20:10:43 kvm-s63702 kernel: [719823.751189] CPU 0: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751190] CPU 1: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751191] CPU 2: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751192] CPU 3: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751193] CPU 4: hi: 186, btch: 31 usd: 23 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751194] CPU 5: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751195] CPU 6: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751196] CPU 7: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751197] CPU 8: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751198] CPU 9: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751199] CPU 10: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751200] CPU 11: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751201] Node 0 Normal per-cpu: Jan 18 20:10:43 kvm-s63702 kernel: [719823.751202] CPU 0: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751203] CPU 1: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751204] CPU 2: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751205] CPU 3: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751206] CPU 4: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751207] CPU 5: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751208] CPU 6: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751209] CPU 7: hi: 186, btch: 31 usd: 5 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751210] CPU 8: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751211] CPU 9: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751212] CPU 10: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751213] CPU 11: hi: 186, btch: 31 usd: 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751216] active_anon:7602638 inactive_anon:73 isolated_anon:0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751216] active_file:274 inactive_file:283 isolated_file:0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751216] unevictable:1394 dirty:0 writeback:0 unstable:0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751216] free:48321 slab_reclaimable:7024 slab_unreclaimable:8574 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751216] mapped:1111 shmem:181 pagetables:18210 bounce:0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751216] free_cma:0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751219] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 18 20:10:43 kvm-s63702 kernel: [719823.751228] lowmem_reserve[]: 0 2978 30143 30143 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751231] Node 0 DMA32 free:116976kB min:6668kB low:8332kB high:10000kB active_anon:2911904kB inactive_anon:32kB active_file:216kB inactive_file:252kB unevictable:756kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:756kB dirty:0kB writeback:0kB mapped:1012kB shmem:56kB slab_reclaimable:2404kB slab_unreclaimable:3112kB kernel_stack:1776kB pagetables:7888kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3440 all_unreclaimable? yes Jan 18 20:10:43 kvm-s63702 kernel: [719823.751235] lowmem_reserve[]: 0 0 27165 27165 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751237] Node 0 Normal free:60416kB min:60876kB low:76092kB high:91312kB active_anon:27498648kB inactive_anon:260kB active_file:880kB inactive_file:880kB unevictable:4820kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:4820kB dirty:0kB writeback:0kB mapped:3432kB shmem:668kB slab_reclaimable:25692kB slab_unreclaimable:31168kB kernel_stack:21504kB pagetables:64952kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9520 all_unreclaimable? no Jan 18 20:10:43 kvm-s63702 kernel: [719823.751240] lowmem_reserve[]: 0 0 0 0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751242] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 18 20:10:43 kvm-s63702 kernel: [719823.751256] Node 0 DMA32: 268*4kB (UEM) 104*8kB (UEM) 145*16kB (UEM) 120*32kB (UEM) 116*64kB (UEM) 62*128kB (UEM) 23*256kB (UEM) 8*512kB (UE) 4*1024kB (UE) 3*2048kB (UM) 18*4096kB (UMR) = 117376kB Jan 18 20:10:43 kvm-s63702 kernel: [719823.751265] Node 0 Normal: 164*4kB (UEM) 114*8kB (UEM) 148*16kB (UEM) 137*32kB (UEM) 105*64kB (UEM) 180*128kB (UEM) 72*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 60608kB Jan 18 20:10:43 kvm-s63702 kernel: [719823.751273] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 18 20:10:43 kvm-s63702 kernel: [719823.751274] 1652 total pagecache pages Jan 18 20:10:43 kvm-s63702 kernel: [719823.751275] 0 pages in swap cache Jan 18 20:10:43 kvm-s63702 kernel: [719823.751276] Swap cache stats: add 0, delete 0, find 0/0 Jan 18 20:10:43 kvm-s63702 kernel: [719823.751277] Free swap = 0kB Jan 18 20:10:43 kvm-s63702 kernel: [719823.751278] Total swap = 0kB Jan 18 20:10:43 kvm-s63702 kernel: [719823.751279] 7864218 pages RAM Jan 18 20:10:43 kvm-s63702 kernel: [719823.751279] 0 pages HighMem/MovableOnly Jan 18 20:10:43 kvm-s63702 kernel: [719823.751280] 143432 pages reserved Jan 18 20:10:43 kvm-s63702 kernel: [719823.751281] 0 pages cma reserved Jan 18 20:10:43 kvm-s63702 kernel: [719823.751282] 0 pages hwpoisoned Jan 18 20:10:43 kvm-s63702 kernel: [719823.751282] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 18 20:10:43 kvm-s63702 kernel: [719823.751289] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 18 20:10:43 kvm-s63702 kernel: [719823.751291] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 18 20:10:43 kvm-s63702 kernel: [719823.751294] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 18 20:10:43 kvm-s63702 kernel: [719823.751295] [ 724] 0 724 2559 865 8 0 0 dhclient Jan 18 20:10:43 kvm-s63702 kernel: [719823.751297] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 18 20:10:43 kvm-s63702 kernel: [719823.751299] [ 932] 101 932 65020 267 29 0 0 rsyslogd Jan 18 20:10:43 kvm-s63702 kernel: [719823.751300] [ 961] 102 961 9809 339 23 0 0 dbus-daemon Jan 18 20:10:43 kvm-s63702 kernel: [719823.751302] [ 964] 0 964 3956 41 12 0 0 getty Jan 18 20:10:43 kvm-s63702 kernel: [719823.751303] [ 967] 0 967 3956 40 13 0 0 getty Jan 18 20:10:43 kvm-s63702 kernel: [719823.751305] [ 972] 0 972 3956 40 13 0 0 getty Jan 18 20:10:43 kvm-s63702 kernel: [719823.751306] [ 973] 0 973 3956 39 13 0 0 getty Jan 18 20:10:43 kvm-s63702 kernel: [719823.751308] [ 975] 0 975 3956 39 13 0 0 getty Jan 18 20:10:43 kvm-s63702 kernel: [719823.751310] [ 995] 0 995 15343 285 33 0 -1000 sshd Jan 18 20:10:43 kvm-s63702 kernel: [719823.751311] [ 998] 0 998 1093 37 8 0 0 acpid Jan 18 20:10:43 kvm-s63702 kernel: [719823.751313] [ 1016] 0 1016 5915 228 18 0 0 cron Jan 18 20:10:43 kvm-s63702 kernel: [719823.751314] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 18 20:10:43 kvm-s63702 kernel: [719823.751319] [ 1046] 0 1046 10864 413 27 0 0 systemd-logind Jan 18 20:10:43 kvm-s63702 kernel: [719823.751321] [ 1052] 0 1052 4799 288 14 0 0 irqbalance Jan 18 20:10:43 kvm-s63702 kernel: [719823.751322] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 18 20:10:43 kvm-s63702 kernel: [719823.751325] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 18 20:10:43 kvm-s63702 kernel: [719823.751327] [ 5396] 1000 5396 353659 8396 90 0 0 beam.smp Jan 18 20:10:43 kvm-s63702 kernel: [719823.751328] [ 5432] 1000 5432 2948 1156 10 0 0 gosecrets Jan 18 20:10:43 kvm-s63702 kernel: [719823.751330] [ 5436] 1000 5436 765830 230032 825 0 0 beam.smp Jan 18 20:10:43 kvm-s63702 kernel: [719823.751332] [ 5474] 1000 5474 1113 101 7 0 0 sh Jan 18 20:10:43 kvm-s63702 kernel: [719823.751333] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 18 20:10:43 kvm-s63702 kernel: [719823.751335] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 18 20:10:43 kvm-s63702 kernel: [719823.751336] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 18 20:10:43 kvm-s63702 kernel: [719823.751338] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 18 20:10:43 kvm-s63702 kernel: [719823.751340] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 18 20:10:43 kvm-s63702 kernel: [719823.751341] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 18 20:10:43 kvm-s63702 kernel: [719823.751343] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 18 20:10:43 kvm-s63702 kernel: [719823.751345] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 18 20:10:43 kvm-s63702 kernel: [719823.751346] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 18 20:10:43 kvm-s63702 kernel: [719823.751348] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 18 20:10:43 kvm-s63702 kernel: [719823.751350] [14025] 1000 14025 1462 169 8 0 0 goport Jan 18 20:10:43 kvm-s63702 kernel: [719823.751351] [14029] 1000 14029 74588 2178 29 0 0 goxdcr Jan 18 20:10:43 kvm-s63702 kernel: [719823.751353] [14040] 1000 14040 280161 19783 144 0 0 beam.smp Jan 18 20:10:43 kvm-s63702 kernel: [719823.751355] [14076] 1000 14076 1113 361 7 0 0 sh Jan 18 20:10:43 kvm-s63702 kernel: [719823.751356] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 18 20:10:43 kvm-s63702 kernel: [719823.751358] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 18 20:10:43 kvm-s63702 kernel: [719823.751359] [14085] 1000 14085 1331 101 8 0 0 godu Jan 18 20:10:43 kvm-s63702 kernel: [719823.751361] [14086] 1000 14086 1113 320 7 0 0 sh Jan 18 20:10:43 kvm-s63702 kernel: [719823.751362] [14087] 1000 14087 1331 100 8 0 0 godu Jan 18 20:10:43 kvm-s63702 kernel: [719823.751364] [14098] 1000 14098 2156 422 10 0 0 sigar_port Jan 18 20:10:43 kvm-s63702 kernel: [719823.751366] [ 6503] 0 6503 4465 1396 15 0 0 atop Jan 18 20:10:43 kvm-s63702 kernel: [719823.751368] [25147] 1000 25147 1526 157 9 0 0 goport Jan 18 20:10:43 kvm-s63702 kernel: [719823.751370] [25152] 1000 25152 13840520 7289462 16401 0 0 indexer Jan 18 20:10:43 kvm-s63702 kernel: [719823.751371] Out of memory: Kill process 25152 (indexer) score 946 or sacrifice child Jan 18 20:10:43 kvm-s63702 kernel: [719823.754396] Killed process 25152 (indexer) total-vm:55362080kB, anon-rss:29155964kB, file-rss:1884kB Jan 18 23:44:11 kvm-s63702 kernel: [732631.512360] indexer invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512367] indexer cpuset=/ mems_allowed=0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512388] CPU: 5 PID: 28143 Comm: indexer Not tainted 3.19.0-25-generic #26~14.04.1-Ubuntu Jan 18 23:44:11 kvm-s63702 kernel: [732631.512390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512392] 0000000000000000 ffff880667ddb9d8 ffffffff817aed00 0000000000000000 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512395] ffff88065b0cd850 ffff880667ddba78 ffffffff817a9c3f 0000000000000000 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512398] 0000000000000000 0000000000000000 0000000000000000 ffff880667ddbac8 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512401] Call Trace: Jan 18 23:44:11 kvm-s63702 kernel: [732631.512454] [] dump_stack+0x45/0x57 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512459] [] dump_header+0x7f/0x1f1 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512482] [] oom_kill_process+0x22b/0x390 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512486] [] out_of_memory+0x4d2/0x520 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512492] [] __alloc_pages_nodemask+0x940/0xa60 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512506] [] alloc_pages_vma+0x97/0x150 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512517] [] handle_mm_fault+0xd13/0x1100 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512532] [] ? sched_clock_cpu+0x4a/0xc0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512547] [] __do_page_fault+0x1c4/0x5b0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512550] [] ? __enqueue_entity+0x78/0x80 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512553] [] ? set_next_entity+0x9d/0xb0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512558] [] ? pick_next_task_fair+0x61b/0x880 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512571] [] ? __switch_to+0x15f/0x570 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512574] [] ? sched_clock_cpu+0x98/0xc0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512578] [] trace_do_page_fault+0x45/0x100 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512582] [] do_async_page_fault+0x1e/0xd0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512594] [] async_page_fault+0x28/0x30 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512597] Mem-Info: Jan 18 23:44:11 kvm-s63702 kernel: [732631.512602] Node 0 DMA per-cpu: Jan 18 23:44:11 kvm-s63702 kernel: [732631.512608] CPU 0: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512609] CPU 1: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512611] CPU 2: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512612] CPU 3: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512614] CPU 4: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512615] CPU 5: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512616] CPU 6: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512618] CPU 7: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512619] CPU 8: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512621] CPU 9: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512622] CPU 10: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512624] CPU 11: hi: 0, btch: 1 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512625] Node 0 DMA32 per-cpu: Jan 18 23:44:11 kvm-s63702 kernel: [732631.512627] CPU 0: hi: 186, btch: 31 usd: 88 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512628] CPU 1: hi: 186, btch: 31 usd: 37 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512630] CPU 2: hi: 186, btch: 31 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512631] CPU 3: hi: 186, btch: 31 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512633] CPU 4: hi: 186, btch: 31 usd: 39 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512634] CPU 5: hi: 186, btch: 31 usd: 10 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512635] CPU 6: hi: 186, btch: 31 usd: 9 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512637] CPU 7: hi: 186, btch: 31 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512638] CPU 8: hi: 186, btch: 31 usd: 11 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512639] CPU 9: hi: 186, btch: 31 usd: 102 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512641] CPU 10: hi: 186, btch: 31 usd: 64 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512642] CPU 11: hi: 186, btch: 31 usd: 38 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512643] Node 0 Normal per-cpu: Jan 18 23:44:11 kvm-s63702 kernel: [732631.512645] CPU 0: hi: 186, btch: 31 usd: 5 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512646] CPU 1: hi: 186, btch: 31 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512648] CPU 2: hi: 186, btch: 31 usd: 30 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512649] CPU 3: hi: 186, btch: 31 usd: 30 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512651] CPU 4: hi: 186, btch: 31 usd: 29 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512652] CPU 5: hi: 186, btch: 31 usd: 25 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512653] CPU 6: hi: 186, btch: 31 usd: 30 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512655] CPU 7: hi: 186, btch: 31 usd: 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512656] CPU 8: hi: 186, btch: 31 usd: 30 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512658] CPU 9: hi: 186, btch: 31 usd: 180 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512659] CPU 10: hi: 186, btch: 31 usd: 19 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512660] CPU 11: hi: 186, btch: 31 usd: 28 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512665] active_anon:7584611 inactive_anon:73 isolated_anon:0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512665] active_file:0 inactive_file:0 isolated_file:0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512665] unevictable:1394 dirty:0 writeback:0 unstable:0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512665] free:48276 slab_reclaimable:6822 slab_unreclaimable:8587 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512665] mapped:816 shmem:188 pagetables:22437 bounce:0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512665] free_cma:0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512669] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jan 18 23:44:11 kvm-s63702 kernel: [732631.512681] lowmem_reserve[]: 0 2978 30143 30143 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512690] Node 0 DMA32 free:115988kB min:6668kB low:8332kB high:10000kB active_anon:2903496kB inactive_anon:32kB active_file:160kB inactive_file:140kB unevictable:756kB isolated(anon):0kB isolated(file):0kB present:3129328kB managed:3049572kB mlocked:756kB dirty:0kB writeback:0kB mapped:896kB shmem:56kB slab_reclaimable:2160kB slab_unreclaimable:2576kB kernel_stack:2256kB pagetables:10828kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2044 all_unreclaimable? yes Jan 18 23:44:11 kvm-s63702 kernel: [732631.512698] lowmem_reserve[]: 0 0 27165 27165 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512701] Node 0 Normal free:61224kB min:60876kB low:76092kB high:91312kB active_anon:27434948kB inactive_anon:260kB active_file:0kB inactive_file:0kB unevictable:4820kB isolated(anon):0kB isolated(file):0kB present:28311552kB managed:27817664kB mlocked:4820kB dirty:0kB writeback:0kB mapped:2368kB shmem:696kB slab_reclaimable:25128kB slab_unreclaimable:31756kB kernel_stack:21024kB pagetables:78920kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:384 all_unreclaimable? yes Jan 18 23:44:11 kvm-s63702 kernel: [732631.512706] lowmem_reserve[]: 0 0 0 0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512713] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB Jan 18 23:44:11 kvm-s63702 kernel: [732631.512728] Node 0 DMA32: 247*4kB (UEM) 113*8kB (UEM) 122*16kB (UEM) 92*32kB (UEM) 61*64kB (UEM) 41*128kB (EM) 21*256kB (UE) 6*512kB (UE) 4*1024kB (UE) 7*2048kB (UM) 18*4096kB (UMR) = 116548kB Jan 18 23:44:11 kvm-s63702 kernel: [732631.512745] Node 0 Normal: 299*4kB (UEM) 232*8kB (UE) 251*16kB (UE) 152*32kB (UEM) 105*64kB (EM) 78*128kB (UE) 45*256kB (EM) 16*512kB (EM) 7*1024kB (E) 0*2048kB 1*4096kB (R) = 59612kB Jan 18 23:44:11 kvm-s63702 kernel: [732631.512758] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jan 18 23:44:11 kvm-s63702 kernel: [732631.512760] 1289 total pagecache pages Jan 18 23:44:11 kvm-s63702 kernel: [732631.512761] 0 pages in swap cache Jan 18 23:44:11 kvm-s63702 kernel: [732631.512763] Swap cache stats: add 0, delete 0, find 0/0 Jan 18 23:44:11 kvm-s63702 kernel: [732631.512764] Free swap = 0kB Jan 18 23:44:11 kvm-s63702 kernel: [732631.512765] Total swap = 0kB Jan 18 23:44:11 kvm-s63702 kernel: [732631.512766] 7864218 pages RAM Jan 18 23:44:11 kvm-s63702 kernel: [732631.512768] 0 pages HighMem/MovableOnly Jan 18 23:44:11 kvm-s63702 kernel: [732631.512769] 143432 pages reserved Jan 18 23:44:11 kvm-s63702 kernel: [732631.512770] 0 pages cma reserved Jan 18 23:44:11 kvm-s63702 kernel: [732631.512771] 0 pages hwpoisoned Jan 18 23:44:11 kvm-s63702 kernel: [732631.512773] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jan 18 23:44:11 kvm-s63702 kernel: [732631.512782] [ 375] 0 375 4870 66 13 0 0 upstart-udev-br Jan 18 23:44:11 kvm-s63702 kernel: [732631.512785] [ 381] 0 381 12939 217 28 0 -1000 systemd-udevd Jan 18 23:44:11 kvm-s63702 kernel: [732631.512789] [ 566] 0 566 3816 51 12 0 0 upstart-socket- Jan 18 23:44:11 kvm-s63702 kernel: [732631.512791] [ 724] 0 724 2559 699 8 0 0 dhclient Jan 18 23:44:11 kvm-s63702 kernel: [732631.512794] [ 878] 0 878 3820 47 12 0 0 upstart-file-br Jan 18 23:44:11 kvm-s63702 kernel: [732631.512796] [ 932] 101 932 65020 267 29 0 0 rsyslogd Jan 18 23:44:11 kvm-s63702 kernel: [732631.512799] [ 961] 102 961 9809 334 23 0 0 dbus-daemon Jan 18 23:44:11 kvm-s63702 kernel: [732631.512802] [ 964] 0 964 3956 41 12 0 0 getty Jan 18 23:44:11 kvm-s63702 kernel: [732631.512804] [ 967] 0 967 3956 40 13 0 0 getty Jan 18 23:44:11 kvm-s63702 kernel: [732631.512806] [ 972] 0 972 3956 40 13 0 0 getty Jan 18 23:44:11 kvm-s63702 kernel: [732631.512809] [ 973] 0 973 3956 39 13 0 0 getty Jan 18 23:44:11 kvm-s63702 kernel: [732631.512811] [ 975] 0 975 3956 39 13 0 0 getty Jan 18 23:44:11 kvm-s63702 kernel: [732631.512814] [ 995] 0 995 15343 285 33 0 -1000 sshd Jan 18 23:44:11 kvm-s63702 kernel: [732631.512817] [ 998] 0 998 1093 37 8 0 0 acpid Jan 18 23:44:11 kvm-s63702 kernel: [732631.512819] [ 1016] 0 1016 5915 238 18 0 0 cron Jan 18 23:44:11 kvm-s63702 kernel: [732631.512822] [ 1017] 0 1017 4786 43 13 0 0 atd Jan 18 23:44:11 kvm-s63702 kernel: [732631.512825] [ 1046] 0 1046 10864 408 27 0 0 systemd-logind Jan 18 23:44:11 kvm-s63702 kernel: [732631.512827] [ 1052] 0 1052 4799 283 14 0 0 irqbalance Jan 18 23:44:11 kvm-s63702 kernel: [732631.512830] [ 1520] 0 1520 3956 40 13 0 0 getty Jan 18 23:44:11 kvm-s63702 kernel: [732631.512834] [ 5353] 1000 5353 1875 41 8 0 0 epmd Jan 18 23:44:11 kvm-s63702 kernel: [732631.512837] [ 5396] 1000 5396 352049 8140 88 0 0 beam.smp Jan 18 23:44:11 kvm-s63702 kernel: [732631.512839] [ 5432] 1000 5432 2948 1192 10 0 0 gosecrets Jan 18 23:44:11 kvm-s63702 kernel: [732631.512842] [ 5436] 1000 5436 748061 207729 743 0 0 beam.smp Jan 18 23:44:11 kvm-s63702 kernel: [732631.512845] [ 5474] 1000 5474 1113 126 7 0 0 sh Jan 18 23:44:11 kvm-s63702 kernel: [732631.512853] [ 5476] 1000 5476 1084 234 7 0 0 memsup Jan 18 23:44:11 kvm-s63702 kernel: [732631.512855] [ 5477] 1000 5477 1084 20 8 0 0 cpu_sup Jan 18 23:44:11 kvm-s63702 kernel: [732631.512858] [ 5479] 1000 5479 1867 26 9 0 0 inet_gethost Jan 18 23:44:11 kvm-s63702 kernel: [732631.512861] [ 5480] 1000 5480 1867 30 9 0 0 inet_gethost Jan 18 23:44:11 kvm-s63702 kernel: [732631.512864] [ 5554] 1000 5554 28872 263 19 0 0 saslauthd-port Jan 18 23:44:11 kvm-s63702 kernel: [732631.512866] [ 5571] 1000 5571 428648 5430 112 0 0 memcached Jan 18 23:44:11 kvm-s63702 kernel: [732631.512869] [ 5772] 1000 5772 1866 22 9 0 0 inet_gethost Jan 18 23:44:11 kvm-s63702 kernel: [732631.512872] [ 5773] 1000 5773 1866 28 9 0 0 inet_gethost Jan 18 23:44:11 kvm-s63702 kernel: [732631.512874] [ 5893] 1000 5893 233207 6550 76 0 0 beam.smp Jan 18 23:44:11 kvm-s63702 kernel: [732631.512877] [ 7058] 1000 7058 1867 30 9 0 0 inet_gethost Jan 18 23:44:11 kvm-s63702 kernel: [732631.512880] [14025] 1000 14025 1462 169 8 0 0 goport Jan 18 23:44:11 kvm-s63702 kernel: [732631.512883] [14029] 1000 14029 74588 1925 29 0 0 goxdcr Jan 18 23:44:11 kvm-s63702 kernel: [732631.512885] [14040] 1000 14040 285273 22724 153 0 0 beam.smp Jan 18 23:44:11 kvm-s63702 kernel: [732631.512888] [14076] 1000 14076 1113 361 7 0 0 sh Jan 18 23:44:11 kvm-s63702 kernel: [732631.512891] [14078] 1000 14078 1084 377 8 0 0 memsup Jan 18 23:44:11 kvm-s63702 kernel: [732631.512893] [14079] 1000 14079 1084 154 8 0 0 cpu_sup Jan 18 23:44:11 kvm-s63702 kernel: [732631.512895] [14085] 1000 14085 1331 101 8 0 0 godu Jan 18 23:44:11 kvm-s63702 kernel: [732631.512898] [14086] 1000 14086 1113 320 7 0 0 sh Jan 18 23:44:11 kvm-s63702 kernel: [732631.512901] [14087] 1000 14087 1331 100 8 0 0 godu Jan 18 23:44:11 kvm-s63702 kernel: [732631.512904] [14098] 1000 14098 2156 418 10 0 0 sigar_port Jan 18 23:44:11 kvm-s63702 kernel: [732631.512907] [ 6503] 0 6503 4465 1396 15 0 0 atop Jan 18 23:44:11 kvm-s63702 kernel: [732631.512910] [27395] 1000 27395 1462 164 8 0 0 goport Jan 18 23:44:11 kvm-s63702 kernel: [732631.512912] [27400] 1000 27400 15581950 7302022 20882 0 0 indexer Jan 18 23:44:11 kvm-s63702 kernel: [732631.512915] Out of memory: Kill process 27400 (indexer) score 948 or sacrifice child Jan 18 23:44:11 kvm-s63702 kernel: [732631.517179] Killed process 27400 (indexer) total-vm:62327800kB, anon-rss:29208088kB, file-rss:0kB Jan 19 00:00:02 kvm-s63702 kernel: [733582.689113] Process accounting resumed