Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7748

[system test] online upgrade added node 2.0.1 down during swap rebalance

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.1
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
    • Environment:
      unix

      Description

      Environment:

      • Both source and destination cluster are in 2.0.0 GA
      • 2 nodes cluster at source with 2 buckets, one doc and 3 views for each doc
      • 2 nodes cluster at source with 2 buckets

      Load 1 M items to both buckets
      Do online upgrade at source cluster by using swap rebalance.
      Add node ubu-2509 with buid 2.0.1-152 to source cluster and remove one 2.0.0 node.
      Rebalance. Failed.
      Node ubu-2509 was down due to operating system killed beam.smp

      Message from dmesg on node ubu-2509

      [2509410.019520] lowmem_reserve[]: 0 0 0 0
      [2509410.019524] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
      [2509410.019535] Node 0 DMA32: 1017*4kB 216*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 9892kB
      [2509410.019545] Node 0 Normal: 486*4kB 4*8kB 3*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2024kB
      [2509410.019556] 10742 total pagecache pages
      [2509410.019558] 717 pages in swap cache
      [2509410.019560] Swap cache stats: add 121992, delete 121275, find 11066/15116
      [2509410.019563] Free swap = 6085400kB
      [2509410.019564] Total swap = 6430712kB
      [2509410.041429] 1048560 pages RAM
      [2509410.041432] 33577 pages reserved
      [2509410.041447] 24046 pages shared
      [2509410.041449] 997455 pages non-shared
      [2509410.041454] Out of memory: kill process 19376 (beam.smp) score 9026 or a child
      [2509410.045160] Killed process 19376 (beam.smp)
      [2524506.554641] process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.default.retrans_time; Use net.ipv6.neigh.default.retrans_time_ms instead.

      Collect info of all nodes from source cluster
      https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201302/3nodes-online-upgrade-src-os-kill-beam-201-node.tgz

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        thuan Thuan Nguyen added a comment -

        At 2013-02-13T17:19:02, node cen-2501 saw node ubu-2509 went down

        [user:warn,2013-02-13T17:19:02.177,ns_1@cen-2501.hq.couchbase.com:ns_node_disco<0.13144.18>:ns_node_disco:handle_info:168]Node 'ns_1@cen-2501.hq.couchbase.com' saw that node 'ns_1@ubu-2509.hq.couchbase.com' went down.

        Look at /var/log/syslog.5, I saw beam.smp was killed by system out of memory

        Feb 13 16:04:01 ubu-2509 epmd: epmd: got partial packet only on file descriptor 5 (0)
        Feb 13 16:17:01 ubu-2509 CRON[23467]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 13 17:17:02 ubu-2509 CRON[24393]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.543954] memcached invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.543960] memcached cpuset=/ mems_allowed=0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.543964] Pid: 23926, comm: memcached Not tainted 2.6.32-41-generic #89-Ubuntu
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.543968] Call Trace:
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544010] [<ffffffff810b3acd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544028] [<ffffffff810f7b64>] oom_kill_process+0xd4/0x2f0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544035] [<ffffffff810f8120>] ? select_bad_process+0xd0/0x110
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544041] [<ffffffff810f81b8>] __out_of_memory+0x58/0xc0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544046] [<ffffffff810f834e>] out_of_memory+0x12e/0x1a0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544068] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544077] [<ffffffff810fb4f1>] __alloc_pages_slowpath+0x571/0x590
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544083] [<ffffffff810fb66e>] __alloc_pages_nodemask+0x15e/0x1a0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544087] [<ffffffff8112e8d2>] alloc_page_vma+0x72/0xf0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544091] [<ffffffff810f5090>] ? sync_page+0x0/0x50
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544096] [<ffffffff81123dce>] read_swap_cache_async+0xde/0x130
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544100] [<ffffffff8112457e>] ? valid_swaphandles+0x16e/0x1a0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544104] [<ffffffff81123ec3>] swapin_readahead+0xa3/0xc0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544120] [<ffffffff8111645e>] do_swap_page+0x11e/0x4c0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544124] [<ffffffff81115bb3>] ? do_wp_page+0xd3/0x860
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544128] [<ffffffff81116acb>] handle_mm_fault+0x2cb/0x3c0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544135] [<ffffffff8154ad4a>] do_page_fault+0x12a/0x3b0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544141] [<ffffffff81548695>] page_fault+0x25/0x30
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544145] Mem-Info:
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544147] Node 0 DMA per-cpu:
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544152] CPU 0: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544155] CPU 1: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544159] CPU 2: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544162] CPU 3: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544164] Node 0 DMA32 per-cpu:
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544169] CPU 0: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544172] CPU 1: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544174] CPU 2: hi: 186, btch: 31 usd: 24
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544176] CPU 3: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544178] Node 0 Normal per-cpu:
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544181] CPU 0: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544183] CPU 1: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544185] CPU 2: hi: 186, btch: 31 usd: 31
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544188] CPU 3: hi: 186, btch: 31 usd: 2
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544193] active_anon:1037 inactive_anon:356 isolated_anon:97
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544194] active_file:32 inactive_file:61 isolated_file:0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544196] unevictable:975368 dirty:3 writeback:478 unstable:0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544197] free:7034 slab_reclaimable:2026 slab_unreclaimable:2675
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544198] mapped:9906 shmem:0 pagetables:2717 bounce:0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544201] Node 0 DMA free:15876kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544212] lowmem_reserve[]: 0 3000 4010 4010
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544217] Node 0 DMA32 free:10248kB min:6052kB low:7564kB high:9076kB active_anon:4124kB inactive_anon:1236kB active_file:128kB inactive_file:28kB unevictable:2966032kB isolated(anon):388kB isolated(file):0kB present:3072160kB mlocked:2966032kB dirty:8kB writeback:1708kB mapped:7940kB shmem:0kB slab_reclaimable:4324kB slab_unreclaimable:2140kB kernel_stack:248kB pagetables:5872kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:256 all_unreclaimable? no
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544229] lowmem_reserve[]: 0 0 1010 1010
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544233] Node 0 Normal free:2012kB min:2036kB low:2544kB high:3052kB active_anon:24kB inactive_anon:188kB active_file:0kB inactive_file:216kB unevictable:935440kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:935440kB dirty:4kB writeback:204kB mapped:31684kB shmem:0kB slab_reclaimable:3780kB slab_unreclaimable:8552kB kernel_stack:1136kB pagetables:4996kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:320 all_unreclaimable? no
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544245] lowmem_reserve[]: 0 0 0 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544249] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544260] Node 0 DMA32: 1036*4kB 266*8kB 7*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 10480kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544270] Node 0 Normal: 497*4kB 3*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2012kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544280] 10622 total pagecache pages
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544282] 744 pages in swap cache
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544285] Swap cache stats: add 121883, delete 121139, find 11058/15087
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544287] Free swap = 6085392kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.544289] Total swap = 6430712kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.567476] 1048560 pages RAM
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.567479] 33577 pages reserved
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.567481] 24036 pages shared
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.567483] 997322 pages non-shared
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.567487] Out of memory: kill process 23209 (memcached) score 16189 or a child
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.569873] Killed process 23209 (memcached)
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786235] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786243] beam.smp cpuset=/ mems_allowed=0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786248] Pid: 19401, comm: beam.smp Not tainted 2.6.32-41-generic #89-Ubuntu
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786252] Call Trace:
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786265] [<ffffffff810b3acd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786274] [<ffffffff810f7b64>] oom_kill_process+0xd4/0x2f0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786279] [<ffffffff810f8120>] ? select_bad_process+0xd0/0x110
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786284] [<ffffffff810f81b8>] __out_of_memory+0x58/0xc0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786289] [<ffffffff810f834e>] out_of_memory+0x12e/0x1a0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786375] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786381] [<ffffffff810fb4f1>] __alloc_pages_slowpath+0x571/0x590
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786387] [<ffffffff810fb66e>] __alloc_pages_nodemask+0x15e/0x1a0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786391] [<ffffffff8112e8d2>] alloc_page_vma+0x72/0xf0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786396] [<ffffffff81111ea0>] do_anonymous_page+0x160/0x330
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786401] [<ffffffff81116b1f>] handle_mm_fault+0x31f/0x3c0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786404] [<ffffffff81112594>] ? follow_page+0x194/0x350
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786409] [<ffffffff81116cf9>] __get_user_pages+0x139/0x480
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786413] [<ffffffff81117ec0>] __mlock_vma_pages_range+0xa0/0x1f0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786417] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786421] [<ffffffff811180a1>] mlock_vma_pages_range+0x91/0xa0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786425] [<ffffffff8111b8e5>] mmap_region+0x305/0x590
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786430] [<ffffffff8111bea5>] do_mmap_pgoff+0x335/0x380
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786434] [<ffffffff8110d6aa>] sys_mmap_pgoff+0x7a/0x2a0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786461] [<ffffffff81016b19>] sys_mmap+0x29/0x30
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786468] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786471] Mem-Info:
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786473] Node 0 DMA per-cpu:
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786476] CPU 0: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786478] CPU 1: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786481] CPU 2: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786483] CPU 3: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786484] Node 0 DMA32 per-cpu:
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786487] CPU 0: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786490] CPU 1: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786492] CPU 2: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786494] CPU 3: hi: 186, btch: 31 usd: 22
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786496] Node 0 Normal per-cpu:
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786499] CPU 0: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786501] CPU 1: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786503] CPU 2: hi: 186, btch: 31 usd: 2
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786505] CPU 3: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786511] active_anon:1037 inactive_anon:382 isolated_anon:97
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786512] active_file:0 inactive_file:68 isolated_file:0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786513] unevictable:975423 dirty:3 writeback:478 unstable:0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786514] free:7004 slab_reclaimable:2026 slab_unreclaimable:2675
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786516] mapped:9906 shmem:0 pagetables:2717 bounce:0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786518] Node 0 DMA free:15876kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786530] lowmem_reserve[]: 0 3000 4010 4010
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786534] Node 0 DMA32 free:10056kB min:6052kB low:7564kB high:9076kB active_anon:4124kB inactive_anon:1320kB active_file:0kB inactive_file:212kB unevictable:2966252kB isolated(anon):388kB isolated(file):0kB present:3072160kB mlocked:2966252kB dirty:8kB writeback:1708kB mapped:7940kB shmem:0kB slab_reclaimable:4324kB slab_unreclaimable:2140kB kernel_stack:248kB pagetables:5872kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:920 all_unreclaimable? no
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786556] lowmem_reserve[]: 0 0 1010 1010
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786560] Node 0 Normal free:2084kB min:2036kB low:2544kB high:3052kB active_anon:24kB inactive_anon:208kB active_file:0kB inactive_file:60kB unevictable:935440kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:935440kB dirty:4kB writeback:204kB mapped:31684kB shmem:0kB slab_reclaimable:3780kB slab_unreclaimable:8552kB kernel_stack:1136kB pagetables:4996kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:128 all_unreclaimable? no
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786572] lowmem_reserve[]: 0 0 0 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786576] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786587] Node 0 DMA32: 1022*4kB 250*8kB 11*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 10360kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786597] Node 0 Normal: 530*4kB 4*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2184kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786608] 10732 total pagecache pages
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786610] 746 pages in swap cache
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786612] Swap cache stats: add 121943, delete 121197, find 11061/15102
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786614] Free swap = 6085404kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.786616] Total swap = 6430712kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.802818] 1048560 pages RAM
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.802822] 33577 pages reserved
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.802823] 24062 pages shared
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.802825] 997337 pages non-shared
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.802829] Out of memory: kill process 23208 (moxi) score 9184 or a child
        Feb 13 17:21:13 ubu-2509 kernel: [2509409.805021] Killed process 23208 (moxi)
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019239] rsyslogd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019245] rsyslogd cpuset=/ mems_allowed=0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019250] Pid: 1131, comm: rsyslogd Not tainted 2.6.32-41-generic #89-Ubuntu
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019253] Call Trace:
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019263] [<ffffffff810b3acd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019270] [<ffffffff810f7b64>] oom_kill_process+0xd4/0x2f0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019274] [<ffffffff810f8120>] ? select_bad_process+0xd0/0x110
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019278] [<ffffffff810f81b8>] __out_of_memory+0x58/0xc0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019282] [<ffffffff810f834e>] out_of_memory+0x12e/0x1a0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019287] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019292] [<ffffffff810fb4f1>] __alloc_pages_slowpath+0x571/0x590
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019297] [<ffffffff810fb66e>] __alloc_pages_nodemask+0x15e/0x1a0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019303] [<ffffffff8112e8d2>] alloc_page_vma+0x72/0xf0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019322] [<ffffffff81123dce>] read_swap_cache_async+0xde/0x130
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019328] [<ffffffff8112457e>] ? valid_swaphandles+0x16e/0x1a0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019333] [<ffffffff81123ea7>] swapin_readahead+0x87/0xc0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019340] [<ffffffff8111645e>] do_swap_page+0x11e/0x4c0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019346] [<ffffffff81115bb3>] ? do_wp_page+0xd3/0x860
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019363] [<ffffffff8102d891>] ? native_smp_send_reschedule+0x41/0x60
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019371] [<ffffffff81116acb>] handle_mm_fault+0x2cb/0x3c0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019378] [<ffffffff8154ad4a>] do_page_fault+0x12a/0x3b0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019383] [<ffffffff81548695>] page_fault+0x25/0x30
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019387] Mem-Info:
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019390] Node 0 DMA per-cpu:
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019394] CPU 0: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019398] CPU 1: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019401] CPU 2: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019405] CPU 3: hi: 0, btch: 1 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019407] Node 0 DMA32 per-cpu:
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019411] CPU 0: hi: 186, btch: 31 usd: 18
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019415] CPU 1: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019418] CPU 2: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019421] CPU 3: hi: 186, btch: 31 usd: 30
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019423] Node 0 Normal per-cpu:
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019427] CPU 0: hi: 186, btch: 31 usd: 2
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019430] CPU 1: hi: 186, btch: 31 usd: 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019433] CPU 2: hi: 186, btch: 31 usd: 30
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019437] CPU 3: hi: 186, btch: 31 usd: 7
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019443] active_anon:1064 inactive_anon:375 isolated_anon:32
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019445] active_file:62 inactive_file:88 isolated_file:0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019446] unevictable:975509 dirty:0 writeback:418 unstable:0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019447] free:6959 slab_reclaimable:1997 slab_unreclaimable:2685
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019448] mapped:9881 shmem:0 pagetables:2717 bounce:0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019451] Node 0 DMA free:15876kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019463] lowmem_reserve[]: 0 3000 4010 4010
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019467] Node 0 DMA32 free:10000kB min:6052kB low:7564kB high:9076kB active_anon:4252kB inactive_anon:1384kB active_file:136kB inactive_file:140kB unevictable:2966468kB isolated(anon):128kB isolated(file):0kB present:3072160kB mlocked:2966468kB dirty:0kB writeback:1528kB mapped:7860kB shmem:0kB slab_reclaimable:4260kB slab_unreclaimable:2164kB kernel_stack:128kB pagetables:5872kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019479] lowmem_reserve[]: 0 0 1010 1010
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019483] Node 0 Normal free:1960kB min:2036kB low:2544kB high:3052kB active_anon:4kB inactive_anon:116kB active_file:112kB inactive_file:212kB unevictable:935568kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:935568kB dirty:0kB writeback:144kB mapped:31664kB shmem:0kB slab_reclaimable:3728kB slab_unreclaimable:8568kB kernel_stack:1136kB pagetables:4996kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019520] lowmem_reserve[]: 0 0 0 0
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019524] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019535] Node 0 DMA32: 1017*4kB 216*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 9892kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019545] Node 0 Normal: 486*4kB 4*8kB 3*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2024kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019556] 10742 total pagecache pages
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019558] 717 pages in swap cache
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019560] Swap cache stats: add 121992, delete 121275, find 11066/15116
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019563] Free swap = 6085400kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.019564] Total swap = 6430712kB
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.041429] 1048560 pages RAM
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.041432] 33577 pages reserved
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.041447] 24046 pages shared
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.041449] 997455 pages non-shared
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.041454] Out of memory: kill process 19376 (beam.smp) score 9026 or a child
        Feb 13 17:21:13 ubu-2509 kernel: [2509410.045160] Killed process 19376 (beam.smp)
        Feb 13 18:17:01 ubu-2509 CRON[24537]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 13 19:17:01 ubu-2509 CRON[24642]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 13 20:17:02 ubu-2509 CRON[24746]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 13 20:25:33 ubu-2509 dhclient: DHCPREQUEST of 10.3.3.29 on eth0 to 10.1.1.250 port 67
        Feb 13 20:25:34 ubu-2509 dhclient: DHCPACK of 10.3.3.29 from 10.1.1.250
        Feb 13 20:25:38 ubu-2509 dhclient: bound to 10.3.3.29 – renewal in 21050 seconds.
        Feb 13 21:17:02 ubu-2509 CRON[24859]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 13 21:33:19 ubu-2509 kernel: [2524506.554641] process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.default.retrans_time; Use net.ipv6.neigh.default.retrans_time_ms instead.
        Feb 13 22:17:02 ubu-2509 CRON[25064]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 13 23:17:02 ubu-2509 CRON[25121]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 14 00:17:01 ubu-2509 CRON[25226]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 14 01:17:01 ubu-2509 CRON[25330]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 14 02:16:28 ubu-2509 dhclient: DHCPREQUEST of 10.3.3.29 on eth0 to 10.1.1.250 port 67
        Feb 14 02:16:28 ubu-2509 dhclient: DHCPACK of 10.3.3.29 from 10.1.1.100
        Feb 14 02:16:31 ubu-2509 dhclient: bound to 10.3.3.29 – renewal in 20099 seconds.
        Feb 14 02:17:02 ubu-2509 CRON[25444]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 14 03:17:01 ubu-2509 CRON[25547]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 14 04:17:01 ubu-2509 CRON[25651]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 14 05:17:01 ubu-2509 CRON[25754]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 14 06:17:02 ubu-2509 CRON[25858]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
        Feb 14 06:25:01 ubu-2509 CRON[25911]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))

        Link to system log of node 29 https://s3.amazonaws.com/bugdb/jira/MB-7748/syslog_of_down_node29_oom_beam_killed-20130218-122348.tgz

        Show
        thuan Thuan Nguyen added a comment - At 2013-02-13T17:19:02, node cen-2501 saw node ubu-2509 went down [user:warn,2013-02-13T17:19:02.177,ns_1@cen-2501.hq.couchbase.com:ns_node_disco<0.13144.18>:ns_node_disco:handle_info:168] Node 'ns_1@cen-2501.hq.couchbase.com' saw that node 'ns_1@ubu-2509.hq.couchbase.com' went down. Look at /var/log/syslog.5, I saw beam.smp was killed by system out of memory Feb 13 16:04:01 ubu-2509 epmd: epmd: got partial packet only on file descriptor 5 (0) Feb 13 16:17:01 ubu-2509 CRON [23467] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 13 17:17:02 ubu-2509 CRON [24393] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 13 17:21:13 ubu-2509 kernel: [2509409.543954] memcached invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.543960] memcached cpuset=/ mems_allowed=0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.543964] Pid: 23926, comm: memcached Not tainted 2.6.32-41-generic #89-Ubuntu Feb 13 17:21:13 ubu-2509 kernel: [2509409.543968] Call Trace: Feb 13 17:21:13 ubu-2509 kernel: [2509409.544010] [<ffffffff810b3acd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544028] [<ffffffff810f7b64>] oom_kill_process+0xd4/0x2f0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544035] [<ffffffff810f8120>] ? select_bad_process+0xd0/0x110 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544041] [<ffffffff810f81b8>] __out_of_memory+0x58/0xc0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544046] [<ffffffff810f834e>] out_of_memory+0x12e/0x1a0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544068] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544077] [<ffffffff810fb4f1>] __alloc_pages_slowpath+0x571/0x590 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544083] [<ffffffff810fb66e>] __alloc_pages_nodemask+0x15e/0x1a0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544087] [<ffffffff8112e8d2>] alloc_page_vma+0x72/0xf0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544091] [<ffffffff810f5090>] ? sync_page+0x0/0x50 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544096] [<ffffffff81123dce>] read_swap_cache_async+0xde/0x130 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544100] [<ffffffff8112457e>] ? valid_swaphandles+0x16e/0x1a0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544104] [<ffffffff81123ec3>] swapin_readahead+0xa3/0xc0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544120] [<ffffffff8111645e>] do_swap_page+0x11e/0x4c0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544124] [<ffffffff81115bb3>] ? do_wp_page+0xd3/0x860 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544128] [<ffffffff81116acb>] handle_mm_fault+0x2cb/0x3c0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544135] [<ffffffff8154ad4a>] do_page_fault+0x12a/0x3b0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544141] [<ffffffff81548695>] page_fault+0x25/0x30 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544145] Mem-Info: Feb 13 17:21:13 ubu-2509 kernel: [2509409.544147] Node 0 DMA per-cpu: Feb 13 17:21:13 ubu-2509 kernel: [2509409.544152] CPU 0: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544155] CPU 1: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544159] CPU 2: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544162] CPU 3: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544164] Node 0 DMA32 per-cpu: Feb 13 17:21:13 ubu-2509 kernel: [2509409.544169] CPU 0: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544172] CPU 1: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544174] CPU 2: hi: 186, btch: 31 usd: 24 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544176] CPU 3: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544178] Node 0 Normal per-cpu: Feb 13 17:21:13 ubu-2509 kernel: [2509409.544181] CPU 0: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544183] CPU 1: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544185] CPU 2: hi: 186, btch: 31 usd: 31 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544188] CPU 3: hi: 186, btch: 31 usd: 2 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544193] active_anon:1037 inactive_anon:356 isolated_anon:97 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544194] active_ file:32 inactive_ file:61 isolated_ file:0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544196] unevictable:975368 dirty:3 writeback:478 unstable:0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544197] free:7034 slab_reclaimable:2026 slab_unreclaimable:2675 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544198] mapped:9906 shmem:0 pagetables:2717 bounce:0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544201] Node 0 DMA free:15876kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_ file:0kB inactive_ file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Feb 13 17:21:13 ubu-2509 kernel: [2509409.544212] lowmem_reserve[]: 0 3000 4010 4010 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544217] Node 0 DMA32 free:10248kB min:6052kB low:7564kB high:9076kB active_anon:4124kB inactive_anon:1236kB active_ file:128kB inactive_ file:28kB unevictable:2966032kB isolated(anon):388kB isolated(file):0kB present:3072160kB mlocked:2966032kB dirty:8kB writeback:1708kB mapped:7940kB shmem:0kB slab_reclaimable:4324kB slab_unreclaimable:2140kB kernel_stack:248kB pagetables:5872kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:256 all_unreclaimable? no Feb 13 17:21:13 ubu-2509 kernel: [2509409.544229] lowmem_reserve[]: 0 0 1010 1010 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544233] Node 0 Normal free:2012kB min:2036kB low:2544kB high:3052kB active_anon:24kB inactive_anon:188kB active_ file:0kB inactive_ file:216kB unevictable:935440kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:935440kB dirty:4kB writeback:204kB mapped:31684kB shmem:0kB slab_reclaimable:3780kB slab_unreclaimable:8552kB kernel_stack:1136kB pagetables:4996kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:320 all_unreclaimable? no Feb 13 17:21:13 ubu-2509 kernel: [2509409.544245] lowmem_reserve[]: 0 0 0 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544249] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Feb 13 17:21:13 ubu-2509 kernel: [2509409.544260] Node 0 DMA32: 1036*4kB 266*8kB 7*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 10480kB Feb 13 17:21:13 ubu-2509 kernel: [2509409.544270] Node 0 Normal: 497*4kB 3*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2012kB Feb 13 17:21:13 ubu-2509 kernel: [2509409.544280] 10622 total pagecache pages Feb 13 17:21:13 ubu-2509 kernel: [2509409.544282] 744 pages in swap cache Feb 13 17:21:13 ubu-2509 kernel: [2509409.544285] Swap cache stats: add 121883, delete 121139, find 11058/15087 Feb 13 17:21:13 ubu-2509 kernel: [2509409.544287] Free swap = 6085392kB Feb 13 17:21:13 ubu-2509 kernel: [2509409.544289] Total swap = 6430712kB Feb 13 17:21:13 ubu-2509 kernel: [2509409.567476] 1048560 pages RAM Feb 13 17:21:13 ubu-2509 kernel: [2509409.567479] 33577 pages reserved Feb 13 17:21:13 ubu-2509 kernel: [2509409.567481] 24036 pages shared Feb 13 17:21:13 ubu-2509 kernel: [2509409.567483] 997322 pages non-shared Feb 13 17:21:13 ubu-2509 kernel: [2509409.567487] Out of memory: kill process 23209 (memcached) score 16189 or a child Feb 13 17:21:13 ubu-2509 kernel: [2509409.569873] Killed process 23209 (memcached) Feb 13 17:21:13 ubu-2509 kernel: [2509409.786235] beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786243] beam.smp cpuset=/ mems_allowed=0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786248] Pid: 19401, comm: beam.smp Not tainted 2.6.32-41-generic #89-Ubuntu Feb 13 17:21:13 ubu-2509 kernel: [2509409.786252] Call Trace: Feb 13 17:21:13 ubu-2509 kernel: [2509409.786265] [<ffffffff810b3acd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786274] [<ffffffff810f7b64>] oom_kill_process+0xd4/0x2f0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786279] [<ffffffff810f8120>] ? select_bad_process+0xd0/0x110 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786284] [<ffffffff810f81b8>] __out_of_memory+0x58/0xc0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786289] [<ffffffff810f834e>] out_of_memory+0x12e/0x1a0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786375] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786381] [<ffffffff810fb4f1>] __alloc_pages_slowpath+0x571/0x590 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786387] [<ffffffff810fb66e>] __alloc_pages_nodemask+0x15e/0x1a0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786391] [<ffffffff8112e8d2>] alloc_page_vma+0x72/0xf0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786396] [<ffffffff81111ea0>] do_anonymous_page+0x160/0x330 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786401] [<ffffffff81116b1f>] handle_mm_fault+0x31f/0x3c0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786404] [<ffffffff81112594>] ? follow_page+0x194/0x350 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786409] [<ffffffff81116cf9>] __get_user_pages+0x139/0x480 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786413] [<ffffffff81117ec0>] __mlock_vma_pages_range+0xa0/0x1f0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786417] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786421] [<ffffffff811180a1>] mlock_vma_pages_range+0x91/0xa0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786425] [<ffffffff8111b8e5>] mmap_region+0x305/0x590 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786430] [<ffffffff8111bea5>] do_mmap_pgoff+0x335/0x380 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786434] [<ffffffff8110d6aa>] sys_mmap_pgoff+0x7a/0x2a0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786461] [<ffffffff81016b19>] sys_mmap+0x29/0x30 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786468] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b Feb 13 17:21:13 ubu-2509 kernel: [2509409.786471] Mem-Info: Feb 13 17:21:13 ubu-2509 kernel: [2509409.786473] Node 0 DMA per-cpu: Feb 13 17:21:13 ubu-2509 kernel: [2509409.786476] CPU 0: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786478] CPU 1: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786481] CPU 2: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786483] CPU 3: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786484] Node 0 DMA32 per-cpu: Feb 13 17:21:13 ubu-2509 kernel: [2509409.786487] CPU 0: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786490] CPU 1: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786492] CPU 2: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786494] CPU 3: hi: 186, btch: 31 usd: 22 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786496] Node 0 Normal per-cpu: Feb 13 17:21:13 ubu-2509 kernel: [2509409.786499] CPU 0: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786501] CPU 1: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786503] CPU 2: hi: 186, btch: 31 usd: 2 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786505] CPU 3: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786511] active_anon:1037 inactive_anon:382 isolated_anon:97 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786512] active_ file:0 inactive_ file:68 isolated_ file:0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786513] unevictable:975423 dirty:3 writeback:478 unstable:0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786514] free:7004 slab_reclaimable:2026 slab_unreclaimable:2675 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786516] mapped:9906 shmem:0 pagetables:2717 bounce:0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786518] Node 0 DMA free:15876kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_ file:0kB inactive_ file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Feb 13 17:21:13 ubu-2509 kernel: [2509409.786530] lowmem_reserve[]: 0 3000 4010 4010 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786534] Node 0 DMA32 free:10056kB min:6052kB low:7564kB high:9076kB active_anon:4124kB inactive_anon:1320kB active_ file:0kB inactive_ file:212kB unevictable:2966252kB isolated(anon):388kB isolated(file):0kB present:3072160kB mlocked:2966252kB dirty:8kB writeback:1708kB mapped:7940kB shmem:0kB slab_reclaimable:4324kB slab_unreclaimable:2140kB kernel_stack:248kB pagetables:5872kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:920 all_unreclaimable? no Feb 13 17:21:13 ubu-2509 kernel: [2509409.786556] lowmem_reserve[]: 0 0 1010 1010 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786560] Node 0 Normal free:2084kB min:2036kB low:2544kB high:3052kB active_anon:24kB inactive_anon:208kB active_ file:0kB inactive_ file:60kB unevictable:935440kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:935440kB dirty:4kB writeback:204kB mapped:31684kB shmem:0kB slab_reclaimable:3780kB slab_unreclaimable:8552kB kernel_stack:1136kB pagetables:4996kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:128 all_unreclaimable? no Feb 13 17:21:13 ubu-2509 kernel: [2509409.786572] lowmem_reserve[]: 0 0 0 0 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786576] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Feb 13 17:21:13 ubu-2509 kernel: [2509409.786587] Node 0 DMA32: 1022*4kB 250*8kB 11*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 10360kB Feb 13 17:21:13 ubu-2509 kernel: [2509409.786597] Node 0 Normal: 530*4kB 4*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2184kB Feb 13 17:21:13 ubu-2509 kernel: [2509409.786608] 10732 total pagecache pages Feb 13 17:21:13 ubu-2509 kernel: [2509409.786610] 746 pages in swap cache Feb 13 17:21:13 ubu-2509 kernel: [2509409.786612] Swap cache stats: add 121943, delete 121197, find 11061/15102 Feb 13 17:21:13 ubu-2509 kernel: [2509409.786614] Free swap = 6085404kB Feb 13 17:21:13 ubu-2509 kernel: [2509409.786616] Total swap = 6430712kB Feb 13 17:21:13 ubu-2509 kernel: [2509409.802818] 1048560 pages RAM Feb 13 17:21:13 ubu-2509 kernel: [2509409.802822] 33577 pages reserved Feb 13 17:21:13 ubu-2509 kernel: [2509409.802823] 24062 pages shared Feb 13 17:21:13 ubu-2509 kernel: [2509409.802825] 997337 pages non-shared Feb 13 17:21:13 ubu-2509 kernel: [2509409.802829] Out of memory: kill process 23208 (moxi) score 9184 or a child Feb 13 17:21:13 ubu-2509 kernel: [2509409.805021] Killed process 23208 (moxi) Feb 13 17:21:13 ubu-2509 kernel: [2509410.019239] rsyslogd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019245] rsyslogd cpuset=/ mems_allowed=0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019250] Pid: 1131, comm: rsyslogd Not tainted 2.6.32-41-generic #89-Ubuntu Feb 13 17:21:13 ubu-2509 kernel: [2509410.019253] Call Trace: Feb 13 17:21:13 ubu-2509 kernel: [2509410.019263] [<ffffffff810b3acd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019270] [<ffffffff810f7b64>] oom_kill_process+0xd4/0x2f0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019274] [<ffffffff810f8120>] ? select_bad_process+0xd0/0x110 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019278] [<ffffffff810f81b8>] __out_of_memory+0x58/0xc0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019282] [<ffffffff810f834e>] out_of_memory+0x12e/0x1a0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019287] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019292] [<ffffffff810fb4f1>] __alloc_pages_slowpath+0x571/0x590 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019297] [<ffffffff810fb66e>] __alloc_pages_nodemask+0x15e/0x1a0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019303] [<ffffffff8112e8d2>] alloc_page_vma+0x72/0xf0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019322] [<ffffffff81123dce>] read_swap_cache_async+0xde/0x130 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019328] [<ffffffff8112457e>] ? valid_swaphandles+0x16e/0x1a0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019333] [<ffffffff81123ea7>] swapin_readahead+0x87/0xc0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019340] [<ffffffff8111645e>] do_swap_page+0x11e/0x4c0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019346] [<ffffffff81115bb3>] ? do_wp_page+0xd3/0x860 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019363] [<ffffffff8102d891>] ? native_smp_send_reschedule+0x41/0x60 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019371] [<ffffffff81116acb>] handle_mm_fault+0x2cb/0x3c0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019378] [<ffffffff8154ad4a>] do_page_fault+0x12a/0x3b0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019383] [<ffffffff81548695>] page_fault+0x25/0x30 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019387] Mem-Info: Feb 13 17:21:13 ubu-2509 kernel: [2509410.019390] Node 0 DMA per-cpu: Feb 13 17:21:13 ubu-2509 kernel: [2509410.019394] CPU 0: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019398] CPU 1: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019401] CPU 2: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019405] CPU 3: hi: 0, btch: 1 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019407] Node 0 DMA32 per-cpu: Feb 13 17:21:13 ubu-2509 kernel: [2509410.019411] CPU 0: hi: 186, btch: 31 usd: 18 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019415] CPU 1: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019418] CPU 2: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019421] CPU 3: hi: 186, btch: 31 usd: 30 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019423] Node 0 Normal per-cpu: Feb 13 17:21:13 ubu-2509 kernel: [2509410.019427] CPU 0: hi: 186, btch: 31 usd: 2 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019430] CPU 1: hi: 186, btch: 31 usd: 0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019433] CPU 2: hi: 186, btch: 31 usd: 30 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019437] CPU 3: hi: 186, btch: 31 usd: 7 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019443] active_anon:1064 inactive_anon:375 isolated_anon:32 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019445] active_ file:62 inactive_ file:88 isolated_ file:0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019446] unevictable:975509 dirty:0 writeback:418 unstable:0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019447] free:6959 slab_reclaimable:1997 slab_unreclaimable:2685 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019448] mapped:9881 shmem:0 pagetables:2717 bounce:0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019451] Node 0 DMA free:15876kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_ file:0kB inactive_ file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Feb 13 17:21:13 ubu-2509 kernel: [2509410.019463] lowmem_reserve[]: 0 3000 4010 4010 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019467] Node 0 DMA32 free:10000kB min:6052kB low:7564kB high:9076kB active_anon:4252kB inactive_anon:1384kB active_ file:136kB inactive_ file:140kB unevictable:2966468kB isolated(anon):128kB isolated(file):0kB present:3072160kB mlocked:2966468kB dirty:0kB writeback:1528kB mapped:7860kB shmem:0kB slab_reclaimable:4260kB slab_unreclaimable:2164kB kernel_stack:128kB pagetables:5872kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Feb 13 17:21:13 ubu-2509 kernel: [2509410.019479] lowmem_reserve[]: 0 0 1010 1010 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019483] Node 0 Normal free:1960kB min:2036kB low:2544kB high:3052kB active_anon:4kB inactive_anon:116kB active_ file:112kB inactive_ file:212kB unevictable:935568kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:935568kB dirty:0kB writeback:144kB mapped:31664kB shmem:0kB slab_reclaimable:3728kB slab_unreclaimable:8568kB kernel_stack:1136kB pagetables:4996kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Feb 13 17:21:13 ubu-2509 kernel: [2509410.019520] lowmem_reserve[]: 0 0 0 0 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019524] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Feb 13 17:21:13 ubu-2509 kernel: [2509410.019535] Node 0 DMA32: 1017*4kB 216*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 9892kB Feb 13 17:21:13 ubu-2509 kernel: [2509410.019545] Node 0 Normal: 486*4kB 4*8kB 3*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2024kB Feb 13 17:21:13 ubu-2509 kernel: [2509410.019556] 10742 total pagecache pages Feb 13 17:21:13 ubu-2509 kernel: [2509410.019558] 717 pages in swap cache Feb 13 17:21:13 ubu-2509 kernel: [2509410.019560] Swap cache stats: add 121992, delete 121275, find 11066/15116 Feb 13 17:21:13 ubu-2509 kernel: [2509410.019563] Free swap = 6085400kB Feb 13 17:21:13 ubu-2509 kernel: [2509410.019564] Total swap = 6430712kB Feb 13 17:21:13 ubu-2509 kernel: [2509410.041429] 1048560 pages RAM Feb 13 17:21:13 ubu-2509 kernel: [2509410.041432] 33577 pages reserved Feb 13 17:21:13 ubu-2509 kernel: [2509410.041447] 24046 pages shared Feb 13 17:21:13 ubu-2509 kernel: [2509410.041449] 997455 pages non-shared Feb 13 17:21:13 ubu-2509 kernel: [2509410.041454] Out of memory: kill process 19376 (beam.smp) score 9026 or a child Feb 13 17:21:13 ubu-2509 kernel: [2509410.045160] Killed process 19376 (beam.smp) Feb 13 18:17:01 ubu-2509 CRON [24537] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 13 19:17:01 ubu-2509 CRON [24642] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 13 20:17:02 ubu-2509 CRON [24746] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 13 20:25:33 ubu-2509 dhclient: DHCPREQUEST of 10.3.3.29 on eth0 to 10.1.1.250 port 67 Feb 13 20:25:34 ubu-2509 dhclient: DHCPACK of 10.3.3.29 from 10.1.1.250 Feb 13 20:25:38 ubu-2509 dhclient: bound to 10.3.3.29 – renewal in 21050 seconds. Feb 13 21:17:02 ubu-2509 CRON [24859] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 13 21:33:19 ubu-2509 kernel: [2524506.554641] process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.default.retrans_time; Use net.ipv6.neigh.default.retrans_time_ms instead. Feb 13 22:17:02 ubu-2509 CRON [25064] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 13 23:17:02 ubu-2509 CRON [25121] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 14 00:17:01 ubu-2509 CRON [25226] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 14 01:17:01 ubu-2509 CRON [25330] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 14 02:16:28 ubu-2509 dhclient: DHCPREQUEST of 10.3.3.29 on eth0 to 10.1.1.250 port 67 Feb 14 02:16:28 ubu-2509 dhclient: DHCPACK of 10.3.3.29 from 10.1.1.100 Feb 14 02:16:31 ubu-2509 dhclient: bound to 10.3.3.29 – renewal in 20099 seconds. Feb 14 02:17:02 ubu-2509 CRON [25444] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 14 03:17:01 ubu-2509 CRON [25547] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 14 04:17:01 ubu-2509 CRON [25651] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 14 05:17:01 ubu-2509 CRON [25754] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 14 06:17:02 ubu-2509 CRON [25858] : (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Feb 14 06:25:01 ubu-2509 CRON [25911] : (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )) Link to system log of node 29 https://s3.amazonaws.com/bugdb/jira/MB-7748/syslog_of_down_node29_oom_beam_killed-20130218-122348.tgz
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        looks like memory leak.
        if this is reproducible with the latest build please assign this back to Aliaksey.

        Show
        farshid Farshid Ghods (Inactive) added a comment - looks like memory leak. if this is reproducible with the latest build please assign this back to Aliaksey.
        Hide
        thuan Thuan Nguyen added a comment -

        Hit this bug again in destination cluster with build 156. I will test again in build 160 if I can reproduce this bug

        Feb 18 12:26:27 ubu-2510 kernel: [295679.148932] lowmem_reserve[]: 0 0 0 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.148937] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.148986] Node 0 DMA32: 1413*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 9684kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.149002] Node 0 Normal: 162*4kB 0*8kB 2*16kB 1*32kB 3*64kB 4*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1928kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.149018] 11966 total pagecache pages
        Feb 18 12:26:27 ubu-2510 kernel: [295679.149021] 1829 pages in swap cache
        Feb 18 12:26:27 ubu-2510 kernel: [295679.149025] Swap cache stats: add 190336, delete 188507, find 23058/30154
        Feb 18 12:26:27 ubu-2510 kernel: [295679.149029] Free swap = 5908404kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.149031] Total swap = 6430712kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.184743] 1048560 pages RAM
        Feb 18 12:26:27 ubu-2510 kernel: [295679.184747] 33577 pages reserved
        Feb 18 12:26:27 ubu-2510 kernel: [295679.184750] 24396 pages shared
        Feb 18 12:26:27 ubu-2510 kernel: [295679.184752] 996556 pages non-shared
        Feb 18 12:26:27 ubu-2510 kernel: [295679.184758] Out of memory: kill process 1359 (beam.smp) score 4078 or a child
        Feb 18 12:26:27 ubu-2510 kernel: [295679.187176] Killed process 1359 (beam.smp)
        Feb 18 12:26:27 ubu-2510 kernel: [295679.447961] ssl_esock invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.447968] ssl_esock cpuset=/ mems_allowed=0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.447972] Pid: 1397, comm: ssl_esock Not tainted 2.6.32-41-generic #89-Ubuntu
        Feb 18 12:26:27 ubu-2510 kernel: [295679.447975] Call Trace:
        Feb 18 12:26:27 ubu-2510 kernel: [295679.447986] [<ffffffff810b3acd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.447992] [<ffffffff810f7b64>] oom_kill_process+0xd4/0x2f0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.447996] [<ffffffff810f8120>] ? select_bad_process+0xd0/0x110
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448000] [<ffffffff810f81b8>] __out_of_memory+0x58/0xc0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448004] [<ffffffff810f834e>] out_of_memory+0x12e/0x1a0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448009] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448014] [<ffffffff810fb4f1>] __alloc_pages_slowpath+0x571/0x590
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448019] [<ffffffff810fb66e>] __alloc_pages_nodemask+0x15e/0x1a0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448026] [<ffffffff81156a10>] ? pollwake+0x0/0x60
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448030] [<ffffffff8112e817>] alloc_pages_current+0x87/0xd0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448034] [<ffffffff810f5357>] __page_cache_alloc+0x67/0x70
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448038] [<ffffffff810f5090>] ? sync_page+0x0/0x50
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448042] [<ffffffff810fecb9>] __do_page_cache_readahead+0xc9/0x210
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448046] [<ffffffff810fee21>] ra_submit+0x21/0x30
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448049] [<ffffffff810ff1d5>] ondemand_readahead+0x115/0x240
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448054] [<ffffffff81085100>] ? wake_bit_function+0x0/0x40
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448058] [<ffffffff810ff401>] page_cache_sync_readahead+0x31/0x50
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448062] [<ffffffff810f6c0c>] filemap_fault+0x42c/0x460
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448067] [<ffffffff811133d4>] __do_fault+0x54/0x500
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448071] [<ffffffff811169a8>] handle_mm_fault+0x1a8/0x3c0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448077] [<ffffffff81018ec9>] ? read_tsc+0x9/0x20
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448082] [<ffffffff8108f8bb>] ? ktime_get_ts+0xab/0xe0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448087] [<ffffffff8154ad4a>] do_page_fault+0x12a/0x3b0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448091] [<ffffffff81548695>] page_fault+0x25/0x30
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448094] Mem-Info:
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448095] Node 0 DMA per-cpu:
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448099] CPU 0: hi: 0, btch: 1 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448101] CPU 1: hi: 0, btch: 1 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448103] CPU 2: hi: 0, btch: 1 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448106] CPU 3: hi: 0, btch: 1 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448107] Node 0 DMA32 per-cpu:
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448110] CPU 0: hi: 186, btch: 31 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448113] CPU 1: hi: 186, btch: 31 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448115] CPU 2: hi: 186, btch: 31 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448117] CPU 3: hi: 186, btch: 31 usd: 2
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448119] Node 0 Normal per-cpu:
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448122] CPU 0: hi: 186, btch: 31 usd: 30
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448124] CPU 1: hi: 186, btch: 31 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448126] CPU 2: hi: 186, btch: 31 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448128] CPU 3: hi: 186, btch: 31 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448134] active_anon:2261 inactive_anon:722 isolated_anon:64
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448135] active_file:14 inactive_file:45 isolated_file:0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448136] unevictable:968905 dirty:0 writeback:789 unstable:0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448137] free:7024 slab_reclaimable:2589 slab_unreclaimable:4469
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448138] mapped:9898 shmem:0 pagetables:2869 bounce:0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448141] Node 0 DMA free:15876kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0k
        B present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclai
        mable? yes
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448181] lowmem_reserve[]: 0 3000 4010 4010
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448186] Node 0 DMA32 free:9988kB min:6052kB low:7564kB high:9076kB active_anon:9044kB inactive_anon:2932kB active_file:16kB inactive_file:0kB unevictable:2953180kB isolated(anon):2
        56kB isolated(file):0kB present:3072160kB mlocked:2953180kB dirty:0kB writeback:3112kB mapped:60kB shmem:0kB slab_reclaimable:2708kB slab_unreclaimable:6756kB kernel_stack:112kB pagetables:6980kB unstable:0kB bounce:0kB w
        riteback_tmp:0kB pages_scanned:1088 all_unreclaimable? no
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448199] lowmem_reserve[]: 0 0 1010 1010
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448203] Node 0 Normal free:2232kB min:2036kB low:2544kB high:3052kB active_anon:0kB inactive_anon:0kB active_file:40kB inactive_file:248kB unevictable:922440kB isolated(anon):0kB i
        solated(file):0kB present:1034240kB mlocked:922440kB dirty:0kB writeback:44kB mapped:39532kB shmem:0kB slab_reclaimable:7648kB slab_unreclaimable:11112kB kernel_stack:1248kB pagetables:4496kB unstable:0kB bounce:0kB write
        back_tmp:0kB pages_scanned:0 all_unreclaimable? no
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448214] lowmem_reserve[]: 0 0 0 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448218] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448229] Node 0 DMA32: 1546*4kB 17*8kB 2*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10384kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448240] Node 0 Normal: 192*4kB 15*8kB 6*16kB 1*32kB 3*64kB 4*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2232kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448250] 11900 total pagecache pages
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448252] 1819 pages in swap cache
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448255] Swap cache stats: add 190439, delete 188620, find 23063/30176
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448257] Free swap = 5908408kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.448258] Total swap = 6430712kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.463277] 1048560 pages RAM
        Feb 18 12:26:27 ubu-2510 kernel: [295679.463281] 33577 pages reserved
        Feb 18 12:26:27 ubu-2510 kernel: [295679.463283] 24393 pages shared
        Feb 18 12:26:27 ubu-2510 kernel: [295679.463285] 996585 pages non-shared
        Feb 18 12:26:27 ubu-2510 kernel: [295679.463290] Out of memory: kill process 12977 (memcached) score 3122 or a child
        Feb 18 12:26:27 ubu-2510 kernel: [295679.465674] Killed process 12977 (memcached)
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535194] irqbalance invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
        Feb 18 12:26:27 ubu-2510 kernekernel: [295679.535203] irqbalance cpuset=/ mems_allowed=0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535209] Pid: 796, comm: irqbalance Not tainted 2.6.32-41-generic #89-Ubuntu
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535213] Call Trace:
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535225] [<ffffffff810b3acd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535234] [<ffffffff810f7b64>] oom_kill_process+0xd4/0x2f0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535239] [<ffffffff810f8120>] ? select_bad_process+0xd0/0x110
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535245] [<ffffffff810f81b8>] __out_of_memory+0x58/0xc0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535250] [<ffffffff810f834e>] out_of_memory+0x12e/0x1a0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535256] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535263] [<ffffffff810fb4f1>] __alloc_pages_slowpath+0x571/0x590
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535271] [<ffffffff810fb66e>] __alloc_pages_nodemask+0x15e/0x1a0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535276] [<ffffffff8112e817>] alloc_pages_current+0x87/0xd0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535280] [<ffffffff810f5357>] __page_cache_alloc+0x67/0x70
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535285] [<ffffffff810fecb9>] __do_page_cache_readahead+0xc9/0x210
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535288] [<ffffffff810fee21>] ra_submit+0x21/0x30
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535292] [<ffffffff810ff1d5>] ondemand_readahead+0x115/0x240
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535297] [<ffffffff8113ed7e>] ? mem_cgroup_uncharge_swapcache+0x2e/0xb0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535301] [<ffffffff810ff401>] page_cache_sync_readahead+0x31/0x50
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535305] [<ffffffff810f6c0c>] filemap_fault+0x42c/0x460
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535310] [<ffffffff811133d4>] __do_fault+0x54/0x500
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535314] [<ffffffff811169a8>] handle_mm_fault+0x1a8/0x3c0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535320] [<ffffffff81089614>] ? hrtimer_nanosleep+0xc4/0x180
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535325] [<ffffffff8154ad4a>] do_page_fault+0x12a/0x3b0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535328] [<ffffffff81548695>] page_fault+0x25/0x30
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535331] Mem-Info:
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535334] Node 0 DMA per-cpu:
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535337] CPU 0: hi: 0, btch: 1 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535435] CPU 1: hi: 0, btch: 1 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535438] CPU 2: hi: 0, btch: 1 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535440] CPU 3: hi: 0, btch: 1 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535442] Node 0 DMA32 per-cpu:
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535445] CPU 0: hi: 186, btch: 31 usd: 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535448] CPU 1: hi: 186, btch: 31 usd: 1
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535450] CPU 2: hi: 186, btch: 31 usd: 29
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535452] CPU 3: hi: 186, btch: 31 usd: 5
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535454] Node 0 Normal per-cpu:
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535457] CPU 0: hi: 186, btch: 31 usd: 30
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535459] CPU 1: hi: 186, btch: 31 usd: 9
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535461] CPU 2: hi: 186, btch: 31 usd: 30
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535463] CPU 3: hi: 186, btch: 31 usd: 11
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535469] active_anon:2261 inactive_anon:722 isolated_anon:64
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535470] active_file:14 inactive_file:57 isolated_file:0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535471] unevictable:968905 dirty:0 writeback:789 unstable:0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535472] free:7001 slab_reclaimable:2589 slab_unreclaimable:4469
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535474] mapped:9898 shmem:0 pagetables:2869 bounce:0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535476] Node 0 DMA free:15876kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0k
        B present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclai
        mable? yes
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535488] lowmem_reserve[]: 0 3000 4010 4010
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535492] Node 0 DMA32 free:10144kB min:6052kB low:7564kB high:9076kB active_anon:9044kB inactive_anon:2932kB active_file:16kB inactive_file:0kB unevictable:2953180kB isolated(anon):
        256kB isolated(file):0kB present:3072160kB mlocked:2953180kB dirty:0kB writeback:3112kB mapped:60kB shmem:0kB slab_reclaimable:2708kB slab_unreclaimable:6756kB kernel_stack:112kB pagetables:6980kB unstable:0kB bounce:0kB
        writeback_tmp:0kB pages_scanned:2787 all_unreclaimable? no
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535505] lowmem_reserve[]: 0 0 1010 1010
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535509] Node 0 Normal free:1984kB min:2036kB low:2544kB high:3052kB active_anon:0kB inactive_anon:0kB active_file:40kB inactive_file:296kB unevictable:922440kB isolated(anon):0kB i
        solated(file):0kB present:1034240kB mlocked:922440kB dirty:0kB writeback:44kB mapped:39532kB shmem:0kB slab_reclaimable:7648kB slab_unreclaimable:11112kB kernel_stack:1248kB pagetables:4496kB unstable:0kB bounce:0kB write
        back_tmp:0kB pages_scanned:0 all_unreclaimable? no
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535521] lowmem_reserve[]: 0 0 0 0
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535524] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535535] Node 0 DMA32: 1515*4kB 22*8kB 5*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10348kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535545] Node 0 Normal: 176*4kB 2*8kB 3*16kB 1*32kB 3*64kB 4*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2016kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535556] 11937 total pagecache pages
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535558] 1819 pages in swap cache
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535560] Swap cache stats: add 190442, delete 188623, find 23064/30183
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535562] Free swap = 5908408kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.535564] Total swap = 6430712kB
        Feb 18 12:26:27 ubu-2510 kernel: [295679.552112] 1048560 pages RAM
        Feb 18 12:26:27 ubu-2510 kernel: [295679.552117] 33577 pages reserved
        Feb 18 12:26:27 ubu-2510 kernel: [295679.552120] 24402 pages shared
        Feb 18 12:26:27 ubu-2510 kernel: [295679.552122] 996615 pages non-shared
        Feb 18 12:26:27 ubu-2510 kernel: [295679.552128] Out of memory: kill process 12976 (moxi) score 2623 or a child
        Feb 18 12:26:27 ubu-2510 kernel: [295679.554551] Killed process 12976 (moxi)

        Show
        thuan Thuan Nguyen added a comment - Hit this bug again in destination cluster with build 156. I will test again in build 160 if I can reproduce this bug Feb 18 12:26:27 ubu-2510 kernel: [295679.148932] lowmem_reserve[]: 0 0 0 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.148937] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Feb 18 12:26:27 ubu-2510 kernel: [295679.148986] Node 0 DMA32: 1413*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 9684kB Feb 18 12:26:27 ubu-2510 kernel: [295679.149002] Node 0 Normal: 162*4kB 0*8kB 2*16kB 1*32kB 3*64kB 4*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1928kB Feb 18 12:26:27 ubu-2510 kernel: [295679.149018] 11966 total pagecache pages Feb 18 12:26:27 ubu-2510 kernel: [295679.149021] 1829 pages in swap cache Feb 18 12:26:27 ubu-2510 kernel: [295679.149025] Swap cache stats: add 190336, delete 188507, find 23058/30154 Feb 18 12:26:27 ubu-2510 kernel: [295679.149029] Free swap = 5908404kB Feb 18 12:26:27 ubu-2510 kernel: [295679.149031] Total swap = 6430712kB Feb 18 12:26:27 ubu-2510 kernel: [295679.184743] 1048560 pages RAM Feb 18 12:26:27 ubu-2510 kernel: [295679.184747] 33577 pages reserved Feb 18 12:26:27 ubu-2510 kernel: [295679.184750] 24396 pages shared Feb 18 12:26:27 ubu-2510 kernel: [295679.184752] 996556 pages non-shared Feb 18 12:26:27 ubu-2510 kernel: [295679.184758] Out of memory: kill process 1359 (beam.smp) score 4078 or a child Feb 18 12:26:27 ubu-2510 kernel: [295679.187176] Killed process 1359 (beam.smp) Feb 18 12:26:27 ubu-2510 kernel: [295679.447961] ssl_esock invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 Feb 18 12:26:27 ubu-2510 kernel: [295679.447968] ssl_esock cpuset=/ mems_allowed=0 Feb 18 12:26:27 ubu-2510 kernel: [295679.447972] Pid: 1397, comm: ssl_esock Not tainted 2.6.32-41-generic #89-Ubuntu Feb 18 12:26:27 ubu-2510 kernel: [295679.447975] Call Trace: Feb 18 12:26:27 ubu-2510 kernel: [295679.447986] [<ffffffff810b3acd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0 Feb 18 12:26:27 ubu-2510 kernel: [295679.447992] [<ffffffff810f7b64>] oom_kill_process+0xd4/0x2f0 Feb 18 12:26:27 ubu-2510 kernel: [295679.447996] [<ffffffff810f8120>] ? select_bad_process+0xd0/0x110 Feb 18 12:26:27 ubu-2510 kernel: [295679.448000] [<ffffffff810f81b8>] __out_of_memory+0x58/0xc0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448004] [<ffffffff810f834e>] out_of_memory+0x12e/0x1a0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448009] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20 Feb 18 12:26:27 ubu-2510 kernel: [295679.448014] [<ffffffff810fb4f1>] __alloc_pages_slowpath+0x571/0x590 Feb 18 12:26:27 ubu-2510 kernel: [295679.448019] [<ffffffff810fb66e>] __alloc_pages_nodemask+0x15e/0x1a0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448026] [<ffffffff81156a10>] ? pollwake+0x0/0x60 Feb 18 12:26:27 ubu-2510 kernel: [295679.448030] [<ffffffff8112e817>] alloc_pages_current+0x87/0xd0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448034] [<ffffffff810f5357>] __page_cache_alloc+0x67/0x70 Feb 18 12:26:27 ubu-2510 kernel: [295679.448038] [<ffffffff810f5090>] ? sync_page+0x0/0x50 Feb 18 12:26:27 ubu-2510 kernel: [295679.448042] [<ffffffff810fecb9>] __do_page_cache_readahead+0xc9/0x210 Feb 18 12:26:27 ubu-2510 kernel: [295679.448046] [<ffffffff810fee21>] ra_submit+0x21/0x30 Feb 18 12:26:27 ubu-2510 kernel: [295679.448049] [<ffffffff810ff1d5>] ondemand_readahead+0x115/0x240 Feb 18 12:26:27 ubu-2510 kernel: [295679.448054] [<ffffffff81085100>] ? wake_bit_function+0x0/0x40 Feb 18 12:26:27 ubu-2510 kernel: [295679.448058] [<ffffffff810ff401>] page_cache_sync_readahead+0x31/0x50 Feb 18 12:26:27 ubu-2510 kernel: [295679.448062] [<ffffffff810f6c0c>] filemap_fault+0x42c/0x460 Feb 18 12:26:27 ubu-2510 kernel: [295679.448067] [<ffffffff811133d4>] __do_fault+0x54/0x500 Feb 18 12:26:27 ubu-2510 kernel: [295679.448071] [<ffffffff811169a8>] handle_mm_fault+0x1a8/0x3c0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448077] [<ffffffff81018ec9>] ? read_tsc+0x9/0x20 Feb 18 12:26:27 ubu-2510 kernel: [295679.448082] [<ffffffff8108f8bb>] ? ktime_get_ts+0xab/0xe0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448087] [<ffffffff8154ad4a>] do_page_fault+0x12a/0x3b0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448091] [<ffffffff81548695>] page_fault+0x25/0x30 Feb 18 12:26:27 ubu-2510 kernel: [295679.448094] Mem-Info: Feb 18 12:26:27 ubu-2510 kernel: [295679.448095] Node 0 DMA per-cpu: Feb 18 12:26:27 ubu-2510 kernel: [295679.448099] CPU 0: hi: 0, btch: 1 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448101] CPU 1: hi: 0, btch: 1 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448103] CPU 2: hi: 0, btch: 1 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448106] CPU 3: hi: 0, btch: 1 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448107] Node 0 DMA32 per-cpu: Feb 18 12:26:27 ubu-2510 kernel: [295679.448110] CPU 0: hi: 186, btch: 31 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448113] CPU 1: hi: 186, btch: 31 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448115] CPU 2: hi: 186, btch: 31 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448117] CPU 3: hi: 186, btch: 31 usd: 2 Feb 18 12:26:27 ubu-2510 kernel: [295679.448119] Node 0 Normal per-cpu: Feb 18 12:26:27 ubu-2510 kernel: [295679.448122] CPU 0: hi: 186, btch: 31 usd: 30 Feb 18 12:26:27 ubu-2510 kernel: [295679.448124] CPU 1: hi: 186, btch: 31 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448126] CPU 2: hi: 186, btch: 31 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448128] CPU 3: hi: 186, btch: 31 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448134] active_anon:2261 inactive_anon:722 isolated_anon:64 Feb 18 12:26:27 ubu-2510 kernel: [295679.448135] active_ file:14 inactive_ file:45 isolated_ file:0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448136] unevictable:968905 dirty:0 writeback:789 unstable:0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448137] free:7024 slab_reclaimable:2589 slab_unreclaimable:4469 Feb 18 12:26:27 ubu-2510 kernel: [295679.448138] mapped:9898 shmem:0 pagetables:2869 bounce:0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448141] Node 0 DMA free:15876kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_ file:0kB inactive_ file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0k B present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclai mable? yes Feb 18 12:26:27 ubu-2510 kernel: [295679.448181] lowmem_reserve[]: 0 3000 4010 4010 Feb 18 12:26:27 ubu-2510 kernel: [295679.448186] Node 0 DMA32 free:9988kB min:6052kB low:7564kB high:9076kB active_anon:9044kB inactive_anon:2932kB active_ file:16kB inactive_ file:0kB unevictable:2953180kB isolated(anon):2 56kB isolated(file):0kB present:3072160kB mlocked:2953180kB dirty:0kB writeback:3112kB mapped:60kB shmem:0kB slab_reclaimable:2708kB slab_unreclaimable:6756kB kernel_stack:112kB pagetables:6980kB unstable:0kB bounce:0kB w riteback_tmp:0kB pages_scanned:1088 all_unreclaimable? no Feb 18 12:26:27 ubu-2510 kernel: [295679.448199] lowmem_reserve[]: 0 0 1010 1010 Feb 18 12:26:27 ubu-2510 kernel: [295679.448203] Node 0 Normal free:2232kB min:2036kB low:2544kB high:3052kB active_anon:0kB inactive_anon:0kB active_ file:40kB inactive_ file:248kB unevictable:922440kB isolated(anon):0kB i solated(file):0kB present:1034240kB mlocked:922440kB dirty:0kB writeback:44kB mapped:39532kB shmem:0kB slab_reclaimable:7648kB slab_unreclaimable:11112kB kernel_stack:1248kB pagetables:4496kB unstable:0kB bounce:0kB write back_tmp:0kB pages_scanned:0 all_unreclaimable? no Feb 18 12:26:27 ubu-2510 kernel: [295679.448214] lowmem_reserve[]: 0 0 0 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.448218] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Feb 18 12:26:27 ubu-2510 kernel: [295679.448229] Node 0 DMA32: 1546*4kB 17*8kB 2*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10384kB Feb 18 12:26:27 ubu-2510 kernel: [295679.448240] Node 0 Normal: 192*4kB 15*8kB 6*16kB 1*32kB 3*64kB 4*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2232kB Feb 18 12:26:27 ubu-2510 kernel: [295679.448250] 11900 total pagecache pages Feb 18 12:26:27 ubu-2510 kernel: [295679.448252] 1819 pages in swap cache Feb 18 12:26:27 ubu-2510 kernel: [295679.448255] Swap cache stats: add 190439, delete 188620, find 23063/30176 Feb 18 12:26:27 ubu-2510 kernel: [295679.448257] Free swap = 5908408kB Feb 18 12:26:27 ubu-2510 kernel: [295679.448258] Total swap = 6430712kB Feb 18 12:26:27 ubu-2510 kernel: [295679.463277] 1048560 pages RAM Feb 18 12:26:27 ubu-2510 kernel: [295679.463281] 33577 pages reserved Feb 18 12:26:27 ubu-2510 kernel: [295679.463283] 24393 pages shared Feb 18 12:26:27 ubu-2510 kernel: [295679.463285] 996585 pages non-shared Feb 18 12:26:27 ubu-2510 kernel: [295679.463290] Out of memory: kill process 12977 (memcached) score 3122 or a child Feb 18 12:26:27 ubu-2510 kernel: [295679.465674] Killed process 12977 (memcached) Feb 18 12:26:27 ubu-2510 kernel: [295679.535194] irqbalance invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 Feb 18 12:26:27 ubu-2510 kernekernel: [295679.535203] irqbalance cpuset=/ mems_allowed=0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535209] Pid: 796, comm: irqbalance Not tainted 2.6.32-41-generic #89-Ubuntu Feb 18 12:26:27 ubu-2510 kernel: [295679.535213] Call Trace: Feb 18 12:26:27 ubu-2510 kernel: [295679.535225] [<ffffffff810b3acd>] ? cpuset_print_task_mems_allowed+0x9d/0xb0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535234] [<ffffffff810f7b64>] oom_kill_process+0xd4/0x2f0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535239] [<ffffffff810f8120>] ? select_bad_process+0xd0/0x110 Feb 18 12:26:27 ubu-2510 kernel: [295679.535245] [<ffffffff810f81b8>] __out_of_memory+0x58/0xc0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535250] [<ffffffff810f834e>] out_of_memory+0x12e/0x1a0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535256] [<ffffffff815481ce>] ? _spin_lock+0xe/0x20 Feb 18 12:26:27 ubu-2510 kernel: [295679.535263] [<ffffffff810fb4f1>] __alloc_pages_slowpath+0x571/0x590 Feb 18 12:26:27 ubu-2510 kernel: [295679.535271] [<ffffffff810fb66e>] __alloc_pages_nodemask+0x15e/0x1a0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535276] [<ffffffff8112e817>] alloc_pages_current+0x87/0xd0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535280] [<ffffffff810f5357>] __page_cache_alloc+0x67/0x70 Feb 18 12:26:27 ubu-2510 kernel: [295679.535285] [<ffffffff810fecb9>] __do_page_cache_readahead+0xc9/0x210 Feb 18 12:26:27 ubu-2510 kernel: [295679.535288] [<ffffffff810fee21>] ra_submit+0x21/0x30 Feb 18 12:26:27 ubu-2510 kernel: [295679.535292] [<ffffffff810ff1d5>] ondemand_readahead+0x115/0x240 Feb 18 12:26:27 ubu-2510 kernel: [295679.535297] [<ffffffff8113ed7e>] ? mem_cgroup_uncharge_swapcache+0x2e/0xb0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535301] [<ffffffff810ff401>] page_cache_sync_readahead+0x31/0x50 Feb 18 12:26:27 ubu-2510 kernel: [295679.535305] [<ffffffff810f6c0c>] filemap_fault+0x42c/0x460 Feb 18 12:26:27 ubu-2510 kernel: [295679.535310] [<ffffffff811133d4>] __do_fault+0x54/0x500 Feb 18 12:26:27 ubu-2510 kernel: [295679.535314] [<ffffffff811169a8>] handle_mm_fault+0x1a8/0x3c0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535320] [<ffffffff81089614>] ? hrtimer_nanosleep+0xc4/0x180 Feb 18 12:26:27 ubu-2510 kernel: [295679.535325] [<ffffffff8154ad4a>] do_page_fault+0x12a/0x3b0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535328] [<ffffffff81548695>] page_fault+0x25/0x30 Feb 18 12:26:27 ubu-2510 kernel: [295679.535331] Mem-Info: Feb 18 12:26:27 ubu-2510 kernel: [295679.535334] Node 0 DMA per-cpu: Feb 18 12:26:27 ubu-2510 kernel: [295679.535337] CPU 0: hi: 0, btch: 1 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535435] CPU 1: hi: 0, btch: 1 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535438] CPU 2: hi: 0, btch: 1 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535440] CPU 3: hi: 0, btch: 1 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535442] Node 0 DMA32 per-cpu: Feb 18 12:26:27 ubu-2510 kernel: [295679.535445] CPU 0: hi: 186, btch: 31 usd: 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535448] CPU 1: hi: 186, btch: 31 usd: 1 Feb 18 12:26:27 ubu-2510 kernel: [295679.535450] CPU 2: hi: 186, btch: 31 usd: 29 Feb 18 12:26:27 ubu-2510 kernel: [295679.535452] CPU 3: hi: 186, btch: 31 usd: 5 Feb 18 12:26:27 ubu-2510 kernel: [295679.535454] Node 0 Normal per-cpu: Feb 18 12:26:27 ubu-2510 kernel: [295679.535457] CPU 0: hi: 186, btch: 31 usd: 30 Feb 18 12:26:27 ubu-2510 kernel: [295679.535459] CPU 1: hi: 186, btch: 31 usd: 9 Feb 18 12:26:27 ubu-2510 kernel: [295679.535461] CPU 2: hi: 186, btch: 31 usd: 30 Feb 18 12:26:27 ubu-2510 kernel: [295679.535463] CPU 3: hi: 186, btch: 31 usd: 11 Feb 18 12:26:27 ubu-2510 kernel: [295679.535469] active_anon:2261 inactive_anon:722 isolated_anon:64 Feb 18 12:26:27 ubu-2510 kernel: [295679.535470] active_ file:14 inactive_ file:57 isolated_ file:0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535471] unevictable:968905 dirty:0 writeback:789 unstable:0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535472] free:7001 slab_reclaimable:2589 slab_unreclaimable:4469 Feb 18 12:26:27 ubu-2510 kernel: [295679.535474] mapped:9898 shmem:0 pagetables:2869 bounce:0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535476] Node 0 DMA free:15876kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_ file:0kB inactive_ file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0k B present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclai mable? yes Feb 18 12:26:27 ubu-2510 kernel: [295679.535488] lowmem_reserve[]: 0 3000 4010 4010 Feb 18 12:26:27 ubu-2510 kernel: [295679.535492] Node 0 DMA32 free:10144kB min:6052kB low:7564kB high:9076kB active_anon:9044kB inactive_anon:2932kB active_ file:16kB inactive_ file:0kB unevictable:2953180kB isolated(anon): 256kB isolated(file):0kB present:3072160kB mlocked:2953180kB dirty:0kB writeback:3112kB mapped:60kB shmem:0kB slab_reclaimable:2708kB slab_unreclaimable:6756kB kernel_stack:112kB pagetables:6980kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2787 all_unreclaimable? no Feb 18 12:26:27 ubu-2510 kernel: [295679.535505] lowmem_reserve[]: 0 0 1010 1010 Feb 18 12:26:27 ubu-2510 kernel: [295679.535509] Node 0 Normal free:1984kB min:2036kB low:2544kB high:3052kB active_anon:0kB inactive_anon:0kB active_ file:40kB inactive_ file:296kB unevictable:922440kB isolated(anon):0kB i solated(file):0kB present:1034240kB mlocked:922440kB dirty:0kB writeback:44kB mapped:39532kB shmem:0kB slab_reclaimable:7648kB slab_unreclaimable:11112kB kernel_stack:1248kB pagetables:4496kB unstable:0kB bounce:0kB write back_tmp:0kB pages_scanned:0 all_unreclaimable? no Feb 18 12:26:27 ubu-2510 kernel: [295679.535521] lowmem_reserve[]: 0 0 0 0 Feb 18 12:26:27 ubu-2510 kernel: [295679.535524] Node 0 DMA: 1*4kB 0*8kB 2*16kB 3*32kB 2*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Feb 18 12:26:27 ubu-2510 kernel: [295679.535535] Node 0 DMA32: 1515*4kB 22*8kB 5*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10348kB Feb 18 12:26:27 ubu-2510 kernel: [295679.535545] Node 0 Normal: 176*4kB 2*8kB 3*16kB 1*32kB 3*64kB 4*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2016kB Feb 18 12:26:27 ubu-2510 kernel: [295679.535556] 11937 total pagecache pages Feb 18 12:26:27 ubu-2510 kernel: [295679.535558] 1819 pages in swap cache Feb 18 12:26:27 ubu-2510 kernel: [295679.535560] Swap cache stats: add 190442, delete 188623, find 23064/30183 Feb 18 12:26:27 ubu-2510 kernel: [295679.535562] Free swap = 5908408kB Feb 18 12:26:27 ubu-2510 kernel: [295679.535564] Total swap = 6430712kB Feb 18 12:26:27 ubu-2510 kernel: [295679.552112] 1048560 pages RAM Feb 18 12:26:27 ubu-2510 kernel: [295679.552117] 33577 pages reserved Feb 18 12:26:27 ubu-2510 kernel: [295679.552120] 24402 pages shared Feb 18 12:26:27 ubu-2510 kernel: [295679.552122] 996615 pages non-shared Feb 18 12:26:27 ubu-2510 kernel: [295679.552128] Out of memory: kill process 12976 (moxi) score 2623 or a child Feb 18 12:26:27 ubu-2510 kernel: [295679.554551] Killed process 12976 (moxi)
        Hide
        thuan Thuan Nguyen added a comment -

        I could not reproduce this bug in build 2.0.1-160

        Show
        thuan Thuan Nguyen added a comment - I could not reproduce this bug in build 2.0.1-160

          People

          • Assignee:
            ketaki Ketaki Gangal
            Reporter:
            thuan Thuan Nguyen
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes