[root@centos-64-x64 ~]# dmesg
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-358.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Fri Feb 22 00:31:26 UTC 2013
Command line: ro root=/dev/mapper/VolGroup00-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD console=hvc0  KEYTABLE=us rd_LVM_LV=VolGroup00/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=VolGroup00/lv_swap rd_NO_DM rhgb quiet graphical utf8
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
ACPI in unprivileged domain disabled
released 0 pages of unused memory
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 00000005dc000000 (usable)
DMI not present or invalid.
e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
last_pfn = 0x5dc000 max_arch_pfn = 0x400000000
last_pfn = 0x100000 max_arch_pfn = 0x400000000
initial memory mapped : 0 - 20000000
init_memory_mapping: 0000000000000000-0000000100000000
 0000000000 - 0100000000 page 4k
kernel direct mapping tables up to 100000000 @ 100000-905000
init_memory_mapping: 0000000100000000-00000005dc000000
 0100000000 - 05dc000000 page 4k
kernel direct mapping tables up to 5dc000000 @ 7960000-a859000
RAMDISK: 0203c000 - 04a3c000
No NUMA configuration found
Faking a node at 0000000000000000-00000005dc000000
Bootmem setup node 0 0000000000000000-00000005dc000000
  NODE_DATA [0000000000008000 - 000000000003bfff]
  bootmap [00000000008c1000 -  000000000097c7ff] pages bc
(8 early reservations) ==> bootmem [0000000000 - 05dc000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [000791f000 - 0007960000]   XEN PAGETABLES ==> [000791f000 - 0007960000]
  #2 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #3 [0001000000 - 000201b0a4]    TEXT DATA BSS ==> [0001000000 - 000201b0a4]
  #4 [000203c000 - 0004a3c000]          RAMDISK ==> [000203c000 - 0004a3c000]
  #5 [0004a3c000 - 000791f000]   XEN START INFO ==> [0004a3c000 - 000791f000]
  #6 [0000100000 - 00008c1000]          PGTABLE ==> [0000100000 - 00008c1000]
  #7 [0007960000 - 000a054000]          PGTABLE ==> [0007960000 - 000a054000]
Reserving 130MB of memory at 176MB for crashkernel (System RAM: 24000MB)
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x005dc000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x000000a0
    0: 0x00000100 -> 0x005dc000
On node 0 totalpages: 6143903
  DMA zone: 56 pages used for memmap
  DMA zone: 1988 pages reserved
  DMA zone: 1955 pages, LIFO batch:0
  DMA32 zone: 14280 pages used for memmap
  DMA32 zone: 1030200 pages, LIFO batch:31
  Normal zone: 69664 pages used for memmap
  Normal zone: 5025760 pages, LIFO batch:31
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 8 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
nr_irqs_gsi: 16
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
PCI: Warning: Cannot find a gap in the 32bit address range
PCI: Unassigned devices with 32bit resource registers may break!
Allocating PCI resources starting at 5dc100000 (gap: 5dc100000:400000)
Booting paravirtualized kernel on Xen
Xen version: 4.1.5 (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff880028050000 s94552 r8192 d24232 u126976
pcpu-alloc: s94552 r8192 d24232 u126976 alloc=31*4096
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5 [0] 6 [0] 7 
trying to map vcpu_info 0 at ffff88002805b020, mfn 7ea80c, offset 32
cpu 0 using vcpu_info at ffff88002805b020
trying to map vcpu_info 1 at ffff88002807a020, mfn 11e475d, offset 32
cpu 1 using vcpu_info at ffff88002807a020
trying to map vcpu_info 2 at ffff880028099020, mfn 7ea7ed, offset 32
cpu 2 using vcpu_info at ffff880028099020
trying to map vcpu_info 3 at ffff8800280b8020, mfn 11e473e, offset 32
cpu 3 using vcpu_info at ffff8800280b8020
trying to map vcpu_info 4 at ffff8800280d7020, mfn 7ea7ce, offset 32
cpu 4 using vcpu_info at ffff8800280d7020
trying to map vcpu_info 5 at ffff8800280f6020, mfn 11e471f, offset 32
cpu 5 using vcpu_info at ffff8800280f6020
trying to map vcpu_info 6 at ffff880028115020, mfn 7ea7af, offset 32
cpu 6 using vcpu_info at ffff880028115020
trying to map vcpu_info 7 at ffff880028134020, mfn 11e4700, offset 32
cpu 7 using vcpu_info at ffff880028134020
Xen: using vcpu_info placement
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 6057915
Policy zone: Normal
Kernel command line: ro root=/dev/mapper/VolGroup00-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD console=hvc0  KEYTABLE=us rd_LVM_LV=VolGroup00/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=130M@0M rd_LVM_LV=VolGroup00/lv_swap rd_NO_DM rhgb quiet graphical utf8
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture...
No AGP bridge found
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 23881780k/24576000k available (5220k kernel code, 388k absent, 693832k reserved, 7121k data, 1264k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:336
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 98566144 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Xen: using vcpuop timer interface
installing Xen timer for CPU 0
  alloc irq_desc for 335 on node 0
  alloc kstat_irqs on node 0
Detected 1995.243 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.48 BogoMIPS (lpj=1995243)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
alternatives: switching to unfair spinlock
SMP alternatives: switching to UP code
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 21428 entries in 85 pages
  alloc irq_desc for 334 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 333 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 332 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 331 on node 0
  alloc kstat_irqs on node 0
Performance Events: unsupported p6 CPU model 62 no PMU driver, software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
installing Xen timer for CPU 1
  alloc irq_desc for 330 on node 0
  alloc kstat_irqs on node 0
SMP alternatives: switching to SMP code
  alloc irq_desc for 329 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 328 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 327 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 326 on node 0
  alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 2
  alloc irq_desc for 325 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 324 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 323 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 322 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 321 on node 0
  alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 3
  alloc irq_desc for 320 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 319 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 318 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 317 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 316 on node 0
  alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 4
  alloc irq_desc for 315 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 314 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 313 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 312 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 311 on node 0
  alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 5
  alloc irq_desc for 310 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 309 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 308 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 307 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 306 on node 0
  alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 6
  alloc irq_desc for 305 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 304 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 303 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 302 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 301 on node 0
  alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
installing Xen timer for CPU 7
  alloc irq_desc for 300 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 299 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 298 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 297 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 296 on node 0
  alloc kstat_irqs on node 0
CPU: CPU feature constant_tsc disabled on xen guest
CPU: Unsupported number of siblings 32
Brought up 8 CPUs
sizeof(vma)=200 bytes
sizeof(page)=56 bytes
sizeof(inode)=592 bytes
sizeof(dentry)=192 bytes
sizeof(ext3inode)=800 bytes
sizeof(buffer_head)=104 bytes
sizeof(skbuff)=232 bytes
sizeof(task_struct)=2648 bytes
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
  alloc irq_desc for 295 on node 0
  alloc kstat_irqs on node 0
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
last_pfn = 0x5dc000 max_arch_pfn = 0x400000000
vgaarb: loaded
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
PCI: max bus depth: 0 pci_try_num: 1
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 43008k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1432664354.045:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 32768
SELinux:  Registering netfilter hooks
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key 6AA10540A705B299
- User ID: CentOS (Kernel Module GPG key)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
ipmi message handler version 39.2
IPMI System Interface driver.
ipmi_si: Adding default-specified kcs state machine
ipmi_si: Trying default-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0
Could not set up I/O space
Trying to free nonexistent resource <0000000000000ca2-0000000000000ca2>
Trying to free nonexistent resource <0000000000000ca3-0000000000000ca3>
ipmi_si: Adding default-specified smic state machine
ipmi_si: Trying default-specified smic state machine at i/o address 0xca9, slave address 0x0, irq 0
Could not set up I/O space
Trying to free nonexistent resource <0000000000000ca9-0000000000000ca9>
Trying to free nonexistent resource <0000000000000caa-0000000000000caa>
Trying to free nonexistent resource <0000000000000cab-0000000000000cab>
ipmi_si: Adding default-specified bt state machine
ipmi_si: Trying default-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0
Could not set up I/O space
Trying to free nonexistent resource <00000000000000e4-00000000000000e4>
Trying to free nonexistent resource <00000000000000e5-00000000000000e5>
Trying to free nonexistent resource <00000000000000e6-00000000000000e6>
ipmi_si: Unable to find any System Interface(s)
  alloc irq_desc for 294 on node 0
  alloc kstat_irqs on node 0
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos: probe of rtc_cmos failed with error -16
cpuidle: using governor ladder
cpuidle: using governor menu
EFI Variables Facility v0.08 2004-May-17
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
XENBUS: Device with no driver: device/vbd/51744
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/vbd/51728
XENBUS: Device with no driver: device/vif/0
XENBUS: Device with no driver: device/vif/1
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 1264k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 904k freed
Freeing unused kernel memory: 1676k freed
dracut: dracut-004-303.el6
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com
udev: starting version 147
dracut: Starting plymouth daemon
xlblk_init: register_blkdev major: 202 
  alloc irq_desc for 293 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 292 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 291 on node 0
  alloc kstat_irqs on node 0
blkfront: xvdc: barriers disabled
 xvdc: unknown partition table
blkfront: xvda: barriers disabled
 xvda: xvda1 xvda2
blkfront: xvdb: barriers disabled
 xvdb: unknown partition table
dracut: Scanning devices xvda2 xvdb xvdc  for LVM logical volumes VolGroup00/lv_root VolGroup00/lv_swap 
dracut: inactive '/dev/VolGroup02/Index' [170.00 GiB] inherit
dracut: inactive '/dev/VolGroup01/Data' [250.00 GiB] inherit
dracut: inactive '/dev/VolGroup00/lv_root' [34.18 GiB] inherit
dracut: inactive '/dev/VolGroup00/lv_swap' [10.33 GiB] inherit
EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: 
dracut: Mounted root filesystem /dev/mapper/VolGroup00-lv_root
SELinux:  Disabled at runtime.
SELinux:  Unregistering netfilter hooks
type=1404 audit(1432664355.252:2): selinux=0 auid=4294967295 ses=4294967295
dracut: 
dracut: Switching root
udev: starting version 147
Initialising Xen virtual ethernet driver.
  alloc irq_desc for 290 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 289 on node 0
  alloc kstat_irqs on node 0
EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: 
EXT4-fs (dm-3): warning: maximal mount count reached, running e2fsck is recommended
EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: 
EXT4-fs (dm-2): warning: maximal mount count reached, running e2fsck is recommended
EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: 
Adding 10829816k swap on /dev/mapper/VolGroup00-lv_swap.  Priority:-1 extents:1 across:10829816k SS
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
beam.smp cpuset=/ mems_allowed=0
Pid: 1192, comm: beam.smp Not tainted 2.6.32-358.el6.x86_64 #1
Call Trace:
 [<ffffffff810cb5d1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
 [<ffffffff8111cd10>] ? dump_header+0x90/0x1b0
 [<ffffffff8111d192>] ? oom_kill_process+0x82/0x2a0
 [<ffffffff8111d0d1>] ? select_bad_process+0xe1/0x120
 [<ffffffff8111d5d0>] ? out_of_memory+0x220/0x3c0
 [<ffffffff8112c27c>] ? __alloc_pages_nodemask+0x8ac/0x8d0
 [<ffffffff8116097a>] ? alloc_pages_vma+0x9a/0x150
 [<ffffffff81143cfb>] ? handle_pte_fault+0x76b/0xb50
 [<ffffffff8100c1fb>] ? xen_hypervisor_callback+0x1b/0x20
 [<ffffffff8100ba1d>] ? retint_restore_args+0x5/0x6
 [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1010
 [<ffffffff81004a49>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
 [<ffffffff8114431a>] ? handle_mm_fault+0x23a/0x310
 [<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff81007ca2>] ? check_events+0x12/0x20
 [<ffffffff810474c9>] ? __do_page_fault+0x139/0x480
 [<ffffffff810032bb>] ? xen_write_msr_safe+0x6b/0x90
 [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
 [<ffffffff81006113>] ? __xen_write_cr3+0x123/0x170
 [<ffffffff8105ab8f>] ? finish_task_switch+0x4f/0xe0
 [<ffffffff8150d6e0>] ? thread_return+0x4e/0x76e
 [<ffffffff81283408>] ? call_rwsem_wake+0x18/0x30
 [<ffffffff8151311e>] ? do_page_fault+0x3e/0xa0
 [<ffffffff815104d5>] ? page_fault+0x25/0x30
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
CPU    4: hi:    0, btch:   1 usd:   0
CPU    5: hi:    0, btch:   1 usd:   0
CPU    6: hi:    0, btch:   1 usd:   0
CPU    7: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd:  30
CPU    1: hi:  186, btch:  31 usd:   0
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:   0
CPU    5: hi:  186, btch:  31 usd:   0
CPU    6: hi:  186, btch:  31 usd:   0
CPU    7: hi:  186, btch:  31 usd:   0
Node 0 Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:   0
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:  11
CPU    5: hi:  186, btch:  31 usd:   0
CPU    6: hi:  186, btch:  31 usd:   0
CPU    7: hi:  186, btch:  31 usd:   0
active_anon:5368617 inactive_anon:500826 isolated_anon:96
 active_file:249 inactive_file:207 isolated_file:0
 unevictable:1987 dirty:0 writeback:0 unstable:0
 free:26472 slab_reclaimable:5310 slab_unreclaimable:8911
 mapped:910 shmem:0 pagetables:18113 bounce:0
Node 0 DMA free:7824kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7820kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 4024 23656 23656
Node 0 DMA32 free:81920kB min:3348kB low:4184kB high:5020kB active_anon:2864604kB inactive_anon:572676kB active_file:16kB inactive_file:0kB unevictable:564kB isolated(anon):0kB isolated(file):0kB present:4120800kB mlocked:564kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:2296kB slab_unreclaimable:1104kB kernel_stack:72kB pagetables:12104kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 19631 19631
Node 0 Normal free:16144kB min:16332kB low:20412kB high:24496kB active_anon:18609864kB inactive_anon:1430628kB active_file:980kB inactive_file:936kB unevictable:7384kB isolated(anon):384kB isolated(file):0kB present:20103040kB mlocked:7384kB dirty:0kB writeback:0kB mapped:3644kB shmem:0kB slab_reclaimable:18944kB slab_unreclaimable:34540kB kernel_stack:3576kB pagetables:60348kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:128 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 2*4kB 1*8kB 2*16kB 1*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7824kB
Node 0 DMA32: 112*4kB 279*8kB 468*16kB 298*32kB 210*64kB 109*128kB 56*256kB 18*512kB 1*1024kB 3*2048kB 1*4096kB = 81912kB
Node 0 Normal: 3116*4kB 41*8kB 5*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 16968kB
56629 total pagecache pages
55290 pages in swap cache
Swap cache stats: add 6486944, delete 6431654, find 1029279/1468118
Free swap  = 0kB
Total swap = 10829816kB
6143999 pages RAM
161841 pages reserved
7521 pages shared
5949515 pages non-shared
[ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[  407]     0   407     2660       61   6     -17         -1000 udevd
[  618]     0   618     2659       67   6     -17         -1000 udevd
[  621]     0   621     2659       59   1     -17         -1000 udevd
[  885]     0   885     2279       32   1       0             0 dhclient
[  923]     0   923     6909      128   1     -17         -1000 auditd
[  939]     0   939    63855      179   0       0             0 rsyslogd
[  960]     0   960     2825      211   5       0             0 xe-daemon
[ 1074]   498  1074     2706       70   4       0             0 epmd
[ 1110]   498  1110   332544     5954   2       0             0 beam.smp
[ 1150]     0  1150    16099       92   6     -17         -1000 sshd
[ 1159]    38  1159     7679      227   1       0             0 ntpd
[ 1168]   498  1168   852017   329981   7       0             0 beam.smp
[ 1209]     0  1209     5150     1988   1       0             0 atop
[ 1221]     0  1221    29303      151   0       0             0 crond
[ 1230]   498  1230    26517      201   7       0             0 sh
[ 1231]   498  1231     1014      139   1       0             0 memsup
[ 1232]   498  1232     1014       88   0       0             0 cpu_sup
[ 1235]     0  1235    32027      197   0       0             0 chef-client
[ 1249]     0  1249     1018      134   0       0             0 agetty
[ 1250]     0  1250     1015      123   0       0             0 mingetty
[ 1252]     0  1252     1015      123   5       0             0 mingetty
[ 1254]     0  1254     1015      123   7       0             0 mingetty
[ 1256]     0  1256     1015      123   3       0             0 mingetty
[ 1258]     0  1258     1015      123   6       0             0 mingetty
[ 1260]     0  1260     1015      123   0       0             0 mingetty
[ 1261]   498  1261     2698      105   6       0             0 inet_gethost
[ 1262]   498  1262     2698       75   1       0             0 inet_gethost
[ 1283]   498  1283   262082     9286   2       0             0 beam.smp
[ 1335]   498  1335    26518      197   0       0             0 sh
[ 1336]   498  1336     1015      135   5       0             0 memsup
[ 1337]   498  1337     1015       88   4       0             0 cpu_sup
[ 1340]   498  1340     1171      177   3       0             0 godu
[ 1341]   498  1341    26517      183   0       0             0 sh
[ 1342]   498  1342     1170        9   6       0             0 godu
[ 1348]   498  1348     2698       78   6       0             0 inet_gethost
[ 1349]   498  1349    51235      224   3       0             0 moxi
[ 1357]   498  1357     2098      168   2       0             0 sigar_port
[ 1377]   498  1377    27740      155   7       0             0 saslauthd-port
[ 1392]   498  1392   228784      522   7       0             0 beam.smp
[ 1435]   498  1435  2020513   755155   1       0             0 memcached
[ 1448]   498  1448     2699      100   5       0             0 inet_gethost
[ 1449]   498  1449     2699       73   5       0             0 inet_gethost
[ 1461]   498  1461     1236       79   4       0             0 goport
[ 1463]   498  1463   372617   257486   2       0             0 projector
[ 1474]   498  1474     1236      150   1       0             0 goport
[ 1476]   498  1476  5828117  4256003   3       0             0 indexer
[ 1488]   498  1488     1236       97   2       0             0 goport
[ 1490]   498  1490   581712   199580   7       0             0 cbq-engine
[ 1502]   498  1502     1236      101   2       0             0 goport
[ 1504]   498  1504    96146      684   4       0             0 goxdcr
[ 7607]     0  7607    24539      259   2       0             0 sshd
[ 7609]     0  7609    27108      290   0       0             0 bash
[ 8221]     0  8221    46880      363   1       0             0 cbq
[ 8309]     0  8309     1017      128   1       0             0 sleep
Out of memory: Kill process 1476 (indexer) score 637 or sacrifice child
Killed process 1476, UID 498, (indexer) total-vm:23312468kB, anon-rss:17022788kB, file-rss:1224kB
beam.smp invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
beam.smp cpuset=/ mems_allowed=0
Pid: 1192, comm: beam.smp Not tainted 2.6.32-358.el6.x86_64 #1
Call Trace:
 [<ffffffff810cb5d1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
 [<ffffffff8111cd10>] ? dump_header+0x90/0x1b0
 [<ffffffff8111d192>] ? oom_kill_process+0x82/0x2a0
 [<ffffffff8111d0d1>] ? select_bad_process+0xe1/0x120
 [<ffffffff8111d5d0>] ? out_of_memory+0x220/0x3c0
 [<ffffffff8112c27c>] ? __alloc_pages_nodemask+0x8ac/0x8d0
 [<ffffffff8116097a>] ? alloc_pages_vma+0x9a/0x150
 [<ffffffff81143cfb>] ? handle_pte_fault+0x76b/0xb50
 [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81006e2b>] ? xen_set_pmd_hyper+0x8b/0xc0
 [<ffffffff81006ecb>] ? xen_set_pmd+0x6b/0xb0
 [<ffffffff81004a49>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
 [<ffffffff8114431a>] ? handle_mm_fault+0x23a/0x310
 [<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff810474c9>] ? __do_page_fault+0x139/0x480
 [<ffffffff810032bb>] ? xen_write_msr_safe+0x6b/0x90
 [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
 [<ffffffff81006113>] ? __xen_write_cr3+0x123/0x170
 [<ffffffff8105ab8f>] ? finish_task_switch+0x4f/0xe0
 [<ffffffff8150d6e0>] ? thread_return+0x4e/0x76e
 [<ffffffff81283408>] ? call_rwsem_wake+0x18/0x30
 [<ffffffff8151311e>] ? do_page_fault+0x3e/0xa0
 [<ffffffff815104d5>] ? page_fault+0x25/0x30
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
CPU    4: hi:    0, btch:   1 usd:   0
CPU    5: hi:    0, btch:   1 usd:   0
CPU    6: hi:    0, btch:   1 usd:   0
CPU    7: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd:   0
CPU    1: hi:  186, btch:  31 usd:   0
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:   0
CPU    4: hi:  186, btch:  31 usd:   7
CPU    5: hi:  186, btch:  31 usd:   0
CPU    6: hi:  186, btch:  31 usd:   0
CPU    7: hi:  186, btch:  31 usd:  30
Node 0 Normal per-cpu:
CPU    0: hi:  186, btch:  31 usd:  28
CPU    1: hi:  186, btch:  31 usd:   0
CPU    2: hi:  186, btch:  31 usd:   0
CPU    3: hi:  186, btch:  31 usd:  30
CPU    4: hi:  186, btch:  31 usd:  11
CPU    5: hi:  186, btch:  31 usd:   3
CPU    6: hi:  186, btch:  31 usd:   0
CPU    7: hi:  186, btch:  31 usd:  30
active_anon:5368792 inactive_anon:500915 isolated_anon:96
 active_file:249 inactive_file:14 isolated_file:0
 unevictable:1987 dirty:0 writeback:0 unstable:0
 free:26456 slab_reclaimable:5310 slab_unreclaimable:8911
 mapped:801 shmem:0 pagetables:18113 bounce:0
Node 0 DMA free:7824kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7820kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 4024 23656 23656
Node 0 DMA32 free:81780kB min:3348kB low:4184kB high:5020kB active_anon:2864468kB inactive_anon:572760kB active_file:16kB inactive_file:0kB unevictable:564kB isolated(anon):0kB isolated(file):0kB present:4120800kB mlocked:564kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:2296kB slab_unreclaimable:1104kB kernel_stack:72kB pagetables:12104kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:44 all_unreclaimable? yes
lowmem_reserve[]: 0 0 19631 19631
Node 0 Normal free:16220kB min:16332kB low:20412kB high:24496kB active_anon:18610700kB inactive_anon:1430900kB active_file:980kB inactive_file:312kB unevictable:7384kB isolated(anon):384kB isolated(file):0kB present:20103040kB mlocked:7384kB dirty:0kB writeback:0kB mapped:3208kB shmem:0kB slab_reclaimable:18944kB slab_unreclaimable:34540kB kernel_stack:3576kB pagetables:60348kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1952 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 2*4kB 1*8kB 2*16kB 1*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 7824kB
Node 0 DMA32: 112*4kB 274*8kB 464*16kB 298*32kB 210*64kB 109*128kB 56*256kB 18*512kB 1*1024kB 3*2048kB 1*4096kB = 81808kB
Node 0 Normal: 2929*4kB 15*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 15948kB
54478 total pagecache pages
53215 pages in swap cache
Swap cache stats: add 6487106, delete 6433891, find 1029304/1468160
Free swap  = 0kB
Total swap = 10829816kB
6143999 pages RAM
161841 pages reserved
7468 pages shared
5950560 pages non-shared
[ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[  407]     0   407     2660       61   6     -17         -1000 udevd
[  618]     0   618     2659       67   6     -17         -1000 udevd
[  621]     0   621     2659       59   1     -17         -1000 udevd
[  885]     0   885     2279       32   1       0             0 dhclient
[  923]     0   923     6909      128   1     -17         -1000 auditd
[  939]     0   939    63855      186   0       0             0 rsyslogd
[  960]     0   960     2825      211   5       0             0 xe-daemon
[ 1074]   498  1074     2706       70   4       0             0 epmd
[ 1110]   498  1110   332544     5954   2       0             0 beam.smp
[ 1150]     0  1150    16099       92   6     -17         -1000 sshd
[ 1159]    38  1159     7679      227   1       0             0 ntpd
[ 1168]   498  1168   852017   332259   7       0             0 beam.smp
[ 1209]     0  1209     5150     1988   1       0             0 atop
[ 1221]     0  1221    29303      151   0       0             0 crond
[ 1230]   498  1230    26517      201   7       0             0 sh
[ 1231]   498  1231     1014      139   1       0             0 memsup
[ 1232]   498  1232     1014       88   0       0             0 cpu_sup
[ 1235]     0  1235    32027      197   0       0             0 chef-client
[ 1249]     0  1249     1018      134   0       0             0 agetty
[ 1250]     0  1250     1015      123   0       0             0 mingetty
[ 1252]     0  1252     1015      123   5       0             0 mingetty
[ 1254]     0  1254     1015      123   7       0             0 mingetty
[ 1256]     0  1256     1015      123   3       0             0 mingetty
[ 1258]     0  1258     1015      123   6       0             0 mingetty
[ 1260]     0  1260     1015      123   0       0             0 mingetty
[ 1261]   498  1261     2698      105   6       0             0 inet_gethost
[ 1262]   498  1262     2698       75   1       0             0 inet_gethost
[ 1283]   498  1283   262082     9286   2       0             0 beam.smp
[ 1335]   498  1335    26518      197   0       0             0 sh
[ 1336]   498  1336     1015      135   5       0             0 memsup
[ 1337]   498  1337     1015       88   4       0             0 cpu_sup
[ 1340]   498  1340     1171      177   3       0             0 godu
[ 1341]   498  1341    26517      183   0       0             0 sh
[ 1342]   498  1342     1170        9   6       0             0 godu
[ 1348]   498  1348     2698       78   6       0             0 inet_gethost
[ 1349]   498  1349    51235      224   3       0             0 moxi
[ 1357]   498  1357     2098      168   2       0             0 sigar_port
[ 1377]   498  1377    27740      155   7       0             0 saslauthd-port
[ 1392]   498  1392   228784      522   7       0             0 beam.smp
[ 1435]   498  1435  2020513   755149   1       0             0 memcached
[ 1448]   498  1448     2699      100   5       0             0 inet_gethost
[ 1449]   498  1449     2699       73   5       0             0 inet_gethost
[ 1461]   498  1461     1236       80   4       0             0 goport
[ 1463]   498  1463   372617   257491   2       0             0 projector
[ 1474]   498  1474     1236      112   1       0             0 goport
[ 1488]   498  1488     1236       98   2       0             0 goport
[ 1490]   498  1490   581712   199586   7       0             0 cbq-engine
[ 1502]   498  1502     1236       95   2       0             0 goport
[ 1504]   498  1504    96146      699   0       0             0 goxdcr
[ 7607]     0  7607    24539      259   2       0             0 sshd
[ 7609]     0  7609    27108      290   0       0             0 bash
[ 8221]     0  8221    46880      363   1       0             0 cbq
[ 8309]     0  8309     1017      128   1       0             0 sleep
Out of memory: Kill process 1435 (memcached) score 221 or sacrifice child
Killed process 1435, UID 498, (memcached) total-vm:8082052kB, anon-rss:3019732kB, file-rss:864kB