Details
Description
python scripts/collect_server_info.py i ubuntu_x6430_02-swap_rebalance-P1.ini
Collecting logs from 10.3.121.93
Waiting for another 15 seconds (time-out after 20 min)
uname (uname -a) - OK
Directory structure membase - previous versions (ls -lR /opt/membase /var/membase /var/opt/membase /etc/opt/membase) - Exit code 2
Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK
Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK
Raw /proc/vmstat (cat /proc/vmstat) - OK
Raw /proc/mounts (cat /proc/mounts) - OK
Raw /proc/partitions (cat /proc/partitions) - OK
Raw /proc/diskstats (cat /proc/diskstats) - OK
Raw /proc/interrupts (cat /proc/interrupts) - OK
Swap configuration (free -t) - OK
Swap configuration (swapon -s) - OK
Kernel modules (lsmod) - OK
Distro version (cat /etc/redhat-release) - Exit code 1
Distro version (lsb_release -a) - OK
Installed software (rpm -qa) - Exit code 127
Installed software (COLUMNS=300 dpkg -l) - OK
Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127
Core dump settings (find /proc/sys/kernel -type f -name 'core' -print -exec cat '{}' ';') - OK
sysctl settings (sysctl -a) - OK
relevant lsof output (echo moxi memcached vbucketmigrator beam couch_compact godu sigar_port | xargs -n1 pgrep | xargs -n1 -r – lsof -n -p) - OK
LVM info (lvdisplay) - OK
LVM info (vgdisplay) - OK
LVM info (pvdisplay) - OK
Network configuration (ifconfig -a) - OK
Taking sample 2 after 10.000000 seconds -
OK
Network configuration (echo link addr neigh rule route netns | xargs -n1 – sh -x -c 'ip $1 list' --) - Exit code 124
Network status (netstat -anp || netstat -an) - OK
Network routing table (netstat -rn) - OK
Arp cache (arp -na) - OK
Filesystem (df -ha) - OK
System activity reporter (sar 1 10) - Exit code 127
System paging activity (vmstat 1 10) - OK
System uptime (uptime) - OK
couchbase user definition (getent passwd couchbase) - OK
couchbase user limits (su couchbase -c "ulimit -a") - OK
membase user definition (getent passwd membase) - OK
couchbase user limits (su couchbase -c "ulimit -a") - OK
membase user limits (su membase -c "ulimit -a") - OK
Interrupt status (intrstat 1 10) - Exit code 127
Processor status (mpstat 1 10) - Exit code 127
System log (cat /var/adm/messages) - Exit code 1
Raw /proc/uptime (cat /proc/uptime) - OK
All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2
Relevant proc data ((pgrep moxi; pgrep beam.smp; pgrep memcached; pgrep couch_compact; pgrep sigar_port ; pgrep godu) | xargs -n1 – sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/sched; echo' --) - OK
NUMA data (numactl --hardware) - Exit code 127
NUMA data (numactl --show) - Exit code 127
NUMA data (cat /sys/devices/system/node/node*/numastat) - OK
Kernel log buffer (dmesg) - OK
Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1
Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1
Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1
Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1
Network statistics (netstat -s) - Exit code 1
Full raw netstat (cat /proc/net/netstat) - OK
CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 – sh -c 'echo $1; cat $1' --) - Exit code 123
Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs...
Got exception:
[{lists,keyfind,
[creds,1,
{"Administrator",
{password,{<<25,150,211,6,134,89,49,233,149,95,210,95,36,119,65,145>>, <<211,187,123,81,193,128,102,117,179,81,6,34,35,220, 200,54,86,157,217,238>>}}}]},
{'dump-guts__escript__1396__90453__375879',search_config,3},
{'dump-guts__escript__1396__90453__375879',extract_rest_creds,1},
{'dump-guts__escript__1396__90453__375879',main_with_everything,4},
{'dump-guts__escript__1396__90453__375879',main,1},
{escript,run,2},
{escript,start,1},
{init,start_it,1}]
Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs...
Got exception: {error,badarg}
[{lists,keyfind,
[creds,1,
{"Administrator",
{password,
{'dump-guts__escript__1396__90454__62803',search_config,3},
{'dump-guts__escript__1396__90454__62803',extract_rest_creds,1},
{'dump-guts__escript__1396__90454__62803',main_with_everything,4},
{'dump-guts__escript__1396__90454__62803',main,1},
{escript,run,2},
{escript,start,1},
{init,start_it,1}]
Checking for server guts in /root/Library/Application Support/Couchbase/var/lib/couchbase/initargs...
Failed to read initargs: {error,enoent}
Downloading zipped logs from 10.3.121.93
Waiting for another 15 seconds (time-out after 20 min)
collect info done
Collecting logs from 10.3.121.94
Waiting for another 15 seconds (time-out after 20 min)
uname (uname -a) - OK
Directory structure membase - previous versions (ls -lR /opt/membase /var/membase /var/opt/membase /etc/opt/membase) - Exit code 2
Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK
Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK
Raw /proc/vmstat (cat /proc/vmstat) - OK
Raw /proc/mounts (cat /proc/mounts) - OK
Raw /proc/partitions (cat /proc/partitions) - OK
Raw /proc/diskstats (cat /proc/diskstats) - OK
Raw /proc/interrupts (cat /proc/interrupts) - OK
Swap configuration (free -t) - OK
Swap configuration (swapon -s) - OK
Kernel modules (lsmod) - OK
Distro version (cat /etc/redhat-release) - Exit code 1
Distro version (lsb_release -a) - OK
Installed software (rpm -qa) - Exit code 127
Installed software (COLUMNS=300 dpkg -l) - OK
Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127
Core dump settings (find /proc/sys/kernel -type f -name 'core' -print -exec cat '{}' ';') - OK
sysctl settings (sysctl -a) - OK
relevant lsof output (echo moxi memcached vbucketmigrator beam couch_compact godu sigar_port | xargs -n1 pgrep | xargs -n1 -r – lsof -n -p) - OK
LVM info (lvdisplay) - OK
LVM info (vgdisplay) - OK
LVM info (pvdisplay) - OK
Network configuration (ifconfig -a) - OK
Taking sample 2 after 10.000000 seconds -
OK
Network configuration (echo link addr neigh rule route netns | xargs -n1 – sh -x -c 'ip $1 list' --) - Exit code 124
Network status (netstat -anp || netstat -an) - OK
Network routing table (netstat -rn) - OK
Arp cache (arp -na) - OK
Filesystem (df -ha) - OK
System activity reporter (sar 1 10) - Exit code 127
System paging activity (vmstat 1 10) - OK
System uptime (uptime) - OK
couchbase user definition (getent passwd couchbase) - OK
couchbase user limits (su couchbase -c "ulimit -a") - OK
membase user definition (getent passwd membase) - OK
couchbase user limits (su couchbase -c "ulimit -a") - OK
membase user limits (su membase -c "ulimit -a") - OK
Interrupt status (intrstat 1 10) - Exit code 127
Processor status (mpstat 1 10) - Exit code 127
System log (cat /var/adm/messages) - Exit code 1
Raw /proc/uptime (cat /proc/uptime) - OK
All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2
Relevant proc data ((pgrep moxi; pgrep beam.smp; pgrep memcached; pgrep couch_compact; pgrep sigar_port ; pgrep godu) | xargs -n1 – sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/sched; echo' --) - OK
NUMA data (numactl --hardware) - Exit code 127
NUMA data (numactl --show) - Exit code 127
NUMA data (cat /sys/devices/system/node/node*/numastat) - OK
Kernel log buffer (dmesg) - OK
Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1
Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1
Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1
Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1
Network statistics (netstat -s) - Exit code 1
Full raw netstat (cat /proc/net/netstat) - OK
CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 – sh -c 'echo $1; cat $1' --) - Exit code 123
Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs...
Got exception: {error,badarg}
[{lists,keyfind,
[creds,1,
{"Administrator",
{password,{<<25,150,211,6,134,89,49,233,149,95,210,95,36,119,65,145>>, <<211,187,123,81,193,128,102,117,179,81,6,34,35,220, 200,54,86,157,217,238>>}
}}]},
,
,
,
,
{escript,start,1},
{init,start_it,1}]
Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs...
Got exception: {error,badarg}
[{lists,keyfind,
[creds,1,
{"Administrator",
{password,{<<25,150,211,6,134,89,49,233,149,95,210,95,36,119,65,145>>, <<211,187,123,81,193,128,102,117,179,81,6,34,35,220, 200,54,86,157,217,238>>}}}]},
{'dump-guts__escript__1396__90483__861363',search_config,3},
{'dump-guts__escript__1396__90483__861363',extract_rest_creds,1},
{'dump-guts__escript__1396__90483__861363',main_with_everything,4},
{'dump-guts__escript__1396__90483__861363',main,1},
{escript,run,2}
,
,
]
Checking for server guts in /root/Library/Application Support/Couchbase/var/lib/couchbase/initargs...
Failed to read initargs:
Downloading zipped logs from 10.3.121.94
Waiting for another 15 seconds (time-out after 20 min)
collect info done
Collecting logs from 10.3.121.98
https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.93-3292014-353-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.94-3292014-354-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.95-3292014-355-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.96-3292014-355-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.97-3292014-356-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.98-3292014-354-diag.zip