Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10687

cbcollect_info: Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... Got exception: {error,badarg}

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 3.0
    • 3.0
    • tools
    • Security Level: Public
    • None
    • 3.0.0-523-rel
    • Untriaged
    • Unknown

    Description

      python scripts/collect_server_info.py i ubuntu_x6430_02-swap_rebalance-P1.ini
      Collecting logs from 10.3.121.93

      Waiting for another 15 seconds (time-out after 20 min)
      uname (uname -a) - OK
      Directory structure membase - previous versions (ls -lR /opt/membase /var/membase /var/opt/membase /etc/opt/membase) - Exit code 2
      Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK
      Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK
      Raw /proc/vmstat (cat /proc/vmstat) - OK
      Raw /proc/mounts (cat /proc/mounts) - OK
      Raw /proc/partitions (cat /proc/partitions) - OK
      Raw /proc/diskstats (cat /proc/diskstats) - OK
      Raw /proc/interrupts (cat /proc/interrupts) - OK
      Swap configuration (free -t) - OK
      Swap configuration (swapon -s) - OK
      Kernel modules (lsmod) - OK
      Distro version (cat /etc/redhat-release) - Exit code 1
      Distro version (lsb_release -a) - OK
      Installed software (rpm -qa) - Exit code 127
      Installed software (COLUMNS=300 dpkg -l) - OK
      Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127
      Core dump settings (find /proc/sys/kernel -type f -name 'core' -print -exec cat '{}' ';') - OK
      sysctl settings (sysctl -a) - OK
      relevant lsof output (echo moxi memcached vbucketmigrator beam couch_compact godu sigar_port | xargs -n1 pgrep | xargs -n1 -r – lsof -n -p) - OK
      LVM info (lvdisplay) - OK
      LVM info (vgdisplay) - OK
      LVM info (pvdisplay) - OK
      Network configuration (ifconfig -a) - OK
      Taking sample 2 after 10.000000 seconds -
      OK
      Network configuration (echo link addr neigh rule route netns | xargs -n1 – sh -x -c 'ip $1 list' --) - Exit code 124
      Network status (netstat -anp || netstat -an) - OK
      Network routing table (netstat -rn) - OK
      Arp cache (arp -na) - OK
      Filesystem (df -ha) - OK
      System activity reporter (sar 1 10) - Exit code 127
      System paging activity (vmstat 1 10) - OK
      System uptime (uptime) - OK
      couchbase user definition (getent passwd couchbase) - OK
      couchbase user limits (su couchbase -c "ulimit -a") - OK
      membase user definition (getent passwd membase) - OK
      couchbase user limits (su couchbase -c "ulimit -a") - OK
      membase user limits (su membase -c "ulimit -a") - OK
      Interrupt status (intrstat 1 10) - Exit code 127
      Processor status (mpstat 1 10) - Exit code 127
      System log (cat /var/adm/messages) - Exit code 1
      Raw /proc/uptime (cat /proc/uptime) - OK
      All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2
      Relevant proc data ((pgrep moxi; pgrep beam.smp; pgrep memcached; pgrep couch_compact; pgrep sigar_port ; pgrep godu) | xargs -n1 – sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/sched; echo' --) - OK
      NUMA data (numactl --hardware) - Exit code 127
      NUMA data (numactl --show) - Exit code 127
      NUMA data (cat /sys/devices/system/node/node*/numastat) - OK
      Kernel log buffer (dmesg) - OK
      Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1
      Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1
      Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1
      Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1
      Network statistics (netstat -s) - Exit code 1
      Full raw netstat (cat /proc/net/netstat) - OK
      CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 – sh -c 'echo $1; cat $1' --) - Exit code 123
      Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs...
      Got exception:

      {error,badarg}
      [{lists,keyfind,
      [creds,1,
      {"Administrator",
      {password,{<<25,150,211,6,134,89,49,233,149,95,210,95,36,119,65,145>>, <<211,187,123,81,193,128,102,117,179,81,6,34,35,220, 200,54,86,157,217,238>>}}}]},
      {'dump-guts__escript__1396__90453__375879',search_config,3},
      {'dump-guts__escript__1396__90453__375879',extract_rest_creds,1},
      {'dump-guts__escript__1396__90453__375879',main_with_everything,4},
      {'dump-guts__escript__1396__90453__375879',main,1},
      {escript,run,2},
      {escript,start,1},
      {init,start_it,1}]

      Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs...
      Got exception: {error,badarg}

      [{lists,keyfind,
      [creds,1,
      {"Administrator",
      {password,

      {<<25,150,211,6,134,89,49,233,149,95,210,95,36,119,65,145>>, <<211,187,123,81,193,128,102,117,179,81,6,34,35,220, 200,54,86,157,217,238>>}}}]},
      {'dump-guts__escript__1396__90454__62803',search_config,3},
      {'dump-guts__escript__1396__90454__62803',extract_rest_creds,1},
      {'dump-guts__escript__1396__90454__62803',main_with_everything,4},
      {'dump-guts__escript__1396__90454__62803',main,1},
      {escript,run,2},
      {escript,start,1},
      {init,start_it,1}]

      Checking for server guts in /root/Library/Application Support/Couchbase/var/lib/couchbase/initargs...
      Failed to read initargs: {error,enoent}

      Downloading zipped logs from 10.3.121.93
      Waiting for another 15 seconds (time-out after 20 min)
      collect info done
      Collecting logs from 10.3.121.94

      Waiting for another 15 seconds (time-out after 20 min)
      uname (uname -a) - OK
      Directory structure membase - previous versions (ls -lR /opt/membase /var/membase /var/opt/membase /etc/opt/membase) - Exit code 2
      Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK
      Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK
      Raw /proc/vmstat (cat /proc/vmstat) - OK
      Raw /proc/mounts (cat /proc/mounts) - OK
      Raw /proc/partitions (cat /proc/partitions) - OK
      Raw /proc/diskstats (cat /proc/diskstats) - OK
      Raw /proc/interrupts (cat /proc/interrupts) - OK
      Swap configuration (free -t) - OK
      Swap configuration (swapon -s) - OK
      Kernel modules (lsmod) - OK
      Distro version (cat /etc/redhat-release) - Exit code 1
      Distro version (lsb_release -a) - OK
      Installed software (rpm -qa) - Exit code 127
      Installed software (COLUMNS=300 dpkg -l) - OK
      Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127
      Core dump settings (find /proc/sys/kernel -type f -name 'core' -print -exec cat '{}' ';') - OK
      sysctl settings (sysctl -a) - OK
      relevant lsof output (echo moxi memcached vbucketmigrator beam couch_compact godu sigar_port | xargs -n1 pgrep | xargs -n1 -r – lsof -n -p) - OK
      LVM info (lvdisplay) - OK
      LVM info (vgdisplay) - OK
      LVM info (pvdisplay) - OK
      Network configuration (ifconfig -a) - OK
      Taking sample 2 after 10.000000 seconds -
      OK
      Network configuration (echo link addr neigh rule route netns | xargs -n1 – sh -x -c 'ip $1 list' --) - Exit code 124
      Network status (netstat -anp || netstat -an) - OK
      Network routing table (netstat -rn) - OK
      Arp cache (arp -na) - OK
      Filesystem (df -ha) - OK
      System activity reporter (sar 1 10) - Exit code 127
      System paging activity (vmstat 1 10) - OK
      System uptime (uptime) - OK
      couchbase user definition (getent passwd couchbase) - OK
      couchbase user limits (su couchbase -c "ulimit -a") - OK
      membase user definition (getent passwd membase) - OK
      couchbase user limits (su couchbase -c "ulimit -a") - OK
      membase user limits (su membase -c "ulimit -a") - OK
      Interrupt status (intrstat 1 10) - Exit code 127
      Processor status (mpstat 1 10) - Exit code 127
      System log (cat /var/adm/messages) - Exit code 1
      Raw /proc/uptime (cat /proc/uptime) - OK
      All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2
      Relevant proc data ((pgrep moxi; pgrep beam.smp; pgrep memcached; pgrep couch_compact; pgrep sigar_port ; pgrep godu) | xargs -n1 – sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/sched; echo' --) - OK
      NUMA data (numactl --hardware) - Exit code 127
      NUMA data (numactl --show) - Exit code 127
      NUMA data (cat /sys/devices/system/node/node*/numastat) - OK
      Kernel log buffer (dmesg) - OK
      Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1
      Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1
      Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1
      Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1
      Network statistics (netstat -s) - Exit code 1
      Full raw netstat (cat /proc/net/netstat) - OK
      CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 – sh -c 'echo $1; cat $1' --) - Exit code 123
      Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs...
      Got exception: {error,badarg}
      [{lists,keyfind,
      [creds,1,
      {"Administrator",
      {password,{<<25,150,211,6,134,89,49,233,149,95,210,95,36,119,65,145>>, <<211,187,123,81,193,128,102,117,179,81,6,34,35,220, 200,54,86,157,217,238>>}

      }}]},

      {'dump-guts__escript__1396__90483__166163',search_config,3}

      ,

      {'dump-guts__escript__1396__90483__166163',extract_rest_creds,1}

      ,

      {'dump-guts__escript__1396__90483__166163',main_with_everything,4}

      ,

      {'dump-guts__escript__1396__90483__166163',main,1}

      ,

      {escript,run,2},
      {escript,start,1},
      {init,start_it,1}]

      Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs...
      Got exception: {error,badarg}
      [{lists,keyfind,
      [creds,1,
      {"Administrator",
      {password,{<<25,150,211,6,134,89,49,233,149,95,210,95,36,119,65,145>>, <<211,187,123,81,193,128,102,117,179,81,6,34,35,220, 200,54,86,157,217,238>>}}}]},
      {'dump-guts__escript__1396__90483__861363',search_config,3},
      {'dump-guts__escript__1396__90483__861363',extract_rest_creds,1},
      {'dump-guts__escript__1396__90483__861363',main_with_everything,4},
      {'dump-guts__escript__1396__90483__861363',main,1},
      {escript,run,2}

      ,

      {escript,start,1}

      ,

      {init,start_it,1}

      ]

      Checking for server guts in /root/Library/Application Support/Couchbase/var/lib/couchbase/initargs...
      Failed to read initargs:

      {error,enoent}

      Downloading zipped logs from 10.3.121.94
      Waiting for another 15 seconds (time-out after 20 min)
      collect info done
      Collecting logs from 10.3.121.98

      https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.93-3292014-353-diag.zip
      https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.94-3292014-354-diag.zip
      https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.95-3292014-355-diag.zip
      https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.96-3292014-355-diag.zip
      https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.97-3292014-356-diag.zip
      https://s3.amazonaws.com/bugdb/jira/MB-10686/42a396cc/10.3.121.98-3292014-354-diag.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty