cb_collect failing: ModuleNotFoundError: No module named 'distutils'

2021-09-21T19:57:44.999-07:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://127.0.0.1:8091/pools/default/buckets/travel-sample/scopes/@ensureManifest/1'

2021-09-21T19:57:45.013-07:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://127.0.0.1:8091/pools/default/buckets/travel-sample/scopes/@ensureManifest/1'

panic: runtime error: index out of range [37969] with length 1024goroutine 91 [running]:

github.com/couchbase/backup/value.(*BucketStats).ValidSeqNo(0xc000224bd0, 0x9451, 0x0)

        /home/couchbase/jenkins/workspace/couchbase-server-unix/backup/value/bucket_stats.go:250 +0x1bc

github.com/couchbase/backup/plan/services/data.(*ValidationFilter).Document(0xc0003fda58, 0x4e70, 0xc000e72000)

        /home/couchbase/jenkins/workspace/couchbase-server-unix/backup/plan/services/data/validation_filter.go:28 +0x36

github.com/couchbase/backup/jsondata.(*Source).Data.func1(0x0)

        /home/couchbase/jenkins/workspace/couchbase-server-unix/backup/jsondata/source.go:222 +0x123

github.com/couchbase/backup/jsondata.(*Source).Data.func2.1()

        /home/couchbase/jenkins/workspace/couchbase-server-unix/backup/jsondata/source.go:230 +0x22

github.com/couchbase/tools-common/hofp.(*Pool).work(0xc000184880)

        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.17/pkg/mod/github.com/couchbase/tools-common@v0.0.0-20210915152739-2338f690a8a7/hofp/pool.go:65 +0x11b

created by github.com/couchbase/tools-common/hofp.NewPool

        /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.17/pkg/mod/github.com/couchbase/tools-common@v0.0.0-20210915152739-2338f690a8a7/hofp/pool.go:46 +0x14c

Collection completedError: Unable to collect logs from the following nodes:172.23.104.92Node errors:172.23.104.92:8091Using temporary dir /tmp

Log Redaction () - OK

uname (uname -a) - OK

time and TZ (date; date -u) - OK

ntp time (ntpdate -q pool.ntp.org || nc time.nist.gov 13 || netcat time.nist.gov 13) - OK

ntp peers (ntpq -p) - OK

raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - Exit code 1

raw /etc/timezone (cat /etc/timezone) - Exit code 1

System Hardware (lshw -json || lshw) - Exit code 127

Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK

Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,comm,command) - OK

Raw /proc/buddyinfo (cat /proc/buddyinfo) - OK

Raw /proc/meminfo (cat /proc/meminfo) - OK

Raw /proc/pagetypeinfo (cat /proc/pagetypeinfo) - OK

Raw /proc/zoneinfo (cat /proc/zoneinfo) - OK

Raw /proc/vmstat (cat /proc/vmstat) - OK

Raw /proc/mounts (cat /proc/mounts) - OK

Raw /proc/partitions (cat /proc/partitions) - OK

Raw /proc/diskstats (cat /proc/diskstats; echo '') - OK

Taking sample 2 after 1.000000 seconds - OK

Taking sample 3 after 1.000000 seconds - OK

Taking sample 4 after 1.000000 seconds - OK

Taking sample 5 after 1.000000 seconds - OK

Taking sample 6 after 1.000000 seconds - OK

Taking sample 7 after 1.000000 seconds - OK

Taking sample 8 after 1.000000 seconds - OK

Taking sample 9 after 1.000000 seconds - OK

Taking sample 10 after 1.000000 seconds - OK

Raw /proc/interrupts (cat /proc/interrupts) - OK

Swap configuration (free -t) - OK

Swap configuration (swapon -s) - OK

Kernel modules (lsmod) - OK

Distro version (cat /etc/redhat-release) - OK

Distro version (cat /etc/oracle-release) - Exit code 1

Distro version (lsb_release -a) - OK

Distro version (cat /etc/SuSE-release) - Exit code 1

Distro version (cat /etc/issue) - OK

Distro version (cat /etc/os-release) - OK

Distro version (cat /etc/system-release) - OK

Installed software (rpm -qa) - OK

Ksplice updates (uptrack-show) - Exit code 127

Hot fix list (rpm -V couchbase-server) - Exit code 1

Installed software (COLUMNS=300 dpkg -l) - Exit code 127

Hot fix list (COLUMNS=300 dpkg -V couchbase-server) - Exit code 127

Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127

Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK

sysctl settings (sysctl -a) - OK

Relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr cbft eventing-producer eventing-consumer | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - OK

LVM info (lvdisplay) - OK

LVM info (vgdisplay) - OK

LVM info (pvdisplay) - Exit code 5

Block device queue settings (find /sys/block/*/queue /sys/block/*/device/queue_* -type f | xargs grep -vH xxxx | sort) - OK

Network configuration (ifconfig -a) - OK

Taking sample 2 after 10.000000 seconds - OK

Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - OK

Raw /proc/net/dev (cat /proc/net/dev) - OK

Network link statistics (ip -s link) - OK

Network status (netstat -anp || netstat -an) - OK

Network routing table (netstat -rn) - OK

Network socket statistics (ss -an) - OK

Extended socket statistics (ss -an --info --processes --memory --options) - OK

Arp cache (arp -na) - OK

Iptables dump (iptables-save) - OK

Raw /etc/hosts (cat /etc/hosts) - OK

Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK

Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK

Filesystem (df -ha) - OK

Filesystem inodes (df -i) - OK

System activity reporter (sar 1 10) - Exit code 127

System paging activity (vmstat 1 10) - OK

System uptime (uptime) - OK

Last logins of users and ttys (last -x || last) - OK

couchbase user definition (getent passwd couchbase) - OK

couchbase user limits (su couchbase -s /bin/sh -c "ulimit -a") - skipped (needs root privs)

Interrupt status (intrstat 1 10) - Exit code 127

Processor status (mpstat 1 10) - Exit code 127

System log (cat /var/adm/messages) - Exit code 1

Raw /proc/uptime (cat /proc/uptime) - OK

Systemd journal (journalctl | gzip -c > systemd_journal.gz) - OK

All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2

Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr cbft eventing-producer eventing-consumer | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK

Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr cbft eventing-producer eventing-consumer | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n | egrep -v ^CB_MASTER_PASSWORD=\|^CBAUTH_REVRPC_URL=); echo' --) - OK

Processes' stack (for program in moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr cbft eventing-producer eventing-consumer; do for thread in $(pgrep --lightweight $program); do echo $program/$thread:; cat /proc/$thread/stack; echo; done; done) - OK

NUMA data (numactl --hardware) - Exit code 127

NUMA data (numactl --show) - Exit code 127

NUMA data (cat /sys/devices/system/node/node*/numastat) - OK

Kernel log buffer (dmesg -T || dmesg -H || dmesg) - OK

Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - OK

Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - OK

Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1

Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1

Network statistics (netstat -s) - OK

Full raw netstat (cat /proc/net/netstat) - OK

CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123

Raw PID 1 scheduler /proc/1/sched (cat /proc/1/sched | head -n 1) - OK

Raw PID 1 control groups /proc/1/cgroup (cat /proc/1/cgroup) - OK

Found dump-guts: /opt/couchbase/bin/dump-guts

Adding persistent buckets '['travel-sample']' to server guts

product diag header () - OK

Traceback (most recent call last):

File "/opt/couchbase/lib/python/cbcollect_info", line 2275, in <module>

Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs...

main()

File "/opt/couchbase/lib/python/cbcollect_info", line 2170, in main

runner.run_tasks(make_product_task(guts, guts_initargs_path,

File "/opt/couchbase/lib/python/cbcollect_info", line 1196, in make_product_task

from distutils.spawn import find_executable

ModuleNotFoundError: No module named 'distutils'