Started by remote host 172.23.109.52 [EnvInject] - Loading node environment variables. Building remotely on slv-sc2304-32g-12c (P0 jython_slave bkrs3) in workspace /data/workspace/centos-p0-epeng-vset00-00-basic_ops [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Running Prebuild steps [centos-p0-epeng-vset00-00-basic_ops] $ /bin/sh -xe /tmp/jenkins5161589383319706873.sh ++ echo epeng-basic_ops-Feb-21-08:14:35-7.1.0-2335 ++ awk '{split($0,r,"-");print r[1],r[2]}' + desc='epeng basic_ops' + echo Desc: 7.1.0-2335 - epeng basic_ops - centos Desc: 7.1.0-2335 - epeng basic_ops - centos + echo newState=available + newState=available Success build forhudson.tasks.Shell@5ee7b57d [description-setter] Description set: 7.1.0-2335 - epeng basic_ops - centos Success build forhudson.plugins.descriptionsetter.DescriptionSetterBuilder@58d0b419 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Success build fororg.jenkinsci.plugins.envinject.EnvInjectBuilder@72813b5f Cloning the remote Git repository Using shallow clone Cloning repository https://github.com/couchbase/testrunner > /usr/bin/git init /data/workspace/centos-p0-epeng-vset00-00-basic_ops # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision f4fa1e90cec03b7a4afe547c15e1a760c747e40a (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f f4fa1e90cec03b7a4afe547c15e1a760c747e40a > /usr/bin/git rev-list f4fa1e90cec03b7a4afe547c15e1a760c747e40a # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #445457 jenkins-test_suite_executor-445457 # timeout=10 No emails were triggered. [centos-p0-epeng-vset00-00-basic_ops] $ /bin/sh -xe /tmp/jenkins8042457839596215090.sh + echo Desc: epeng-basic_ops-Feb-21-08:14:35-7.1.0-2335 Desc: epeng-basic_ops-Feb-21-08:14:35-7.1.0-2335 [description-setter] Could not determine description. [centos-p0-epeng-vset00-00-basic_ops] $ /bin/sh -xe /tmp/jenkins6042480163773331865.sh [centos-p0-epeng-vset00-00-basic_ops] $ /bin/sh -xe /tmp/jenkins111110070253269294.sh + py_executable=python2 + echo 7.1.0-2335 + grep -E '7\.0|7\.1|0\.0' 7.1.0-2335 + py_executable=python3 + [[ 7.1.0-2335 > 6.5 ]] + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + rerun_job=true + touch rerun_props_file + '[' false == false ']' + '[' true == true ']' + python3 scripts/rerun_jobs.py 7.1.0-2335 --executor_jenkins_job --manual_run [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'rerun_props_file' [EnvInject] - Variables injected successfully. [centos-p0-epeng-vset00-00-basic_ops] $ /bin/bash /tmp/jenkins5073549914031125634.sh 7.1.0-2335 Already on 'master' From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. Set ALLOW_HTP to False so test could run. HEAD is now at f4fa1e9 Adding LDAP users the major release is 7 "172.23.123.51","172.23.123.49","172.23.123.50" Searching for httplib2 Best match: httplib2 0.18.1 Adding httplib2 0.18.1 to easy-install.pth file Using /usr/local/lib/python3.7/site-packages Processing dependencies for httplib2 Finished processing dependencies for httplib2 centos Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.sonic.net * extras: linux.mirrors.es.net * updates: mirrors.cmich.edu https://download.fedoraproject.org/pub/epel/7/x86_64/repodata/a1836e3ec89d24b178ebb8780d0b07e576155ac70d045eaaa6f59ef22fa67282-primary.sqlite.bz2: [Errno 14] HTTPS Error 302 - Found Trying other mirror. Package 2:docker-1.13.1-208.git7d71120.el7_9.x86_64 already installed and latest version Nothing to do Using default tag: latest Trying to pull repository docker.io/jamesdbloom/mockserver ... latest: Pulling from docker.io/jamesdbloom/mockserver Digest: sha256:bc41010b9e920f5e1b75a226e8eda05eb4eb9843dad042e9e3950d1b7de823f4 Status: Image is up to date for docker.io/jamesdbloom/mockserver:latest [global] username:root password:couchbase [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 [_1] ip:dynamic port:8091 [_2] ip:dynamic port:8091 [_3] ip:dynamic port:8091 python3 scripts/populateIni.py -s "172.23.123.51","172.23.123.49","172.23.123.50" -d None -a None -i /tmp/testexec_reformat.129923.ini -p centos -o /tmp/testexec.129923.ini -k {} INFO:root:SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.123.49 as root INFO:root:SSH Connected to 172.23.123.51 as root INFO:root:SSH Connected to 172.23.123.50 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.123.50: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.123.49: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.123.51: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root in main the ini file is /tmp/testexec_reformat.129923.ini the given server info is "172.23.123.51","172.23.123.49","172.23.123.50" Collecting memory info from 172.23.123.50 Collecting memory info from 172.23.123.49 Collecting memory info from 172.23.123.51 the servers memory info is [('172.23.123.49', 4103196), ('172.23.123.51', 4103196), ('172.23.123.50', 4103200)] [global] username:root password:couchbase [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 [_1] ip:172.23.123.49 port:8091 [_2] ip:172.23.123.51 port:8091 [_3] ip:172.23.123.50 port:8091 extra install is Local time: Mon 2022-02-21 08:14:51 PST Universal time: Mon 2022-02-21 16:14:51 UTC RTC time: Mon 2022-02-21 16:14:51 Time zone: America/Los_Angeles (PST, -0800) NTP enabled: no NTP synchronized: yes RTC in local TZ: no DST active: no Last DST change: DST ended at Sun 2021-11-07 01:59:59 PDT Sun 2021-11-07 01:00:00 PST Next DST change: DST begins (the clock jumps one hour forward) at Sun 2022-03-13 01:59:59 PST Sun 2022-03-13 03:00:00 PDT python3 scripts/ssh.py -i /tmp/testexec_root.129923.ini iptables -F INFO:root:SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.123.49 as root INFO:root:SSH Connected to 172.23.123.50 as root INFO:root:SSH Connected to 172.23.123.51 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.123.50: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.123.49: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.123.51: iptables -F INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root 172.23.123.51 172.23.123.50 172.23.123.49 Initial version: python3 scripts/new_install.py -i /tmp/testexec.129923.ini -p timeout=1800,skip_local_download=False,get-cbcollect-info=True,version=7.1.0-2335,product=cb,debug_logs=True,ntp=True,url= 2022-02-21 08:14:53,508 - root - WARNING - URL: is not valid, will use version to locate build 2022-02-21 08:14:53,509 - root - INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:14:53,607 - root - INFO - SSH Connected to 172.23.123.49 as root 2022-02-21 08:14:53,860 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:14:54,147 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:14:54,149 - root - INFO - SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 2022-02-21 08:14:54,247 - root - INFO - SSH Connected to 172.23.123.51 as root 2022-02-21 08:14:54,500 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:14:54,803 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:14:54,804 - root - INFO - SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 2022-02-21 08:14:54,901 - root - INFO - SSH Connected to 172.23.123.50 as root 2022-02-21 08:14:55,157 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:14:55,449 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:14:55,451 - root - INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:14:55,550 - root - INFO - SSH Connected to 172.23.123.49 as root 2022-02-21 08:14:55,803 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:14:56,097 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:14:56,099 - root - INFO - SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 2022-02-21 08:14:56,200 - root - INFO - SSH Connected to 172.23.123.51 as root 2022-02-21 08:14:56,454 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:14:56,753 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:14:56,755 - root - INFO - SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 2022-02-21 08:14:56,850 - root - INFO - SSH Connected to 172.23.123.50 as root 2022-02-21 08:14:57,099 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:14:57,383 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:14:57,384 - root - INFO - Check if ntp is installed 2022-02-21 08:14:57,384 - root - INFO - This OS version centos 7 2022-02-21 08:14:57,384 - root - INFO - running command.raw on 172.23.123.49: systemctl status ntpd 2022-02-21 08:14:57,385 - root - INFO - Check if ntp is installed 2022-02-21 08:14:57,385 - root - INFO - This OS version centos 7 2022-02-21 08:14:57,385 - root - INFO - running command.raw on 172.23.123.51: systemctl status ntpd 2022-02-21 08:14:57,385 - root - INFO - Check if ntp is installed 2022-02-21 08:14:57,386 - root - INFO - This OS version centos 7 2022-02-21 08:14:57,386 - root - INFO - running command.raw on 172.23.123.50: systemctl status ntpd 2022-02-21 08:14:57,409 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,410 - root - INFO - running command.raw on 172.23.123.49: ntpstat 2022-02-21 08:14:57,411 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,411 - root - INFO - running command.raw on 172.23.123.51: ntpstat 2022-02-21 08:14:57,444 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,444 - root - INFO - running command.raw on 172.23.123.50: ntpstat 2022-02-21 08:14:57,502 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,502 - root - INFO - running command.raw on 172.23.123.49: timedatectl status 2022-02-21 08:14:57,504 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,504 - root - INFO - running command.raw on 172.23.123.51: timedatectl status 2022-02-21 08:14:57,530 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,530 - root - INFO - running command.raw on 172.23.123.50: timedatectl status 2022-02-21 08:14:57,584 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,584 - root - INFO - running command.raw on 172.23.123.49: date 2022-02-21 08:14:57,587 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,587 - root - INFO - running command.raw on 172.23.123.51: date 2022-02-21 08:14:57,609 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,609 - root - INFO - running command.raw on 172.23.123.50: date 2022-02-21 08:14:57,637 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,641 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,661 - root - INFO - command executed successfully with root 2022-02-21 08:14:57,689 - root - INFO - ['Mon Feb 21 08:14:57 PST 2022'] IP: 172.23.123.49 2022-02-21 08:14:57,694 - root - INFO - ['Mon Feb 21 08:14:57 PST 2022'] IP: 172.23.123.51 2022-02-21 08:14:57,712 - root - INFO - ['Mon Feb 21 08:14:57 PST 2022'] IP: 172.23.123.50 2022-02-21 08:14:57,712 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2335/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:14:57,716 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2335/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm is live 2022-02-21 08:14:57,716 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2335/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:14:57,717 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2335/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm is live 2022-02-21 08:14:57,718 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2335/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:14:57,719 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2335/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm is live 2022-02-21 08:14:57,719 - root - INFO - running command.raw on 172.23.123.49: umount -a -t nfs,nfs4 -f -l;yes | yum remove 'couchbase*' > /dev/null; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2022-02-21 08:14:57,720 - root - INFO - running command.raw on 172.23.123.51: umount -a -t nfs,nfs4 -f -l;yes | yum remove 'couchbase*' > /dev/null; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2022-02-21 08:14:57,720 - root - INFO - running command.raw on 172.23.123.50: umount -a -t nfs,nfs4 -f -l;yes | yum remove 'couchbase*' > /dev/null; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2022-02-21 08:15:03,878 - root - INFO - command executed successfully with root 2022-02-21 08:15:04,064 - root - INFO - command executed successfully with root 2022-02-21 08:15:04,069 - root - INFO - command executed successfully with root 2022-02-21 08:15:04,888 - root - INFO - Done with uninstall on 172.23.123.49. 2022-02-21 08:15:05,018 - root - INFO - Done with uninstall on 172.23.123.51. 2022-02-21 08:15:05,058 - root - INFO - Done with uninstall on 172.23.123.50. 2022-02-21 08:15:07,731 - root - INFO - Downloading build binary to /tmp/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm.. 2022-02-21 08:15:14,082 - root - INFO - running command.raw on 172.23.123.49: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:15:14,095 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,097 - root - INFO - running command.raw on 172.23.123.51: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:15:14,111 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,112 - root - INFO - running command.raw on 172.23.123.50: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:15:14,127 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,127 - root - INFO - running command.raw on 172.23.123.49: ls -lh /tmp/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:15:14,148 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,149 - root - INFO - running command.raw on 172.23.123.49: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:15:14,201 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,201 - root - INFO - running command.raw on 172.23.123.51: ls -lh /tmp/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:15:14,218 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,219 - root - INFO - running command.raw on 172.23.123.51: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:15:14,273 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,274 - root - INFO - running command.raw on 172.23.123.50: ls -lh /tmp/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:15:14,288 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,289 - root - INFO - running command.raw on 172.23.123.50: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm 2022-02-21 08:15:14,340 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,341 - root - INFO - running command.raw on 172.23.123.49: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2022-02-21 08:15:14,341 - root - INFO - running command.raw on 172.23.123.51: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2022-02-21 08:15:14,342 - root - INFO - running command.raw on 172.23.123.50: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2022-02-21 08:15:14,356 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,356 - root - INFO - running command.raw on 172.23.123.49: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2022-02-21 08:15:14,357 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,357 - root - INFO - running command.raw on 172.23.123.51: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2022-02-21 08:15:14,396 - root - INFO - command executed successfully with root 2022-02-21 08:15:14,396 - root - INFO - running command.raw on 172.23.123.50: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.1.0-2335-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2022-02-21 08:16:11,256 - root - INFO - command executed successfully with root 2022-02-21 08:16:11,256 - root - INFO - running command.raw on 172.23.123.49: systemctl -q is-active couchbase-server && echo 1 || echo 0 2022-02-21 08:16:11,277 - root - INFO - command executed successfully with root 2022-02-21 08:16:11,277 - root - INFO - Done with install on 172.23.123.49. 2022-02-21 08:16:11,277 - root - INFO - Waiting for couchbase to be reachable 2022-02-21 08:16:11,279 - root - ERROR - socket error while connecting to http://172.23.123.49:8091/nodes/self error [Errno 111] Connection refused 2022-02-21 08:16:11,476 - root - INFO - command executed successfully with root 2022-02-21 08:16:11,477 - root - INFO - running command.raw on 172.23.123.50: systemctl -q is-active couchbase-server && echo 1 || echo 0 2022-02-21 08:16:11,540 - root - INFO - command executed successfully with root 2022-02-21 08:16:11,540 - root - INFO - Done with install on 172.23.123.50. 2022-02-21 08:16:11,540 - root - INFO - Waiting for couchbase to be reachable 2022-02-21 08:16:11,541 - root - ERROR - socket error while connecting to http://172.23.123.50:8091/nodes/self error [Errno 111] Connection refused 2022-02-21 08:16:11,667 - root - INFO - command executed successfully with root 2022-02-21 08:16:11,667 - root - INFO - running command.raw on 172.23.123.51: systemctl -q is-active couchbase-server && echo 1 || echo 0 2022-02-21 08:16:11,729 - root - INFO - command executed successfully with root 2022-02-21 08:16:11,729 - root - INFO - Done with install on 172.23.123.51. 2022-02-21 08:16:11,729 - root - INFO - Waiting for couchbase to be reachable 2022-02-21 08:16:11,730 - root - ERROR - socket error while connecting to http://172.23.123.51:8091/nodes/self error [Errno 111] Connection refused 2022-02-21 08:16:14,282 - root - ERROR - socket error while connecting to http://172.23.123.49:8091/nodes/self error [Errno 111] Connection refused 2022-02-21 08:16:14,545 - root - ERROR - socket error while connecting to http://172.23.123.50:8091/nodes/self error [Errno 111] Connection refused 2022-02-21 08:16:14,734 - root - ERROR - socket error while connecting to http://172.23.123.51:8091/nodes/self error [Errno 111] Connection refused 2022-02-21 08:16:20,295 - root - INFO - running command.raw on 172.23.123.49: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.123.49 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-21 08:16:20,559 - root - INFO - running command.raw on 172.23.123.50: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.123.50 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-21 08:16:20,719 - root - INFO - command executed successfully with root 2022-02-21 08:16:20,746 - root - INFO - running command.raw on 172.23.123.51: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.123.51 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-21 08:16:20,986 - root - INFO - command executed successfully with root 2022-02-21 08:16:21,170 - root - INFO - command executed successfully with root 2022-02-21 08:16:21,726 - root - INFO - Setting KV memory quota as 2684 MB on 172.23.123.49 2022-02-21 08:16:21,726 - root - INFO - pools/default params : memoryQuota=2684 2022-02-21 08:16:21,734 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-21 08:16:21,734 - root - INFO - /node/controller/setupServices params on 172.23.123.49: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-21 08:16:21,762 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-21 08:16:21,762 - root - INFO - settings/web params on 172.23.123.49:8091:port=8091&username=Administrator&password=password 2022-02-21 08:16:21,825 - root - INFO - --> status:True 2022-02-21 08:16:21,825 - root - INFO - Done with init on 172.23.123.49. 2022-02-21 08:16:21,825 - root - INFO - running command.raw on 172.23.123.49: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-02-21 08:16:21,843 - root - INFO - command executed successfully with root 2022-02-21 08:16:21,843 - root - INFO - Done with cleanup on 172.23.123.49. 2022-02-21 08:16:21,993 - root - INFO - Setting KV memory quota as 2684 MB on 172.23.123.50 2022-02-21 08:16:21,993 - root - INFO - pools/default params : memoryQuota=2684 2022-02-21 08:16:21,999 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-21 08:16:21,999 - root - INFO - /node/controller/setupServices params on 172.23.123.50: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-21 08:16:22,027 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-21 08:16:22,027 - root - INFO - settings/web params on 172.23.123.50:8091:port=8091&username=Administrator&password=password 2022-02-21 08:16:22,087 - root - INFO - --> status:True 2022-02-21 08:16:22,088 - root - INFO - Done with init on 172.23.123.50. 2022-02-21 08:16:22,088 - root - INFO - running command.raw on 172.23.123.50: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-02-21 08:16:22,107 - root - INFO - command executed successfully with root 2022-02-21 08:16:22,107 - root - INFO - Done with cleanup on 172.23.123.50. 2022-02-21 08:16:22,177 - root - INFO - Setting KV memory quota as 2684 MB on 172.23.123.51 2022-02-21 08:16:22,177 - root - INFO - pools/default params : memoryQuota=2684 2022-02-21 08:16:22,183 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-21 08:16:22,183 - root - INFO - /node/controller/setupServices params on 172.23.123.51: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-21 08:16:22,212 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-21 08:16:22,212 - root - INFO - settings/web params on 172.23.123.51:8091:port=8091&username=Administrator&password=password 2022-02-21 08:16:22,270 - root - INFO - --> status:True 2022-02-21 08:16:22,270 - root - INFO - Done with init on 172.23.123.51. 2022-02-21 08:16:22,270 - root - INFO - running command.raw on 172.23.123.51: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-02-21 08:16:22,299 - root - INFO - command executed successfully with root 2022-02-21 08:16:22,300 - root - INFO - Done with cleanup on 172.23.123.51. 2022-02-21 08:16:24,414 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-21 08:16:24,425 - root - INFO - cluster:C1 node:172.23.123.49:8091 version:7.1.0-2335-enterprise aFamily:inet services:['kv'] 2022-02-21 08:16:24,425 - root - INFO - cluster:C2 node:172.23.123.51:8091 version:7.1.0-2335-enterprise aFamily:inet services:['kv'] 2022-02-21 08:16:24,426 - root - INFO - cluster:C3 node:172.23.123.50:8091 version:7.1.0-2335-enterprise aFamily:inet services:['kv'] 2022-02-21 08:16:24,426 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-21 08:16:24,426 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-21 08:16:24,426 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-21 08:16:24,426 - root - INFO - INSTALL COMPLETED ON: 172.23.123.49 2022-02-21 08:16:24,426 - root - INFO - INSTALL COMPLETED ON: 172.23.123.51 2022-02-21 08:16:24,426 - root - INFO - INSTALL COMPLETED ON: 172.23.123.50 2022-02-21 08:16:24,426 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-21 08:16:24,426 - root - INFO - TOTAL INSTALL TIME = 91 seconds success INFO:root:SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.123.50 as root INFO:root:SSH Connected to 172.23.123.51 as root INFO:root:SSH Connected to 172.23.123.49 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.123.50: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.123.49: iptables -F INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.123.51: iptables -F INFO:root:command executed successfully with root INFO:root:command executed successfully with root 172.23.123.50 172.23.123.49 172.23.123.51 Need to set ALLOW_HTP back to True to do git pull branch Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client' Cloning into 'java_sdk_client'... Submodule path 'java_sdk_client': checked out 'b23994ef285cfeb976e9557a630dcf003f95da5e' Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.sonic.net * extras: linux.mirrors.es.net * updates: atl.mirrors.clouvider.net Package maven-3.0.5-17.el7.noarch already installed and latest version Nothing to do find: ‘/data/workspace/centos-p0-collections-vset00-00-compaction_7.0_P2’: No such file or directory find: ‘/data/workspace/centos-p0-xdcr-vset00-00-unidirectional-1-magma-1DGM/job_logs’: No such file or directory find: ‘/data/workspace/centos-p0-durability-vset00-00-bucket_compaction_6.5_P0_majority’: No such file or directory find: ‘/data/workspace/centos-p0-transaction-vset00-00-autofailover_durability_majority_6.5_P0_Set1’: No such file or directory find: ‘/data/workspace/centos-p0-durability-vset00-00-steady_state_subdoc_6.5_P0_persist_majority’: No such file or directory find: ‘/data/workspace/centos-p0-transaction-vset00-00-doc_isolation_durability_majority_and_persist_on_master_6.6_P1’: No such file or directory find: ‘/data/workspace/centos-p0-transaction-vset00-00-steady_state_durability_majority_and_persist_on_master_6.5_P0/scripts’: No such file or directory find: ‘/data/workspace/centos-p0-transaction-vset00-00-steady_state_durability_majority_and_persist_on_master_6.5_P0/job_logs’: No such file or directory find: ‘/data/workspace/centos-p0-collections-vset00-00-rebalance_collections_maxt_ttl_durability_MAJORITY_AND_PERSIST_ACTIVE_7.0_P1’: No such file or directory find: ‘/data/workspace/centos-p0-durability-vset00-00-bucket_compaction_6.5_P1_persist_majority’: No such file or directory find: ‘/data/workspace/centos-p0-transaction-vset00-00-crash_test_durability_persist_to_majority_6.5_P0_Set1’: No such file or directory find: ‘/data/workspace/centos-p0-durability-vset00-00-steady_state_subdoc_xattr_6.5_P0_persist_majority’: No such file or directory find: ‘/data/workspace/centos-p0-transaction-vset00-00-crash_test_durability_majority_and_persist_on_master_6.5_P1’: No such file or directory find: ‘/data/workspace/centos-p0-durability-vset00-00-bucket_compaction_6.5_P0_persist_majority’: No such file or directory find: ‘/data/workspace/centos-p0-transaction-vset00-00-steady_state_durability_persist_to_majority_retry_6.5_P0’: No such file or directory find: ‘/data/workspace/centos-p0-transaction-vset00-00-newfailover_durability_majority_6.5_P0_Set1’: No such file or directory find: ‘/data/workspace/centos-p0-2i-vset00-00-index-partitioning-cbd-plasma_ca_G4_7.1_P0’: No such file or directory find: ‘/root/workspace/*/logs/*’: No such file or directory find: ‘/root/workspace/’: No such file or directory python3: no process found python3 testrunner.py -i /tmp/testexec.129923.ini -c py-epeng-basic-ops.conf -p skip_log_scan=False,bucket_storage=couchstore -d failed=http://qa.sc.couchbase.com/job/test_suite_executor/445402/ INFO:root:__main__ INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:Conf filename: py-epeng-basic-ops.conf INFO:__main__:Test prefix: epengine.basic_ops.basic_ops INFO:__main__:Downloading http://qa.sc.couchbase.com/job/test_suite_executor/445402//testReport/api/xml?pretty=true to logs/test_suite_executor_445402__testresult.xml INFO:__main__:Loading result data from logs/test_suite_executor_445402__testresult.xml INFO:__main__:-- logs/test_suite_executor_445402__testresult.xml -- INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 3 INFO:__main__:--> Running test: epengine.basic_ops.basic_ops.test_compression_active_and_off,items=10000,compression_mode=active,skip_log_scan=False,bucket_storage=couchstore INFO:__main__:Logs folder: /data/workspace/centos-p0-epeng-vset00-00-basic_ops/logs/testrunner-22-Feb-21_08-16-39/test_1 *** TestRunner *** {'bucket_storage': 'couchstore', 'cluster_name': 'testexec.129923', 'conf_file': 'py-epeng-basic-ops.conf', 'ini': '/tmp/testexec.129923.ini', 'num_nodes': 3, 'skip_log_scan': 'False', 'spec': 'py-epeng-basic-ops'} Logs will be stored at /data/workspace/centos-p0-epeng-vset00-00-basic_ops/logs/testrunner-22-Feb-21_08-16-39/test_1 ./testrunner -i /tmp/testexec.129923.ini -p skip_log_scan=False,bucket_storage=couchstore -t epengine.basic_ops.basic_ops.test_compression_active_and_off,items=10000,compression_mode=active,skip_log_scan=False,bucket_storage=couchstore Test Input params: {'items': '10000', 'compression_mode': 'active', 'skip_log_scan': 'False', 'bucket_storage': 'couchstore', 'ini': '/tmp/testexec.129923.ini', 'cluster_name': 'testexec.129923', 'spec': 'py-epeng-basic-ops', 'conf_file': 'py-epeng-basic-ops.conf', 'num_nodes': 3, 'case_number': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/data/workspace/centos-p0-epeng-vset00-00-basic_ops/logs/testrunner-22-Feb-21_08-16-39/test_1'} Run before suite setup for epengine.basic_ops.basic_ops.test_compression_active_and_off suite_setUp (unittest.loader._FailedTest) ... ERROR ====================================================================== ERROR: suite_setUp (unittest.loader._FailedTest) ---------------------------------------------------------------------- AttributeError: type object 'basic_ops' has no attribute 'suite_setUp' ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (errors=1) test_compression_active_and_off (epengine.basic_ops.basic_ops) ... -->before_suite_name:epengine.basic_ops.basic_ops.suite_setUp,suite: ]> -->result: 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2335-enterprise 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_compression_active_and_off============== 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.123.49:8091 => {'services': ['kv'], 'cpu_utilization': 1.682571572074335, 'mem_free': 3548008448, 'mem_total': 4201672704, 'swap_mem_used': 2883584, 'swap_mem_total': 3758092288} 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:16:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:16:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:16:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:16:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 2022-02-21 08:16:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.51 as root 2022-02-21 08:16:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:16:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:16:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 2022-02-21 08:16:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.50 as root 2022-02-21 08:16:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:16:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_compression_active_and_off ============== 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.49, nothing to delete 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.51, nothing to delete 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.50, nothing to delete 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-02-21 08:16:44 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.123.49:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.49:8091 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.49:8091 is running 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.51:8091 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.51:8091 is running 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.50:8091 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.50:8091 is running 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_compression_active_and_off ============== 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-02-21 08:16:44 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2022-02-21 08:16:45 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.123.49 port:8091 ssh_username:root, nodes/self 2022-02-21 08:16:45 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '31', 'memoryTotal': 4201672704, 'memoryFree': 3555176448, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.49:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2684, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.49', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2022-02-21 08:16:45 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-02-21 08:16:45 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-21 08:16:45 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.123.49:8091:port=8091&username=Administrator&password=password 2022-02-21 08:16:45 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-21 08:16:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:16:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.49: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.49:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.49:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.123.51 port:8091 ssh_username:root, nodes/self 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '31', 'memoryTotal': 4201672704, 'memoryFree': 3604938752, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.51:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2684, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.51', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.123.51:8091:port=8091&username=Administrator&password=password 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.51 as root 2022-02-21 08:16:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.51: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.51:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.51:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.123.50 port:8091 ssh_username:root, nodes/self 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '31', 'memoryTotal': 4201676800, 'memoryFree': 3626995712, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.50:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2684, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.50', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.123.50:8091:port=8091&username=Administrator&password=password 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.50 as root 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:16:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.50: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-21 08:16:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-21 08:16:48 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-21 08:16:48 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.50:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-21 08:16:48 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.50:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-21 08:16:48 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-21 08:16:48 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.123.49 **** 2022-02-21 08:16:48 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.123.49:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-02-21 08:16:48 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-02-21 08:16:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-02-21 08:16:53 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-02-21 08:16:53 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2022-02-21 08:16:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:16:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:16:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:16:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:16:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.49: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-21 08:16:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-21 08:16:53 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-21 08:16:53 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.123.49 2022-02-21 08:16:54 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.123.49:8091/pools/default/buckets with param: name=default&ramQuotaMB=2147&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=active&storageBackend=couchstore 2022-02-21 08:16:54 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket default 2022-02-21 08:16:54 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.123.49 to accept set ops 2022-02-21 08:16:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.123.49:11210 default 2022-02-21 08:16:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.123.49:11210 default 2022-02-21 08:16:55 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 2147 2022-02-21 08:16:55 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_compression_active_and_off ============== 2022-02-21 08:16:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:16:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:16:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:16:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:16:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 2022-02-21 08:16:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.51 as root 2022-02-21 08:16:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:16:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:16:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 2022-02-21 08:16:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.50 as root 2022-02-21 08:16:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:16:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:17:00 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-02-21 08:17:00 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.123.49:8091 => {'services': ['kv'], 'cpu_utilization': 3.316582914572864, 'mem_free': 3554754560, 'mem_total': 4201672704, 'swap_mem_used': 2883584, 'swap_mem_total': 3758092288} 2022-02-21 08:17:00 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-02-21 08:17:00 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.0461208820343 2022-02-21 08:17:00 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.123.49:11210 default 2022-02-21 08:17:01 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.123.49:11210 default 2022-02-21 08:17:01 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.123.49:11210 default 2022-02-21 08:17:01 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.123.49:11210 default 2022-02-21 08:17:01 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.123.49:11210 default 2022-02-21 08:17:02 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2335-enterprise'] 2022-02-21 08:17:02 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.123.49 port:8091 ssh_username:root] 2022-02-21 08:17:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:17:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:17:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:17:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:17:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.49: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-21 08:17:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-21 08:17:03 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.123.49:8091',default bucket 2022-02-21 08:17:03 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2335-enterprise'] 2022-02-21 08:17:03 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.123.49 port:8091 ssh_username:root] 2022-02-21 08:17:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:17:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:17:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:17:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:17:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.49: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-21 08:17:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-21 08:17:04 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items 10000 == 10000 expected on '172.23.123.49:8091',default bucket 2022-02-21 08:17:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.49: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-21 08:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-21 08:17:05 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 10000 == 10000 expected on '172.23.123.49:8091',default bucket 2022-02-21 08:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.49: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-21 08:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-21 08:17:06 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_replica_curr_items 0 == 0 expected on '172.23.123.49:8091',default bucket 2022-02-21 08:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:17:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:17:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.49: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-21 08:17:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-21 08:17:07 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items_tot 10000 == 10000 expected on '172.23.123.49:8091',default bucket 2022-02-21 08:17:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:17:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:17:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:17:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:17:07 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-02-21 08:17:07 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.123.49:8091 => {'services': ['kv'], 'cpu_utilization': 13.96946564885496, 'mem_free': 3468763136, 'mem_total': 4201672704, 'swap_mem_used': 2883584, 'swap_mem_total': 3758092288} 2022-02-21 08:17:07 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-02-21 08:17:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 2022-02-21 08:17:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.49 as root 2022-02-21 08:17:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:17:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:17:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 2022-02-21 08:17:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.51 as root 2022-02-21 08:17:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:17:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:17:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 2022-02-21 08:17:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.50 as root 2022-02-21 08:17:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-21 08:17:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-21 08:17:12 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_compression_active_and_off ============== 2022-02-21 08:17:12 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 172.23.123.49 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.123.49 existing buckets : [] 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.123.49 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.51, nothing to delete 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.50, nothing to delete 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-02-21 08:17:13 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.123.49:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.49:8091 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.49:8091 is running 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.51:8091 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.51:8091 is running 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.50:8091 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.50:8091 is running 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_compression_active_and_off ============== 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-02-21 08:17:13 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force summary so far suite epengine.basic_ops.basic_ops , pass 0 , fail 1 failures so far... epengine.basic_ops.basic_ops.test_compression_active_and_off testrunner logs, diags and results are available under /data/workspace/centos-p0-epeng-vset00-00-basic_ops/logs/testrunner-22-Feb-21_08-16-39/test_1 ERROR ====================================================================== ERROR: test_compression_active_and_off (epengine.basic_ops.basic_ops) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/epengine/basic_ops.py", line 344, in test_compression_active_and_off self.verify_stat(items=self.num_items) File "pytests/epengine/basic_ops.py", line 321, in verify_stat self.assertNotEqual(int(stats['vb_active_itm_memory']), int(stats['vb_active_itm_memory_uncompressed'])) KeyError: 'vb_active_itm_memory' ---------------------------------------------------------------------- Ran 1 test in 34.091s FAILED (errors=1) test_MB_32114 (epengine.basic_ops.basic_ops) ... Logs will be stored at /data/workspace/centos-p0-epeng-vset00-00-basic_ops/logs/testrunner-22-Feb-21_08-16-39/test_2 ./testrunner -i /tmp/testexec.129923.ini -p skip_log_scan=False,bucket_storage=couchstore -t epengine.basic_ops.basic_ops.test_MB_32114,nodes_init=3,replicas=2,skip_cleanup=False,bucket_size=100,items=100000,active_resident_threshold=5,maxttl=300,skip_log_scan=False,bucket_storage=couchstore Test Input params: {'nodes_init': '3', 'replicas': '2', 'skip_cleanup': 'False', 'bucket_size': '100', 'items': '100000', 'active_resident_threshold': '5', 'maxttl': '300', 'skip_log_scan': 'False', 'bucket_storage': 'couchstore', 'ini': '/tmp/testexec.129923.ini', 'cluster_name': 'testexec.129923', 'spec': 'py-epeng-basic-ops', 'conf_file': 'py-epeng-basic-ops.conf', 'num_nodes': 3, 'case_number': 2, 'last_case_fail': 'True', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centos-p0-epeng-vset00-00-basic_ops/logs/testrunner-22-Feb-21_08-16-39/test_2'} [2022-02-21 08:17:13,314] - [remote_util:302] INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 [2022-02-21 08:17:13,411] - [remote_util:338] INFO - SSH Connected to 172.23.123.49 as root [2022-02-21 08:17:13,666] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:17:13,956] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:17:13,959] - [rest_client:44] INFO - -->is_ns_server_running? [2022-02-21 08:17:13,966] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2335-enterprise [2022-02-21 08:17:13,966] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #2 test_MB_32114============== [2022-02-21 08:17:13,967] - [basetestcase:331] INFO - initializing cluster [2022-02-21 08:17:14,317] - [task:154] INFO - server: ip:172.23.123.49 port:8091 ssh_username:root, nodes/self [2022-02-21 08:17:14,320] - [task:159] INFO - {'uptime': '60', 'memoryTotal': 4201672704, 'memoryFree': 3468763136, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.49:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.49', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2022-02-21 08:17:14,323] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-02-21 08:17:14,326] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-02-21 08:17:14,326] - [rest_client:1149] INFO - settings/web params on 172.23.123.49:8091:port=8091&username=Administrator&password=password [2022-02-21 08:17:14,374] - [rest_client:1151] INFO - --> status:True [2022-02-21 08:17:14,374] - [remote_util:302] INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 [2022-02-21 08:17:14,471] - [remote_util:338] INFO - SSH Connected to 172.23.123.49 as root [2022-02-21 08:17:14,729] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:17:15,019] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:17:15,020] - [remote_util:3317] INFO - running command.raw on 172.23.123.49: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-02-21 08:17:15,081] - [remote_util:3366] INFO - command executed successfully with root [2022-02-21 08:17:15,081] - [remote_util:5200] INFO - ['ok'] [2022-02-21 08:17:15,083] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.49:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:17:15,085] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.49:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:17:15,088] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-02-21 08:17:15,100] - [task:154] INFO - server: ip:172.23.123.51 port:8091 ssh_username:root, nodes/self [2022-02-21 08:17:15,103] - [task:159] INFO - {'uptime': '61', 'memoryTotal': 4201672704, 'memoryFree': 3609878528, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.51:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.51', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2022-02-21 08:17:15,105] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-02-21 08:17:15,109] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-02-21 08:17:15,109] - [rest_client:1149] INFO - settings/web params on 172.23.123.51:8091:port=8091&username=Administrator&password=password [2022-02-21 08:17:15,157] - [rest_client:1151] INFO - --> status:True [2022-02-21 08:17:15,158] - [remote_util:302] INFO - SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 [2022-02-21 08:17:15,263] - [remote_util:338] INFO - SSH Connected to 172.23.123.51 as root [2022-02-21 08:17:15,523] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:17:15,821] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:17:15,821] - [remote_util:3317] INFO - running command.raw on 172.23.123.51: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-02-21 08:17:15,885] - [remote_util:3366] INFO - command executed successfully with root [2022-02-21 08:17:15,885] - [remote_util:5200] INFO - ['ok'] [2022-02-21 08:17:15,887] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.51:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:17:15,889] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.51:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:17:15,892] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-02-21 08:17:15,906] - [task:154] INFO - server: ip:172.23.123.50 port:8091 ssh_username:root, nodes/self [2022-02-21 08:17:15,909] - [task:159] INFO - {'uptime': '61', 'memoryTotal': 4201676800, 'memoryFree': 3635298304, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.50:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.50', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2022-02-21 08:17:15,912] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-02-21 08:17:15,915] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-02-21 08:17:15,915] - [rest_client:1149] INFO - settings/web params on 172.23.123.50:8091:port=8091&username=Administrator&password=password [2022-02-21 08:17:15,964] - [rest_client:1151] INFO - --> status:True [2022-02-21 08:17:15,965] - [remote_util:302] INFO - SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 [2022-02-21 08:17:16,065] - [remote_util:338] INFO - SSH Connected to 172.23.123.50 as root [2022-02-21 08:17:16,314] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:17:16,603] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:17:16,603] - [remote_util:3317] INFO - running command.raw on 172.23.123.50: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-02-21 08:17:16,665] - [remote_util:3366] INFO - command executed successfully with root [2022-02-21 08:17:16,665] - [remote_util:5200] INFO - ['ok'] [2022-02-21 08:17:16,667] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.50:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:17:16,668] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.50:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:17:16,671] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-02-21 08:17:16,678] - [basetestcase:2377] INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.49 **** [2022-02-21 08:17:16,747] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-02-21 08:17:21,752] - [basetestcase:2382] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-02-21 08:17:21,762] - [basetestcase:369] INFO - done initializing cluster [2022-02-21 08:17:22,685] - [task:771] INFO - adding node 172.23.123.51:8091 to cluster [2022-02-21 08:17:22,690] - [rest_client:1613] INFO - adding remote node @172.23.123.51:18091 to this cluster @172.23.123.49:8091 [2022-02-21 08:17:32,704] - [rest_client:1951] INFO - rebalance progress took 10.01 seconds [2022-02-21 08:17:32,705] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-02-21 08:17:46,561] - [task:771] INFO - adding node 172.23.123.50:8091 to cluster [2022-02-21 08:17:46,567] - [rest_client:1613] INFO - adding remote node @172.23.123.50:18091 to this cluster @172.23.123.49:8091 [2022-02-21 08:17:56,582] - [rest_client:1951] INFO - rebalance progress took 10.01 seconds [2022-02-21 08:17:56,582] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-02-21 08:18:09,905] - [rest_client:2621] INFO - Node 172.23.123.50 not part of cluster inactiveAdded [2022-02-21 08:18:09,906] - [rest_client:2621] INFO - Node 172.23.123.51 not part of cluster inactiveAdded [2022-02-21 08:18:09,910] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.49,ns_1@172.23.123.50,ns_1@172.23.123.51', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-02-21 08:18:09,959] - [rest_client:1850] INFO - rebalance operation started [2022-02-21 08:18:19,972] - [task:841] INFO - Rebalance - status: none, progress: 100.00% [2022-02-21 08:18:19,977] - [task:900] INFO - rebalancing was completed with progress: 100% in 10.017283201217651 sec [2022-02-21 08:18:19,978] - [remote_util:302] INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 [2022-02-21 08:18:20,075] - [remote_util:338] INFO - SSH Connected to 172.23.123.49 as root [2022-02-21 08:18:20,332] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:18:20,623] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:18:20,624] - [remote_util:3317] INFO - running command.raw on 172.23.123.49: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-02-21 08:18:20,688] - [remote_util:3366] INFO - command executed successfully with root [2022-02-21 08:18:20,688] - [remote_util:5200] INFO - ['ok'] [2022-02-21 08:18:20,688] - [basetestcase:2970] INFO - Enabled diag/eval for non-local hosts from 172.23.123.49 [2022-02-21 08:18:20,991] - [rest_client:2989] INFO - http://172.23.123.49:8091/pools/default/buckets with param: name=default&ramQuotaMB=100&replicaNumber=2&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&maxTTL=300&compressionMode=passive&storageBackend=couchstore [2022-02-21 08:18:21,017] - [rest_client:3014] INFO - 0.03 seconds to create bucket default [2022-02-21 08:18:21,017] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.123.49 to accept set ops [2022-02-21 08:18:21,651] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:18:21,716] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:18:21,792] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:18:21,921] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:18:21,984] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:18:22,047] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:18:23,901] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 100 [2022-02-21 08:18:23,903] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #2 test_MB_32114 ============== [2022-02-21 08:18:23,908] - [remote_util:302] INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 [2022-02-21 08:18:24,006] - [remote_util:338] INFO - SSH Connected to 172.23.123.49 as root [2022-02-21 08:18:24,265] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:18:24,556] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:18:24,558] - [remote_util:302] INFO - SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 [2022-02-21 08:18:24,665] - [remote_util:338] INFO - SSH Connected to 172.23.123.51 as root [2022-02-21 08:18:24,927] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:18:25,223] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:18:25,225] - [remote_util:302] INFO - SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 [2022-02-21 08:18:25,325] - [remote_util:338] INFO - SSH Connected to 172.23.123.50 as root [2022-02-21 08:18:25,589] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:18:25,883] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:18:28,333] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-02-21 08:18:28,333] - [basetestcase:585] INFO - 172.23.123.49:8091 => {'services': ['kv'], 'cpu_utilization': 1.634808853118712, 'mem_free': 3496083456, 'mem_total': 4201672704, 'swap_mem_used': 2883584, 'swap_mem_total': 3758092288} [2022-02-21 08:18:28,334] - [basetestcase:585] INFO - 172.23.123.50:8091 => {'services': ['kv'], 'cpu_utilization': 0.4017072558373085, 'mem_free': 3629207552, 'mem_total': 4201676800, 'swap_mem_used': 70778880, 'swap_mem_total': 3758092288} [2022-02-21 08:18:28,334] - [basetestcase:585] INFO - 172.23.123.51:8091 => {'services': ['kv'], 'cpu_utilization': 26.61654135338346, 'mem_free': 3591221248, 'mem_total': 4201672704, 'swap_mem_used': 57147392, 'swap_mem_total': 3758092288} [2022-02-21 08:18:28,334] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-02-21 08:18:28,334] - [basetestcase:559] INFO - Time to execute basesetup : 75.0210874080658 [2022-02-21 08:18:28,446] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:18:28,494] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:18:28,542] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:18:28,638] - [data_helper:305] INFO - dict:{'ip': '172.23.123.49', 'port': '8091', 'username': 'Administrator', 'password': 'password'} [2022-02-21 08:18:28,638] - [data_helper:306] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:18:28,674] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '172.23.123.49', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10 on default [2022-02-21 08:18:28,674] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10 [2022-02-21 08:18:28,675] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '172.23.123.49', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10, result: (2699164694, 0, b'') [2022-02-21 08:18:28,686] - [data_helper:305] INFO - dict:{'ip': '172.23.123.50', 'port': '8091', 'username': 'Administrator', 'password': 'password'} [2022-02-21 08:18:28,686] - [data_helper:306] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:18:28,723] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '172.23.123.50', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10 on default [2022-02-21 08:18:28,723] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10 [2022-02-21 08:18:28,723] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '172.23.123.50', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10, result: (3661772246, 0, b'') [2022-02-21 08:18:28,735] - [data_helper:305] INFO - dict:{'ip': '172.23.123.51', 'port': '8091', 'username': 'Administrator', 'password': 'password'} [2022-02-21 08:18:28,735] - [data_helper:306] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:18:28,801] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '172.23.123.51', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10 on default [2022-02-21 08:18:28,801] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10 [2022-02-21 08:18:28,801] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '172.23.123.51', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10, result: (4170252650, 0, b'') [2022-02-21 08:18:29,810] - [basetestcase:1445] INFO - create 100000 documents... Loading keys to reach dgm limit using python sdk [2022-02-21 08:18:29,851] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:18:29,899] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:18:29,985] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:18:30,378] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:18:30,383] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:18:30,485] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:18:30,489] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:18:30,537] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:18:30,539] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:19:30,104] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 100, desired: 5.0 bucket default [2022-02-21 08:19:30,311] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:19:30,407] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:19:30,465] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:19:31,502] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:19:31,508] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:19:31,551] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:19:31,558] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:19:31,604] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:19:31,606] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:20:30,530] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 100, desired: 5.0 bucket default [2022-02-21 08:20:30,731] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:20:30,780] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:20:30,868] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:20:31,526] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:20:31,530] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:20:31,634] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:20:31,640] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:20:31,683] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:20:31,687] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:21:30,989] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 100, desired: 5.0 bucket default [2022-02-21 08:21:31,181] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:21:31,231] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:21:31,280] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:21:31,598] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:21:31,603] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:21:31,650] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:21:31,658] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:21:31,754] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:21:31,759] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:22:31,437] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 100, desired: 5.0 bucket default [2022-02-21 08:22:31,681] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:22:31,738] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:22:31,794] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:22:32,672] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:22:32,681] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:22:32,722] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:22:32,733] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:22:32,836] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:22:32,843] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:23:31,963] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 100, desired: 5.0 bucket default [2022-02-21 08:23:32,220] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:23:32,269] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:23:32,320] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:23:32,739] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:23:32,743] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:23:32,790] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:23:32,796] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:23:32,904] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:23:32,905] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:24:32,488] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 71.378, desired: 5.0 bucket default [2022-02-21 08:24:32,749] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:24:32,799] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:24:32,851] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:24:33,827] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:24:33,881] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:24:33,991] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:25:32,992] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 46.2812, desired: 5.0 bucket default [2022-02-21 08:25:33,197] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:25:33,306] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:25:33,364] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:25:33,950] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:25:33,997] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:25:34,051] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:26:33,502] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 31.502, desired: 5.0 bucket default [2022-02-21 08:26:33,700] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:26:33,802] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:26:33,858] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:26:35,020] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:26:35,069] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:26:35,119] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:27:34,002] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 23.108, desired: 5.0 bucket default [2022-02-21 08:27:34,205] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:27:34,303] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:27:34,359] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:27:35,086] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:27:35,137] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:27:35,184] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:28:34,483] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 15.544, desired: 5.0 bucket default [2022-02-21 08:28:34,873] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:28:34,978] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:28:35,035] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:28:36,144] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:28:36,198] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:28:36,247] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:29:35,167] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 8.81311111111111, desired: 5.0 bucket default [2022-02-21 08:29:35,369] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:29:35,478] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:29:35,535] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:29:36,275] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:29:36,328] - [data_helper:308] INFO - creating direct client 172.23.123.50:11210 default [2022-02-21 08:29:36,377] - [data_helper:308] INFO - creating direct client 172.23.123.51:11210 default [2022-02-21 08:30:35,660] - [rest_client:2535] INFO - http://172.23.123.49:8091/pools/default/buckets/default/stats?zoom=minute Current resident ratio: 4.579, desired: 5.0 bucket default [2022-02-21 08:30:35,823] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default Doc size=512 bytes, Number of docs=167000 [2022-02-21 08:30:35,989] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-02-21 08:30:35,989] - [basetestcase:585] INFO - 172.23.123.49:8091 => {'services': ['kv'], 'cpu_utilization': 1.55896404324868, 'mem_free': 3307282432, 'mem_total': 4201672704, 'swap_mem_used': 2883584, 'swap_mem_total': 3758092288} [2022-02-21 08:30:35,989] - [basetestcase:585] INFO - 172.23.123.50:8091 => {'services': ['kv'], 'cpu_utilization': 1.331658291457287, 'mem_free': 3432022016, 'mem_total': 4201676800, 'swap_mem_used': 70778880, 'swap_mem_total': 3758092288} [2022-02-21 08:30:35,989] - [basetestcase:585] INFO - 172.23.123.51:8091 => {'services': ['kv'], 'cpu_utilization': 40.38316107890093, 'mem_free': 3395493888, 'mem_total': 4201672704, 'swap_mem_used': 57147392, 'swap_mem_total': 3758092288} [2022-02-21 08:30:35,990] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-02-21 08:30:35,991] - [remote_util:302] INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 [2022-02-21 08:30:36,089] - [remote_util:338] INFO - SSH Connected to 172.23.123.49 as root [2022-02-21 08:30:36,346] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:30:36,631] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:30:36,633] - [remote_util:302] INFO - SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 [2022-02-21 08:30:36,735] - [remote_util:338] INFO - SSH Connected to 172.23.123.51 as root [2022-02-21 08:30:37,003] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:30:37,301] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:30:37,303] - [remote_util:302] INFO - SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 [2022-02-21 08:30:37,404] - [remote_util:338] INFO - SSH Connected to 172.23.123.50 as root [2022-02-21 08:30:37,661] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:30:37,947] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:30:41,278] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #2 test_MB_32114 ============== [2022-02-21 08:30:41,303] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.123.49 [2022-02-21 08:30:42,143] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-02-21 08:30:42,146] - [rest_client:140] INFO - node 172.23.123.49 existing buckets : [] [2022-02-21 08:30:42,146] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.123.49 [2022-02-21 08:30:42,152] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.123.51, nothing to delete [2022-02-21 08:30:42,157] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.123.50, nothing to delete [2022-02-21 08:30:42,159] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-02-21 08:30:42,167] - [rest_client:1073] ERROR - DELETE http://172.23.123.49:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-02-21 08:30:42,168] - [basetestcase:724] INFO - b'"User was not found."' [2022-02-21 08:30:42,170] - [rest_client:44] INFO - -->is_ns_server_running? [2022-02-21 08:30:42,180] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-02-21 08:30:42,181] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.49,ns_1@172.23.123.50,ns_1@172.23.123.51', 'ejectedNodes': 'ns_1@172.23.123.50,ns_1@172.23.123.51', 'user': 'Administrator', 'password': 'password'} [2022-02-21 08:30:42,230] - [rest_client:1850] INFO - rebalance operation started [2022-02-21 08:30:42,232] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-02-21 08:31:02,257] - [rest_client:1951] INFO - rebalance progress took 20.03 seconds [2022-02-21 08:31:02,257] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-02-21 08:31:12,278] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.123.49 port:8091 ssh_username:root ? [('ns_1@172.23.123.50', 8091), ('ns_1@172.23.123.51', 8091)] [2022-02-21 08:31:12,282] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.123.49:8091 [2022-02-21 08:31:12,282] - [rest_client:44] INFO - -->is_ns_server_running? [2022-02-21 08:31:12,285] - [cluster_helper:94] INFO - ns_server @ 172.23.123.49:8091 is running [2022-02-21 08:31:12,287] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.123.51:8091 [2022-02-21 08:31:12,288] - [rest_client:44] INFO - -->is_ns_server_running? [2022-02-21 08:31:12,290] - [cluster_helper:94] INFO - ns_server @ 172.23.123.51:8091 is running [2022-02-21 08:31:12,292] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.123.50:8091 [2022-02-21 08:31:12,292] - [rest_client:44] INFO - -->is_ns_server_running? [2022-02-21 08:31:12,295] - [cluster_helper:94] INFO - ns_server @ 172.23.123.50:8091 is running [2022-02-21 08:31:12,295] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #2 test_MB_32114 ============== [2022-02-21 08:31:12,295] - [basetestcase:755] INFO - closing all ssh connections [2022-02-21 08:31:12,295] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite epengine.basic_ops.basic_ops , pass 0 , fail 2 failures so far... epengine.basic_ops.basic_ops.test_compression_active_and_off epengine.basic_ops.basic_ops.test_MB_32114 testrunner logs, diags and results are available under /data/workspace/centos-p0-epeng-vset00-00-basic_ops/logs/testrunner-22-Feb-21_08-16-39/test_2 ERROR ====================================================================== ERROR: test_MB_32114 (epengine.basic_ops.basic_ops) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/epengine/basic_ops.py", line 125, in test_MB_32114 rc = mcd.del_with_meta(KEY_NAME + str(i), 0, 0, 2, cas + 1) File "lib/mc_bin_client.py", line 448, in del_with_meta exp, seqno, cas, options, 0), scope=scope, collection=collection) File "lib/mc_bin_client.py", line 229, in _doCmd return self._handleSingleResponse(opaque) File "lib/mc_bin_client.py", line 220, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "lib/mc_bin_client.py", line 216, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.MemcachedError: Memcached error #2 'Exists': b'' ---------------------------------------------------------------------- Ran 1 test in 838.987s FAILED (errors=1) test_MB_36087 (epengine.basic_ops.basic_ops) ... Logs will be stored at /data/workspace/centos-p0-epeng-vset00-00-basic_ops/logs/testrunner-22-Feb-21_08-16-39/test_3 ./testrunner -i /tmp/testexec.129923.ini -p skip_log_scan=False,bucket_storage=couchstore -t epengine.basic_ops.basic_ops.test_MB_36087,items=1,nodes_init=1,replicas=0,skip_log_scan=False,bucket_storage=couchstore Test Input params: {'items': '1', 'nodes_init': '1', 'replicas': '0', 'skip_log_scan': 'False', 'bucket_storage': 'couchstore', 'ini': '/tmp/testexec.129923.ini', 'cluster_name': 'testexec.129923', 'spec': 'py-epeng-basic-ops', 'conf_file': 'py-epeng-basic-ops.conf', 'num_nodes': 3, 'case_number': 3, 'last_case_fail': 'True', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centos-p0-epeng-vset00-00-basic_ops/logs/testrunner-22-Feb-21_08-16-39/test_3'} [2022-02-21 08:31:12,312] - [remote_util:302] INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 [2022-02-21 08:31:12,407] - [remote_util:338] INFO - SSH Connected to 172.23.123.49 as root [2022-02-21 08:31:12,657] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:31:12,943] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:31:12,947] - [rest_client:44] INFO - -->is_ns_server_running? [2022-02-21 08:31:12,956] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2335-enterprise [2022-02-21 08:31:12,956] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #3 test_MB_36087============== [2022-02-21 08:31:12,956] - [basetestcase:331] INFO - initializing cluster [2022-02-21 08:31:13,316] - [task:154] INFO - server: ip:172.23.123.49 port:8091 ssh_username:root, nodes/self [2022-02-21 08:31:13,319] - [task:159] INFO - {'uptime': '899', 'memoryTotal': 4201672704, 'memoryFree': 3425824768, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.49:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@172.23.123.49', 'ip': '172.23.123.49', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2022-02-21 08:31:13,323] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-02-21 08:31:13,327] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-02-21 08:31:13,327] - [rest_client:1149] INFO - settings/web params on 172.23.123.49:8091:port=8091&username=Administrator&password=password [2022-02-21 08:31:13,375] - [rest_client:1151] INFO - --> status:True [2022-02-21 08:31:13,376] - [remote_util:302] INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 [2022-02-21 08:31:13,472] - [remote_util:338] INFO - SSH Connected to 172.23.123.49 as root [2022-02-21 08:31:13,725] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:31:14,015] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:31:14,015] - [remote_util:3317] INFO - running command.raw on 172.23.123.49: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-02-21 08:31:14,041] - [remote_util:3366] INFO - command executed successfully with root [2022-02-21 08:31:14,041] - [remote_util:5200] INFO - ['ok'] [2022-02-21 08:31:14,043] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.49:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:31:14,047] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.49:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:31:14,051] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-02-21 08:31:14,064] - [task:154] INFO - server: ip:172.23.123.51 port:8091 ssh_username:root, nodes/self [2022-02-21 08:31:14,066] - [task:159] INFO - {'uptime': '897', 'memoryTotal': 4201672704, 'memoryFree': 3586531328, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.51:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.123.51', 'ip': '172.23.123.51', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2022-02-21 08:31:14,068] - [rest_client:1073] ERROR - GET http://172.23.123.51:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-02-21 08:31:14,069] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-02-21 08:31:14,072] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-02-21 08:31:14,072] - [rest_client:1149] INFO - settings/web params on 172.23.123.51:8091:port=8091&username=Administrator&password=password [2022-02-21 08:31:14,125] - [rest_client:1151] INFO - --> status:True [2022-02-21 08:31:14,126] - [remote_util:302] INFO - SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 [2022-02-21 08:31:14,232] - [remote_util:338] INFO - SSH Connected to 172.23.123.51 as root [2022-02-21 08:31:14,501] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:31:14,804] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:31:14,805] - [remote_util:3317] INFO - running command.raw on 172.23.123.51: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-02-21 08:31:14,830] - [remote_util:3366] INFO - command executed successfully with root [2022-02-21 08:31:14,831] - [remote_util:5200] INFO - ['ok'] [2022-02-21 08:31:14,833] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.51:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:31:14,834] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.51:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:31:14,837] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-02-21 08:31:14,849] - [task:154] INFO - server: ip:172.23.123.50 port:8091 ssh_username:root, nodes/self [2022-02-21 08:31:14,851] - [task:159] INFO - {'uptime': '897', 'memoryTotal': 4201676800, 'memoryFree': 3609780224, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.50:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.123.50', 'ip': '172.23.123.50', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2022-02-21 08:31:14,853] - [rest_client:1073] ERROR - GET http://172.23.123.50:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-02-21 08:31:14,853] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-02-21 08:31:14,856] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-02-21 08:31:14,856] - [rest_client:1149] INFO - settings/web params on 172.23.123.50:8091:port=8091&username=Administrator&password=password [2022-02-21 08:31:14,912] - [rest_client:1151] INFO - --> status:True [2022-02-21 08:31:14,913] - [remote_util:302] INFO - SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 [2022-02-21 08:31:15,011] - [remote_util:338] INFO - SSH Connected to 172.23.123.50 as root [2022-02-21 08:31:15,262] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:31:15,546] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:31:15,546] - [remote_util:3317] INFO - running command.raw on 172.23.123.50: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-02-21 08:31:15,608] - [remote_util:3366] INFO - command executed successfully with root [2022-02-21 08:31:15,608] - [remote_util:5200] INFO - ['ok'] [2022-02-21 08:31:15,610] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.50:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:31:15,612] - [rest_client:1868] INFO - /diag/eval status on 172.23.123.50:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-02-21 08:31:15,615] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-02-21 08:31:15,626] - [basetestcase:2377] INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.49 **** [2022-02-21 08:31:15,697] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-02-21 08:31:20,703] - [basetestcase:2382] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-02-21 08:31:20,713] - [basetestcase:369] INFO - done initializing cluster [2022-02-21 08:31:20,714] - [remote_util:302] INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 [2022-02-21 08:31:20,812] - [remote_util:338] INFO - SSH Connected to 172.23.123.49 as root [2022-02-21 08:31:21,070] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:31:21,357] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:31:21,357] - [remote_util:3317] INFO - running command.raw on 172.23.123.49: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-02-21 08:31:21,419] - [remote_util:3366] INFO - command executed successfully with root [2022-02-21 08:31:21,419] - [remote_util:5200] INFO - ['ok'] [2022-02-21 08:31:21,419] - [basetestcase:2970] INFO - Enabled diag/eval for non-local hosts from 172.23.123.49 [2022-02-21 08:31:21,643] - [rest_client:2989] INFO - http://172.23.123.49:8091/pools/default/buckets with param: name=default&ramQuotaMB=2147&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-02-21 08:31:21,669] - [rest_client:3014] INFO - 0.03 seconds to create bucket default [2022-02-21 08:31:21,669] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.123.49 to accept set ops [2022-02-21 08:31:22,229] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:31:22,337] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:31:23,353] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 2147 [2022-02-21 08:31:23,355] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #3 test_MB_36087 ============== [2022-02-21 08:31:23,361] - [remote_util:302] INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 [2022-02-21 08:31:23,463] - [remote_util:338] INFO - SSH Connected to 172.23.123.49 as root [2022-02-21 08:31:23,719] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:31:24,005] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:31:24,006] - [remote_util:302] INFO - SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 [2022-02-21 08:31:24,113] - [remote_util:338] INFO - SSH Connected to 172.23.123.51 as root [2022-02-21 08:31:24,374] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:31:24,678] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:31:24,679] - [remote_util:302] INFO - SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 [2022-02-21 08:31:24,775] - [remote_util:338] INFO - SSH Connected to 172.23.123.50 as root [2022-02-21 08:31:25,027] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:31:25,312] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:31:28,122] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-02-21 08:31:28,122] - [basetestcase:585] INFO - 172.23.123.49:8091 => {'services': ['kv'], 'cpu_utilization': 2.71424981151043, 'mem_free': 3425763328, 'mem_total': 4201672704, 'swap_mem_used': 2883584, 'swap_mem_total': 3758092288} [2022-02-21 08:31:28,122] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-02-21 08:31:28,122] - [basetestcase:559] INFO - Time to execute basesetup : 15.811471700668335 [2022-02-21 08:31:28,167] - [data_helper:308] INFO - creating direct client 172.23.123.49:11210 default [2022-02-21 08:31:28,203] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-02-21 08:31:28,203] - [basetestcase:585] INFO - 172.23.123.49:8091 => {'services': ['kv'], 'cpu_utilization': 2.71424981151043, 'mem_free': 3425763328, 'mem_total': 4201672704, 'swap_mem_used': 2883584, 'swap_mem_total': 3758092288} [2022-02-21 08:31:28,203] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-02-21 08:31:28,204] - [remote_util:302] INFO - SSH Connecting to 172.23.123.49 with username:root, attempt#1 of 5 [2022-02-21 08:31:28,302] - [remote_util:338] INFO - SSH Connected to 172.23.123.49 as root [2022-02-21 08:31:28,558] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:31:28,845] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:31:28,847] - [remote_util:302] INFO - SSH Connecting to 172.23.123.51 with username:root, attempt#1 of 5 [2022-02-21 08:31:28,946] - [remote_util:338] INFO - SSH Connected to 172.23.123.51 as root [2022-02-21 08:31:29,206] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:31:29,507] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:31:29,508] - [remote_util:302] INFO - SSH Connecting to 172.23.123.50 with username:root, attempt#1 of 5 [2022-02-21 08:31:29,609] - [remote_util:338] INFO - SSH Connected to 172.23.123.50 as root [2022-02-21 08:31:29,860] - [remote_util:3480] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-02-21 08:31:30,152] - [remote_util:3637] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-02-21 08:31:32,634] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #3 test_MB_36087 ============== [2022-02-21 08:31:32,653] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.123.49 [2022-02-21 08:31:33,045] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-02-21 08:31:33,048] - [rest_client:140] INFO - node 172.23.123.49 existing buckets : [] [2022-02-21 08:31:33,048] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.123.49 [2022-02-21 08:31:33,052] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.123.51, nothing to delete [2022-02-21 08:31:33,056] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.123.50, nothing to delete [2022-02-21 08:31:33,058] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-02-21 08:31:33,065] - [rest_client:1073] ERROR - DELETE http://172.23.123.49:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-02-21 08:31:33,066] - [basetestcase:724] INFO - b'"User was not found."' [2022-02-21 08:31:33,068] - [rest_client:44] INFO - -->is_ns_server_running? [2022-02-21 08:31:33,079] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.123.49:8091 [2022-02-21 08:31:33,079] - [rest_client:44] INFO - -->is_ns_server_running? [2022-02-21 08:31:33,081] - [cluster_helper:94] INFO - ns_server @ 172.23.123.49:8091 is running [2022-02-21 08:31:33,084] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.123.51:8091 [2022-02-21 08:31:33,084] - [rest_client:44] INFO - -->is_ns_server_running? [2022-02-21 08:31:33,086] - [cluster_helper:94] INFO - ns_server @ 172.23.123.51:8091 is running [2022-02-21 08:31:33,088] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.123.50:8091 [2022-02-21 08:31:33,088] - [rest_client:44] INFO - -->is_ns_server_running? [2022-02-21 08:31:33,091] - [cluster_helper:94] INFO - ns_server @ 172.23.123.50:8091 is running [2022-02-21 08:31:33,091] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #3 test_MB_36087 ============== [2022-02-21 08:31:33,091] - [basetestcase:755] INFO - closing all ssh connections [2022-02-21 08:31:33,091] - [basetestcase:760] INFO - closing all memcached connections ERROR ====================================================================== ERROR: test_MB_36087 (epengine.basic_ops.basic_ops) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/epengine/basic_ops.py", line 169, in test_MB_36087 rc = mcd.evict_key(key) File "lib/mc_bin_client.py", line 706, in evict_key return self._doCmd(memcacheConstants.CMD_EVICT_KEY, key, '', scope=scope, collection=collection) File "lib/mc_bin_client.py", line 229, in _doCmd return self._handleSingleResponse(opaque) File "lib/mc_bin_client.py", line 220, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "lib/mc_bin_client.py", line 216, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.MemcachedError: Memcached error #2 'Exists': b'' ---------------------------------------------------------------------- Ran 1 test in 20.784s FAILED (errors=1) suite_tearDown (unittest.loader._FailedTest) ... *** TestRunner *** workspace is /data/workspace/centos-p0-epeng-vset00-00-basic_ops fails is 3 3 Desc1: 7.1.0-2335 - epeng basic_ops - centos (0/3) python3 scripts/rerun_jobs.py 7.1.0-2335 --executor_jenkins_job --run_params=skip_log_scan=False,bucket_storage=couchstore INFO:merge_reports:Merging of report files from ['Old_Report_mergedreport-22-Feb-21_00-27-06-epengine.basic_ops.basic_ops.xml', 'logs/**/*.xml'] INFO:merge_reports:-- Old_Report_mergedreport-22-Feb-21_00-27-06-epengine.basic_ops.basic_ops.xml -- INFO:merge_reports:-- logs/testrunner-22-Feb-21_08-16-39/report-22-Feb-21_08-16-39-epengine.basic_ops.basic_ops.xml -- INFO:merge_reports: Number of TestSuites=1 INFO:merge_reports: TestSuite#1) epengine.basic_ops.basic_ops, Number of Tests=16 INFO:merge_reports:Summary file is at /data/workspace/centos-p0-epeng-vset00-00-basic_ops/logs/testrunner-22-Feb-21_08-31-33/merged_summary/mergedreport-22-Feb-21_08-31-33-epengine.basic_ops.basic_ops.xml Merging xmls summary so far suite epengine.basic_ops.basic_ops , pass 13 , fail 3 failures so far... epengine.basic_ops.basic_ops.test_compression_active_and_off epengine.basic_ops.basic_ops.test_MB_32114 epengine.basic_ops.basic_ops.test_MB_36087 merged xmls No more failed tests. Stopping reruns [description-setter] Description set: 7.1.0-2335 - epeng basic_ops - centos (0/3) [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) Sending email to: hemant.rajput@couchbase.com Triggering a new build of savejoblogs Triggering a new build of test-executor-cleanup Triggering a new build of post-to-gerrit Triggering a new build of test-executor-cleanup-aws Finished: UNSTABLE