Started by remote host 172.23.109.52 [EnvInject] - Loading node environment variables. Building remotely on slv-s12713-fts (P0) in workspace /root/jenkins/workspace/centos-p0-2i-vset00-00-replica-indexes-plasma-P0 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Running Prebuild steps [centos-p0-2i-vset00-00-replica-indexes-plasma-P0] $ /bin/sh -xe /tmp/jenkins3710133115503372151.sh ++ echo 2i-replica-indexes-plasma-P0-Feb-17-21:29:17-7.1.0-2284 ++ awk '{split($0,r,"-");print r[1],r[2]}' + desc='2i replica' + echo Desc: 7.1.0-2284 - 2i replica - centos Desc: 7.1.0-2284 - 2i replica - centos + echo newState=available + newState=available Success build forhudson.tasks.Shell@5ee7b57d [description-setter] Description set: 7.1.0-2284 - 2i replica - centos Success build forhudson.plugins.descriptionsetter.DescriptionSetterBuilder@58d0b419 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Success build fororg.jenkinsci.plugins.envinject.EnvInjectBuilder@72813b5f Cloning the remote Git repository Using shallow clone Cloning repository https://github.com/couchbase/testrunner > /usr/bin/git init /root/jenkins/workspace/centos-p0-2i-vset00-00-replica-indexes-plasma-P0 # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 7cafc3afaee6e7c57fe74f16c67f795779dcb29e (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 7cafc3afaee6e7c57fe74f16c67f795779dcb29e > /usr/bin/git rev-list 7cafc3afaee6e7c57fe74f16c67f795779dcb29e # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #443889 jenkins-test_suite_executor-443889 # timeout=10 No emails were triggered. [centos-p0-2i-vset00-00-replica-indexes-plasma-P0] $ /bin/sh -xe /tmp/jenkins917845180713788082.sh + echo Desc: 2i-replica-indexes-plasma-P0-Feb-17-21:29:17-7.1.0-2284 Desc: 2i-replica-indexes-plasma-P0-Feb-17-21:29:17-7.1.0-2284 [description-setter] Could not determine description. [centos-p0-2i-vset00-00-replica-indexes-plasma-P0] $ /bin/sh -xe /tmp/jenkins7714484105301921340.sh [centos-p0-2i-vset00-00-replica-indexes-plasma-P0] $ /bin/sh -xe /tmp/jenkins6257560666866062838.sh + py_executable=python2 + echo 7.1.0-2284 + grep -E '7\.0|7\.1|0\.0' 7.1.0-2284 + py_executable=python3 + [[ 7.1.0-2284 > 6.5 ]] + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + rerun_job=true + touch rerun_props_file + '[' false == false ']' + '[' true == true ']' + python3 scripts/rerun_jobs.py 7.1.0-2284 --executor_jenkins_job --manual_run [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'rerun_props_file' [EnvInject] - Variables injected successfully. [centos-p0-2i-vset00-00-replica-indexes-plasma-P0] $ /bin/bash /tmp/jenkins1137940625611759256.sh 7.1.0-2284 Already on 'master' From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. Set ALLOW_HTP to False so test could run. HEAD is now at 7cafc3a Fixes for gsi job failure the major release is 7 "172.23.108.42","172.23.108.36","172.23.108.108","172.23.107.250" Searching for httplib2 Best match: httplib2 0.17.0 Adding httplib2 0.17.0 to easy-install.pth file Using /usr/local/lib/python3.7/site-packages Processing dependencies for httplib2 Finished processing dependencies for httplib2 centos Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.sfo12.us.leaseweb.net * extras: la.mirrors.clouvider.net * updates: mirror.sfo12.us.leaseweb.net Package 2:docker-1.13.1-208.git7d71120.el7_9.x86_64 already installed and latest version Nothing to do Using default tag: latest Trying to pull repository docker.io/jamesdbloom/mockserver ... latest: Pulling from docker.io/jamesdbloom/mockserver Digest: sha256:bc41010b9e920f5e1b75a226e8eda05eb4eb9843dad042e9e3950d1b7de823f4 Status: Image is up to date for docker.io/jamesdbloom/mockserver:latest [global] port:8091 username:root password:couchbase index_port:9102 n1ql_port:8093 [servers] 1:vm1 2:vm2 3:vm3 4:vm4 [vm1] ip:dynamic services:n1ql,kv,index [vm2] ip:dynamic services:kv,index,n1ql [vm3] ip:dynamic services:kv,index [vm4] ip:dynamic services:kv [membase] rest_username:Administrator rest_password:password python3 scripts/populateIni.py -s "172.23.108.42","172.23.108.36","172.23.108.108","172.23.107.250" -d None -a None -i /tmp/testexec_reformat.27486.ini -p centos -o /tmp/testexec.27486.ini -k {} INFO:root:SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.107.250 as root INFO:root:SSH Connected to 172.23.108.42 as root INFO:root:SSH Connected to 172.23.108.36 as root INFO:root:SSH Connected to 172.23.108.108 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.36: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.107.250: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.42: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.108: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root in main the ini file is /tmp/testexec_reformat.27486.ini the given server info is "172.23.108.42","172.23.108.36","172.23.108.108","172.23.107.250" Collecting memory info from 172.23.108.36 Collecting memory info from 172.23.107.250 Collecting memory info from 172.23.108.42 Collecting memory info from 172.23.108.108 the servers memory info is [('172.23.107.250', 3853296), ('172.23.108.36', 3853296), ('172.23.108.42', 3853296), ('172.23.108.108', 3853296)] [global] port:8091 username:root password:couchbase index_port:9102 n1ql_port:8093 [servers] 1:vm1 2:vm2 3:vm3 4:vm4 [vm1] ip:172.23.107.250 services:n1ql,kv,index [vm2] ip:172.23.108.36 services:kv,index,n1ql [vm3] ip:172.23.108.42 services:kv,index [vm4] ip:172.23.108.108 services:kv [membase] rest_username:Administrator rest_password:password extra install is Local time: Thu 2022-02-17 21:29:30 PST Universal time: Fri 2022-02-18 05:29:30 UTC RTC time: Fri 2022-02-18 05:29:30 Time zone: America/Los_Angeles (PST, -0800) NTP enabled: yes NTP synchronized: yes RTC in local TZ: no DST active: no Last DST change: DST ended at Sun 2021-11-07 01:59:59 PDT Sun 2021-11-07 01:00:00 PST Next DST change: DST begins (the clock jumps one hour forward) at Sun 2022-03-13 01:59:59 PST Sun 2022-03-13 03:00:00 PDT python3 scripts/ssh.py -i /tmp/testexec_root.27486.ini iptables -F INFO:root:SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.107.250 as root INFO:root:SSH Connected to 172.23.108.42 as root INFO:root:SSH Connected to 172.23.108.108 as root INFO:root:SSH Connected to 172.23.108.36 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.107.250: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.108: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.36: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.42: iptables -F INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root 172.23.107.250 172.23.108.108 172.23.108.36 172.23.108.42 Initial version: python3 scripts/new_install.py -i /tmp/testexec.27486.ini -p timeout=1800,skip_local_download=False,get-cbcollect-info=True,version=7.1.0-2284,product=cb,debug_logs=True,ntp=True,url= 2022-02-17 21:29:32,867 - root - WARNING - URL: is not valid, will use version to locate build 2022-02-17 21:29:32,868 - root - INFO - SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:29:33,028 - root - INFO - SSH Connected to 172.23.107.250 as root 2022-02-17 21:29:33,390 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:29:33,808 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:29:33,810 - root - INFO - SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 2022-02-17 21:29:33,972 - root - INFO - SSH Connected to 172.23.108.36 as root 2022-02-17 21:29:34,359 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:29:34,787 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:29:34,788 - root - INFO - SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 2022-02-17 21:29:34,950 - root - INFO - SSH Connected to 172.23.108.42 as root 2022-02-17 21:29:35,301 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:29:35,732 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:29:35,734 - root - INFO - SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 2022-02-17 21:29:35,896 - root - INFO - SSH Connected to 172.23.108.108 as root 2022-02-17 21:29:36,275 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:29:36,705 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:29:36,708 - root - INFO - SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:29:36,862 - root - INFO - SSH Connected to 172.23.107.250 as root 2022-02-17 21:29:37,229 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:29:37,651 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:29:37,654 - root - INFO - SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 2022-02-17 21:29:37,812 - root - INFO - SSH Connected to 172.23.108.36 as root 2022-02-17 21:29:38,180 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:29:38,605 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:29:38,607 - root - INFO - SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 2022-02-17 21:29:38,767 - root - INFO - SSH Connected to 172.23.108.42 as root 2022-02-17 21:29:39,130 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:29:39,559 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:29:39,561 - root - INFO - SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 2022-02-17 21:29:39,730 - root - INFO - SSH Connected to 172.23.108.108 as root 2022-02-17 21:29:40,110 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:29:40,534 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:29:40,535 - root - INFO - Check if ntp is installed 2022-02-17 21:29:40,536 - root - INFO - Check if ntp is installed 2022-02-17 21:29:40,536 - root - INFO - Check if ntp is installed 2022-02-17 21:29:40,536 - root - INFO - Check if ntp is installed 2022-02-17 21:29:40,536 - root - INFO - This OS version centos 7 2022-02-17 21:29:40,536 - root - INFO - This OS version centos 7 2022-02-17 21:29:40,537 - root - INFO - running command.raw on 172.23.108.36: systemctl status ntpd 2022-02-17 21:29:40,537 - root - INFO - running command.raw on 172.23.107.250: systemctl status ntpd 2022-02-17 21:29:40,537 - root - INFO - This OS version centos 7 2022-02-17 21:29:40,537 - root - INFO - running command.raw on 172.23.108.42: systemctl status ntpd 2022-02-17 21:29:40,539 - root - INFO - This OS version centos 7 2022-02-17 21:29:40,539 - root - INFO - running command.raw on 172.23.108.108: systemctl status ntpd 2022-02-17 21:29:40,595 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,595 - root - INFO - running command.raw on 172.23.108.42: ntpstat 2022-02-17 21:29:40,598 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,598 - root - INFO - running command.raw on 172.23.108.36: ntpstat 2022-02-17 21:29:40,600 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,600 - root - INFO - running command.raw on 172.23.107.250: ntpstat 2022-02-17 21:29:40,630 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,630 - root - INFO - running command.raw on 172.23.108.108: ntpstat 2022-02-17 21:29:40,722 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,722 - root - INFO - running command.raw on 172.23.108.42: timedatectl status 2022-02-17 21:29:40,726 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,726 - root - INFO - running command.raw on 172.23.108.36: timedatectl status 2022-02-17 21:29:40,728 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,729 - root - INFO - running command.raw on 172.23.107.250: timedatectl status 2022-02-17 21:29:40,754 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,755 - root - INFO - running command.raw on 172.23.108.108: timedatectl status 2022-02-17 21:29:40,817 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,818 - root - INFO - running command.raw on 172.23.108.42: date 2022-02-17 21:29:40,820 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,820 - root - INFO - running command.raw on 172.23.108.36: date 2022-02-17 21:29:40,861 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,861 - root - INFO - running command.raw on 172.23.107.250: date 2022-02-17 21:29:40,888 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,888 - root - INFO - running command.raw on 172.23.108.108: date 2022-02-17 21:29:40,889 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,892 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,921 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,933 - root - INFO - command executed successfully with root 2022-02-17 21:29:40,959 - root - INFO - ['Thu Feb 17 21:29:40 PST 2022'] IP: 172.23.108.42 2022-02-17 21:29:40,960 - root - INFO - ['Thu Feb 17 21:29:40 PST 2022'] IP: 172.23.108.36 2022-02-17 21:29:40,992 - root - INFO - ['Thu Feb 17 21:29:40 PST 2022'] IP: 172.23.108.108 2022-02-17 21:29:41,003 - root - INFO - ['Thu Feb 17 21:29:41 PST 2022'] IP: 172.23.107.250 2022-02-17 21:29:41,003 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2284/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:41,019 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2284/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm is live 2022-02-17 21:29:41,019 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2284/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:41,021 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2284/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm is live 2022-02-17 21:29:41,021 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2284/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:41,022 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2284/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm is live 2022-02-17 21:29:41,022 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2284/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:41,023 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2284/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm is live 2022-02-17 21:29:41,024 - root - INFO - running command.raw on 172.23.107.250: umount -a -t nfs,nfs4 -f -l;yes | yum remove 'couchbase*' > /dev/null; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2022-02-17 21:29:41,024 - root - INFO - running command.raw on 172.23.108.36: umount -a -t nfs,nfs4 -f -l;yes | yum remove 'couchbase*' > /dev/null; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2022-02-17 21:29:41,025 - root - INFO - running command.raw on 172.23.108.42: umount -a -t nfs,nfs4 -f -l;yes | yum remove 'couchbase*' > /dev/null; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2022-02-17 21:29:41,026 - root - INFO - running command.raw on 172.23.108.108: umount -a -t nfs,nfs4 -f -l;yes | yum remove 'couchbase*' > /dev/null; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2022-02-17 21:29:46,660 - root - INFO - command executed successfully with root 2022-02-17 21:29:46,693 - root - INFO - command executed successfully with root 2022-02-17 21:29:47,651 - root - INFO - Done with uninstall on 172.23.107.250. 2022-02-17 21:29:47,732 - root - INFO - Done with uninstall on 172.23.108.108. 2022-02-17 21:29:47,844 - root - INFO - command executed successfully with root 2022-02-17 21:29:47,938 - root - INFO - command executed successfully with root 2022-02-17 21:29:48,746 - root - INFO - Done with uninstall on 172.23.108.36. 2022-02-17 21:29:48,809 - root - INFO - Done with uninstall on 172.23.108.42. 2022-02-17 21:29:51,037 - root - INFO - Downloading build binary to /tmp/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm.. 2022-02-17 21:29:51,047 - root - INFO - running command.raw on 172.23.107.250: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,079 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,081 - root - INFO - running command.raw on 172.23.108.36: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,112 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,114 - root - INFO - running command.raw on 172.23.108.42: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,146 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,149 - root - INFO - running command.raw on 172.23.108.108: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,183 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,183 - root - INFO - running command.raw on 172.23.107.250: ls -lh /tmp/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,217 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,218 - root - INFO - running command.raw on 172.23.107.250: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,288 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,288 - root - INFO - running command.raw on 172.23.108.36: ls -lh /tmp/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,321 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,323 - root - INFO - running command.raw on 172.23.108.36: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,391 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,391 - root - INFO - running command.raw on 172.23.108.42: ls -lh /tmp/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,426 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,428 - root - INFO - running command.raw on 172.23.108.42: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,496 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,496 - root - INFO - running command.raw on 172.23.108.108: ls -lh /tmp/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,530 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,532 - root - INFO - running command.raw on 172.23.108.108: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm 2022-02-17 21:29:51,600 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,601 - root - INFO - running command.raw on 172.23.107.250: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2022-02-17 21:29:51,601 - root - INFO - running command.raw on 172.23.108.36: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2022-02-17 21:29:51,603 - root - INFO - running command.raw on 172.23.108.42: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2022-02-17 21:29:51,603 - root - INFO - running command.raw on 172.23.108.108: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2022-02-17 21:29:51,638 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,638 - root - INFO - running command.raw on 172.23.108.36: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2022-02-17 21:29:51,639 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,640 - root - INFO - running command.raw on 172.23.108.42: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2022-02-17 21:29:51,641 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,642 - root - INFO - running command.raw on 172.23.107.250: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2022-02-17 21:29:51,676 - root - INFO - command executed successfully with root 2022-02-17 21:29:51,676 - root - INFO - running command.raw on 172.23.108.108: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.1.0-2284-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2022-02-17 21:30:47,332 - root - INFO - command executed successfully with root 2022-02-17 21:30:47,332 - root - INFO - running command.raw on 172.23.108.42: systemctl -q is-active couchbase-server && echo 1 || echo 0 2022-02-17 21:30:47,418 - root - INFO - command executed successfully with root 2022-02-17 21:30:47,418 - root - INFO - Done with install on 172.23.108.42. 2022-02-17 21:30:47,418 - root - INFO - Waiting for couchbase to be reachable 2022-02-17 21:30:47,420 - root - ERROR - socket error while connecting to http://172.23.108.42:8091/nodes/self error [Errno 111] Connection refused 2022-02-17 21:30:47,971 - root - INFO - command executed successfully with root 2022-02-17 21:30:47,971 - root - INFO - running command.raw on 172.23.107.250: systemctl -q is-active couchbase-server && echo 1 || echo 0 2022-02-17 21:30:48,007 - root - INFO - command executed successfully with root 2022-02-17 21:30:48,007 - root - INFO - Done with install on 172.23.107.250. 2022-02-17 21:30:48,007 - root - INFO - Waiting for couchbase to be reachable 2022-02-17 21:30:48,009 - root - ERROR - socket error while connecting to http://172.23.107.250:8091/nodes/self error [Errno 111] Connection refused 2022-02-17 21:30:49,386 - root - INFO - command executed successfully with root 2022-02-17 21:30:49,386 - root - INFO - running command.raw on 172.23.108.36: systemctl -q is-active couchbase-server && echo 1 || echo 0 2022-02-17 21:30:49,443 - root - INFO - command executed successfully with root 2022-02-17 21:30:49,443 - root - INFO - Done with install on 172.23.108.36. 2022-02-17 21:30:49,443 - root - INFO - Waiting for couchbase to be reachable 2022-02-17 21:30:49,445 - root - ERROR - socket error while connecting to http://172.23.108.36:8091/nodes/self error [Errno 111] Connection refused 2022-02-17 21:30:50,425 - root - ERROR - socket error while connecting to http://172.23.108.42:8091/nodes/self error [Errno 111] Connection refused 2022-02-17 21:30:50,535 - root - INFO - command executed successfully with root 2022-02-17 21:30:50,535 - root - INFO - running command.raw on 172.23.108.108: systemctl -q is-active couchbase-server && echo 1 || echo 0 2022-02-17 21:30:50,566 - root - INFO - command executed successfully with root 2022-02-17 21:30:50,566 - root - INFO - Done with install on 172.23.108.108. 2022-02-17 21:30:50,566 - root - INFO - Waiting for couchbase to be reachable 2022-02-17 21:30:50,568 - root - ERROR - socket error while connecting to http://172.23.108.108:8091/nodes/self error [Errno 111] Connection refused 2022-02-17 21:30:51,011 - root - ERROR - socket error while connecting to http://172.23.107.250:8091/nodes/self error [Errno 111] Connection refused 2022-02-17 21:30:52,449 - root - ERROR - socket error while connecting to http://172.23.108.36:8091/nodes/self error [Errno 111] Connection refused 2022-02-17 21:30:53,572 - root - ERROR - socket error while connecting to http://172.23.108.108:8091/nodes/self error [Errno 111] Connection refused 2022-02-17 21:30:56,444 - root - INFO - running command.raw on 172.23.108.42: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.108.42 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-17 21:30:56,962 - root - INFO - command executed successfully with root 2022-02-17 21:30:57,025 - root - INFO - running command.raw on 172.23.107.250: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.107.250 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-17 21:30:57,539 - root - INFO - command executed successfully with root 2022-02-17 21:30:57,971 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.108.42 2022-02-17 21:30:57,972 - root - INFO - pools/default params : indexMemoryQuota=256 2022-02-17 21:30:57,985 - root - INFO - Setting KV memory quota as 2264 MB on 172.23.108.42 2022-02-17 21:30:57,985 - root - INFO - pools/default params : memoryQuota=2264 2022-02-17 21:30:57,991 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index']) 2022-02-17 21:30:57,991 - root - INFO - /node/controller/setupServices params on 172.23.108.42: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex 2022-02-17 21:30:58,029 - root - INFO - settings/indexes params : storageMode=plasma 2022-02-17 21:30:58,041 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-17 21:30:58,042 - root - INFO - settings/web params on 172.23.108.42:8091:port=8091&username=Administrator&password=password 2022-02-17 21:30:58,115 - root - INFO - --> status:True 2022-02-17 21:30:58,116 - root - INFO - Done with init on 172.23.108.42. 2022-02-17 21:30:58,116 - root - INFO - running command.raw on 172.23.108.42: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-02-17 21:30:58,147 - root - INFO - command executed successfully with root 2022-02-17 21:30:58,147 - root - INFO - Done with cleanup on 172.23.108.42. 2022-02-17 21:30:58,551 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.107.250 2022-02-17 21:30:58,551 - root - INFO - pools/default params : indexMemoryQuota=256 2022-02-17 21:30:58,564 - root - INFO - Setting KV memory quota as 2264 MB on 172.23.107.250 2022-02-17 21:30:58,564 - root - INFO - pools/default params : memoryQuota=2264 2022-02-17 21:30:58,571 - root - INFO - --> init_node_services(Administrator,password,None,8091,['n1ql', 'kv', 'index']) 2022-02-17 21:30:58,571 - root - INFO - /node/controller/setupServices params on 172.23.107.250: 8091:hostname=None&user=Administrator&password=password&services=n1ql%2Ckv%2Cindex 2022-02-17 21:30:58,608 - root - INFO - settings/indexes params : storageMode=plasma 2022-02-17 21:30:58,620 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-17 21:30:58,620 - root - INFO - settings/web params on 172.23.107.250:8091:port=8091&username=Administrator&password=password 2022-02-17 21:30:58,654 - root - INFO - running command.raw on 172.23.108.36: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.108.36 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-17 21:30:58,687 - root - INFO - --> status:True 2022-02-17 21:30:58,687 - root - INFO - Done with init on 172.23.107.250. 2022-02-17 21:30:58,687 - root - INFO - running command.raw on 172.23.107.250: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-02-17 21:30:58,723 - root - INFO - command executed successfully with root 2022-02-17 21:30:58,723 - root - INFO - Done with cleanup on 172.23.107.250. 2022-02-17 21:30:59,135 - root - INFO - command executed successfully with root 2022-02-17 21:30:59,580 - root - ERROR - socket error while connecting to http://172.23.108.108:8091/nodes/self error [Errno 111] Connection refused 2022-02-17 21:31:00,147 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.108.36 2022-02-17 21:31:00,148 - root - INFO - pools/default params : indexMemoryQuota=256 2022-02-17 21:31:00,161 - root - INFO - Setting KV memory quota as 2264 MB on 172.23.108.36 2022-02-17 21:31:00,161 - root - INFO - pools/default params : memoryQuota=2264 2022-02-17 21:31:00,168 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql']) 2022-02-17 21:31:00,168 - root - INFO - /node/controller/setupServices params on 172.23.108.36: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql 2022-02-17 21:31:00,203 - root - INFO - settings/indexes params : storageMode=plasma 2022-02-17 21:31:00,213 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-17 21:31:00,213 - root - INFO - settings/web params on 172.23.108.36:8091:port=8091&username=Administrator&password=password 2022-02-17 21:31:00,279 - root - INFO - --> status:True 2022-02-17 21:31:00,279 - root - INFO - Done with init on 172.23.108.36. 2022-02-17 21:31:00,279 - root - INFO - running command.raw on 172.23.108.36: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-02-17 21:31:00,314 - root - INFO - command executed successfully with root 2022-02-17 21:31:00,314 - root - INFO - Done with cleanup on 172.23.108.36. 2022-02-17 21:31:11,605 - root - INFO - running command.raw on 172.23.108.108: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.108.108 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-17 21:31:12,143 - root - INFO - command executed successfully with root 2022-02-17 21:31:13,156 - root - INFO - Setting KV memory quota as 2520 MB on 172.23.108.108 2022-02-17 21:31:13,156 - root - INFO - pools/default params : memoryQuota=2520 2022-02-17 21:31:13,171 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-17 21:31:13,171 - root - INFO - /node/controller/setupServices params on 172.23.108.108: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-17 21:31:13,466 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-17 21:31:13,466 - root - INFO - settings/web params on 172.23.108.108:8091:port=8091&username=Administrator&password=password 2022-02-17 21:31:13,562 - root - INFO - --> status:True 2022-02-17 21:31:13,562 - root - INFO - Done with init on 172.23.108.108. 2022-02-17 21:31:13,562 - root - INFO - running command.raw on 172.23.108.108: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-02-17 21:31:13,592 - root - INFO - command executed successfully with root 2022-02-17 21:31:13,592 - root - INFO - Done with cleanup on 172.23.108.108. 2022-02-17 21:31:16,680 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-17 21:31:16,714 - root - INFO - cluster:C1 node:172.23.107.250:8091 version:7.1.0-2284-enterprise aFamily:inet services:['index', 'kv', 'n1ql'] 2022-02-17 21:31:16,714 - root - INFO - cluster:C2 node:172.23.108.36:8091 version:7.1.0-2284-enterprise aFamily:inet services:['index', 'kv', 'n1ql'] 2022-02-17 21:31:16,714 - root - INFO - cluster:C3 node:172.23.108.42:8091 version:7.1.0-2284-enterprise aFamily:inet services:['index', 'kv'] 2022-02-17 21:31:16,714 - root - INFO - cluster:C4 node:172.23.108.108:8091 version:7.1.0-2284-enterprise aFamily:inet services:['kv'] 2022-02-17 21:31:16,714 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-17 21:31:16,714 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-17 21:31:16,715 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-17 21:31:16,715 - root - INFO - INSTALL COMPLETED ON: 172.23.107.250 2022-02-17 21:31:16,715 - root - INFO - INSTALL COMPLETED ON: 172.23.108.36 2022-02-17 21:31:16,715 - root - INFO - INSTALL COMPLETED ON: 172.23.108.42 2022-02-17 21:31:16,715 - root - INFO - INSTALL COMPLETED ON: 172.23.108.108 2022-02-17 21:31:16,715 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-17 21:31:16,715 - root - INFO - TOTAL INSTALL TIME = 104 seconds /root/jenkins/workspace/centos-p0-2i-vset00-00-replica-indexes-plasma-P0/scripts/install_constants.py:50: DeprecationWarning: invalid escape sequence \ "MACOS_VERSIONS": "/Applications/Couchbase\ Server.app", /root/jenkins/workspace/centos-p0-2i-vset00-00-replica-indexes-plasma-P0/scripts/install_constants.py:119: DeprecationWarning: invalid escape sequence \C "rm -rf /Applications\Couchbase\ Server.app; " /root/jenkins/workspace/centos-p0-2i-vset00-00-replica-indexes-plasma-P0/scripts/install_constants.py:140: DeprecationWarning: invalid escape sequence \| "grep 'buildversion.*[Configuration\|Installation] completed successfully.' install_status.txt && echo 1 || echo 0", /root/jenkins/workspace/centos-p0-2i-vset00-00-replica-indexes-plasma-P0/scripts/install_constants.py:247: DeprecationWarning: invalid escape sequence \- "post_install": NON_ROOT_DOWNLOAD_DIR["LINUX_DISTROS"] + "opt/couchbase/bin/couchbase-server \-- -noinput -detached", success INFO:root:SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.107.250 as root INFO:root:SSH Connected to 172.23.108.108 as root INFO:root:SSH Connected to 172.23.108.36 as root INFO:root:SSH Connected to 172.23.108.42 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.42: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.107.250: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.36: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.108: iptables -F INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root 172.23.108.42 172.23.107.250 172.23.108.36 172.23.108.108 Need to set ALLOW_HTP back to True to do git pull branch Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client' Cloning into 'java_sdk_client'... Submodule path 'java_sdk_client': checked out 'b23994ef285cfeb976e9557a630dcf003f95da5e' Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.sfo12.us.leaseweb.net * extras: la.mirrors.clouvider.net * updates: mirror.sfo12.us.leaseweb.net Package maven-3.0.5-17.el7.noarch already installed and latest version Nothing to do find: ‘/data/workspace/*/logs/*’: No such file or directory find: ‘/data/workspace/’: No such file or directory find: ‘/root/workspace/*/logs/*’: No such file or directory find: ‘/root/workspace/’: No such file or directory /tmp/jenkins1137940625611759256.sh: line 229: killall: command not found python3 testrunner.py -i /tmp/testexec.27486.ini -c gsi/py-gsi-replica-indexes.conf -p gsi_type=plasma,reset_services=True,EXCLUDE_GROUP=MOI_ONLY,GROUP=P0,get-cbcollect-info=True -d failed=http://qa.sc.couchbase.com/job/test_suite_executor/443532/ INFO:root:__main__ INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:Conf filename: gsi/py-gsi-replica-indexes.conf INFO:__main__:Test prefix: gsi.gsi_replica_indexes.GSIReplicaIndexesTests INFO:__main__:Downloading http://qa.sc.couchbase.com/job/test_suite_executor/443532//testReport/api/xml?pretty=true to logs/test_suite_executor_443532__testresult.xml INFO:__main__:Loading result data from logs/test_suite_executor_443532__testresult.xml INFO:__main__:-- logs/test_suite_executor_443532__testresult.xml -- INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_build_index_with_replica_failover_addback,doc-per-day=1000,dataset=bigdata,nodes_init=4,services_init=kv:index:n1ql-index-index-index,groups=simple,create_primary_index=False,reset_services=True,nodes=0:1:2,node_out=2,recovery_type=full,GROUP=P0,gsi_type=plasma,EXCLUDE_GROUP=MOI_ONLY,get-cbcollect-info=False,bucket_storage=couchstore INFO:__main__:Logs folder: /root/jenkins/workspace/centos-p0-2i-vset00-00-replica-indexes-plasma-P0/logs/testrunner-22-Feb-17_21-31-21/test_1 *** TestRunner *** {'EXCLUDE_GROUP': 'MOI_ONLY', 'GROUP': 'P0', 'cluster_name': 'testexec.27486', 'conf_file': 'gsi/py-gsi-replica-indexes.conf', 'get-cbcollect-info': 'True', 'gsi_type': 'plasma', 'ini': '/tmp/testexec.27486.ini', 'num_nodes': 4, 'reset_services': 'True', 'spec': 'py-gsi-replica-indexes'} Only cases in GROUPs 'P0' will be executed Cases from GROUPs 'MOI_ONLY' will be excluded Logs will be stored at /root/jenkins/workspace/centos-p0-2i-vset00-00-replica-indexes-plasma-P0/logs/testrunner-22-Feb-17_21-31-21/test_1 ./testrunner -i /tmp/testexec.27486.ini -p gsi_type=plasma,reset_services=True,EXCLUDE_GROUP=MOI_ONLY,GROUP=P0,get-cbcollect-info=True -t gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_build_index_with_replica_failover_addback,doc-per-day=1000,dataset=bigdata,nodes_init=4,services_init=kv:index:n1ql-index-index-index,groups=simple,create_primary_index=False,reset_services=True,nodes=0:1:2,node_out=2,recovery_type=full,GROUP=P0,gsi_type=plasma,EXCLUDE_GROUP=MOI_ONLY,get-cbcollect-info=False,bucket_storage=couchstore Test Input params: {'doc-per-day': '1000', 'dataset': 'bigdata', 'nodes_init': '4', 'services_init': 'kv:index:n1ql-index-index-index', 'groups': 'simple', 'create_primary_index': 'False', 'reset_services': 'True', 'nodes': '0:1:2', 'node_out': '2', 'recovery_type': 'full', 'GROUP': 'P0', 'gsi_type': 'plasma', 'EXCLUDE_GROUP': 'MOI_ONLY', 'get-cbcollect-info': 'True', 'bucket_storage': 'couchstore', 'ini': '/tmp/testexec.27486.ini', 'cluster_name': 'testexec.27486', 'spec': 'py-gsi-replica-indexes', 'conf_file': 'gsi/py-gsi-replica-indexes.conf', 'num_nodes': 4, 'case_number': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/root/jenkins/workspace/centos-p0-2i-vset00-00-replica-indexes-plasma-P0/logs/testrunner-22-Feb-17_21-31-21/test_1'} Run before suite setup for gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_build_index_with_replica_failover_addback lib/couchbase_helper/tuq_generators.py:113: DeprecationWarning: invalid escape sequence \w regex = re.compile("[\w']+\.[\w']+") lib/couchbase_helper/tuq_generators.py:363: DeprecationWarning: invalid escape sequence \[ diff = set(order_clause.split(',')) - set(re.compile('doc\["[\w\']+"\]').findall(select_clause)) pytests/basetestcase.py:3159: DeprecationWarning: invalid escape sequence \ copy_servers, root_cn='Root\ Authority', type="openssl", lib/couchbase_helper/data_analysis_helper.py:889: DeprecationWarning: invalid escape sequence \{ master_key = ".\{12\}$" ./pytests/security/x509main.py:80: DeprecationWarning: invalid escape sequence \. ''' suite_setUp (unittest.loader._FailedTest) ... ERROR ====================================================================== ERROR: suite_setUp (unittest.loader._FailedTest) ---------------------------------------------------------------------- AttributeError: type object 'GSIReplicaIndexesTests' has no attribute 'suite_setUp' ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (errors=1) test_build_index_with_replica_failover_addback (gsi.gsi_replica_indexes.GSIReplicaIndexesTests) ... -->before_suite_name:gsi.gsi_replica_indexes.GSIReplicaIndexesTests.suite_setUp,suite: ]> -->result: 2022-02-17 21:31:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:31:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:31:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 21:31:22 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2284-enterprise 2022-02-17 21:31:22 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_build_index_with_replica_failover_addback============== 2022-02-17 21:31:22 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.107.250 port:8091 ssh_username:root] 2022-02-17 21:31:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:31:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:31:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:23 | ERROR | MainProcess | test_thread | [rest_client._http_request] POST http://172.23.107.250:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2022-02-17 21:31:23 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.250:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-02-17 21:31:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: zgrep "panic" API is accessible from localhost only/indexer.log* | wc -l 2022-02-17 21:31:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/indexer.log*.gz: No such file or directory'] ... 2022-02-17 21:31:23 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.107.250 port:8091 ssh_username:root] 2022-02-17 21:31:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:31:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:31:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:24 | ERROR | MainProcess | test_thread | [rest_client._http_request] POST http://172.23.107.250:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2022-02-17 21:31:24 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.250:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-02-17 21:31:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: zgrep "panic" API is accessible from localhost only/projector.log* | wc -l 2022-02-17 21:31:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/projector.log*.gz: No such file or directory'] ... 2022-02-17 21:31:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-02-17 21:31:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.107.250:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.516316721477359, 'mem_free': 3201228800, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-02-17 21:31:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-02-17 21:31:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:31:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:31:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 2022-02-17 21:31:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.36 as root 2022-02-17 21:31:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 2022-02-17 21:31:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.42 as root 2022-02-17 21:31:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 2022-02-17 21:31:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.108 as root 2022-02-17 21:31:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_build_index_with_replica_failover_addback ============== 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.250, nothing to delete 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.108.36, nothing to delete 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.108.42, nothing to delete 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.108.108, nothing to delete 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-02-17 21:31:39 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.107.250:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.250:8091 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.250:8091 is running 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.108.36:8091 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.108.36:8091 is running 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.108.42:8091 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.108.42:8091 is running 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.108.108:8091 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.108.108:8091 is running 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_build_index_with_replica_failover_addback ============== 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:31:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:31:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:31:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:31:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:41 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.250 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 21:31:41 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-02-17 21:31:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: systemctl stop couchbase-server.service 2022-02-17 21:31:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:31:44 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:31:44 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-02-17 21:31:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-02-17 21:31:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:31:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-02-17 21:31:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:31:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:31:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:31:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:45 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.250 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 21:31:45 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:31:45 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-02-17 21:31:45 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-02-17 21:31:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: systemctl start couchbase-server.service 2022-02-17 21:31:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:31:45 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.107.250:sleep for 5 secs. waiting for couchbase server to come up ... 2022-02-17 21:31:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-02-17 21:31:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:31:50 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 19036 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-02-17 21:31:50 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:31:50 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.250: with pid 19176 2022-02-17 21:31:50 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-02-17 21:31:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 2022-02-17 21:31:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.36 as root 2022-02-17 21:31:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 2022-02-17 21:31:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.36 as root 2022-02-17 21:31:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:52 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.108.36 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 21:31:52 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-02-17 21:31:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.36: systemctl stop couchbase-server.service 2022-02-17 21:31:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:31:56 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:31:56 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-02-17 21:31:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.36: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-02-17 21:31:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:31:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.36: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-02-17 21:31:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:31:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 2022-02-17 21:31:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.36 as root 2022-02-17 21:31:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:31:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:31:57 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.108.36 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 21:31:57 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:31:57 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-02-17 21:31:57 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-02-17 21:31:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.36: systemctl start couchbase-server.service 2022-02-17 21:31:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:31:57 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.108.36:sleep for 5 secs. waiting for couchbase server to come up ... 2022-02-17 21:32:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.36: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-02-17 21:32:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:02 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 29594 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-02-17 21:32:02 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:32:02 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.108.36: with pid 29744 2022-02-17 21:32:02 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-02-17 21:32:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 2022-02-17 21:32:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.42 as root 2022-02-17 21:32:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:32:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:32:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 2022-02-17 21:32:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.42 as root 2022-02-17 21:32:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:32:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:32:04 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.108.42 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 21:32:04 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-02-17 21:32:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.42: systemctl stop couchbase-server.service 2022-02-17 21:32:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:07 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:32:07 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-02-17 21:32:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.42: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-02-17 21:32:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.42: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-02-17 21:32:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 2022-02-17 21:32:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.42 as root 2022-02-17 21:32:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:32:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:32:08 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.108.42 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 21:32:08 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:32:08 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-02-17 21:32:09 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-02-17 21:32:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.42: systemctl start couchbase-server.service 2022-02-17 21:32:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:09 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.108.42:sleep for 5 secs. waiting for couchbase server to come up ... 2022-02-17 21:32:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.42: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-02-17 21:32:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:14 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 26269 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-02-17 21:32:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:32:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.108.42: with pid 26411 2022-02-17 21:32:14 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-02-17 21:32:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 2022-02-17 21:32:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.108 as root 2022-02-17 21:32:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:32:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:32:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 2022-02-17 21:32:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.108 as root 2022-02-17 21:32:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:32:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:32:16 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.108.108 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 21:32:16 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-02-17 21:32:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.108: systemctl stop couchbase-server.service 2022-02-17 21:32:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:18 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:32:18 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-02-17 21:32:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.108: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-02-17 21:32:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.108: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-02-17 21:32:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 2022-02-17 21:32:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.108 as root 2022-02-17 21:32:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:32:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:32:19 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.108.108 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 21:32:19 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:32:19 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-02-17 21:32:20 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-02-17 21:32:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.108: systemctl start couchbase-server.service 2022-02-17 21:32:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:20 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.108.108:sleep for 5 secs. waiting for couchbase server to come up ... 2022-02-17 21:32:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.108: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-02-17 21:32:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:25 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 85175 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-02-17 21:32:25 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 21:32:25 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.108.108: with pid 85317 2022-02-17 21:32:25 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.250 port:8091 ssh_username:root, nodes/self 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '44', 'memoryTotal': 3945775104, 'memoryFree': 3306475520, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.250:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2284-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11210, 'id': 'ns_1@172.23.107.250', 'ip': '172.23.107.250', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-02-17 21:32:34 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.107.250:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.107.250 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1760 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.107.250,8091,['kv', 'index', 'n1ql']) 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.107.250: 8091:hostname=172.23.107.250&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.250:8091:port=8091&username=Administrator&password=password 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-17 21:32:34 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.250:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.250:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.108.36 port:8091 ssh_username:root, nodes/self 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '33', 'memoryTotal': 3945775104, 'memoryFree': 3303157760, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.108.36:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2284-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11210, 'id': 'ns_1@172.23.108.36', 'ip': '172.23.108.36', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-02-17 21:32:35 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.108.36:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-17 21:32:35 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.108.36:8091:port=8091&username=Administrator&password=password 2022-02-17 21:32:36 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-17 21:32:36 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 2022-02-17 21:32:36 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.36 as root 2022-02-17 21:32:36 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.36: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.108.36:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.108.36:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.108.42 port:8091 ssh_username:root, nodes/self 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '22', 'memoryTotal': 3945775104, 'memoryFree': 3282309120, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.108.42:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2284-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11210, 'id': 'ns_1@172.23.108.42', 'ip': '172.23.108.42', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-02-17 21:32:37 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.108.42:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.108.42:8091:port=8091&username=Administrator&password=password 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.42 as root 2022-02-17 21:32:37 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.42: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.108.42:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.108.42:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.108.108 port:8091 ssh_username:root, nodes/self 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '12', 'memoryTotal': 3945775104, 'memoryFree': 3312148480, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.108.108:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2284-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11210, 'id': 'ns_1@172.23.108.108', 'ip': '172.23.108.108', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-02-17 21:32:38 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.108.108:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.108.108:8091:port=8091&username=Administrator&password=password 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.108 as root 2022-02-17 21:32:38 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:32:39 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:32:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.108: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-17 21:32:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:32:39 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-17 21:32:39 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.108.108:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 21:32:39 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.108.108:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 21:32:39 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-17 21:32:39 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.107.250 **** 2022-02-17 21:32:39 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.107.250:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-02-17 21:32:39 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-02-17 21:32:39 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-02-17 21:32:44 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-02-17 21:32:44 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2022-02-17 21:32:45 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.108.36:8091 to cluster 2022-02-17 21:32:45 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.108.36:18091 to this cluster @172.23.107.250:8091 2022-02-17 21:32:55 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-02-17 21:32:55 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-02-17 21:33:09 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.108.42:8091 to cluster 2022-02-17 21:33:09 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.108.42:18091 to this cluster @172.23.107.250:8091 2022-02-17 21:33:19 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-02-17 21:33:19 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-02-17 21:33:33 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.108.108:8091 to cluster 2022-02-17 21:33:33 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.108.108:18091 to this cluster @172.23.107.250:8091 2022-02-17 21:33:43 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2022-02-17 21:33:43 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-02-17 21:33:57 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.108.108 not part of cluster inactiveAdded 2022-02-17 21:33:57 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.108.36 not part of cluster inactiveAdded 2022-02-17 21:33:57 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.108.42 not part of cluster inactiveAdded 2022-02-17 21:33:57 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.107.250,ns_1@172.23.108.108,ns_1@172.23.108.36,ns_1@172.23.108.42', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-02-17 21:33:57 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-02-17 21:34:07 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-02-17 21:34:07 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.024034261703491 sec 2022-02-17 21:34:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:34:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:34:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:34:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:34:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-17 21:34:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:34:08 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-17 21:34:08 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.107.250 2022-02-17 21:34:09 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.107.250:8091/pools/default/buckets with param: name=default&ramQuotaMB=1760&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-02-17 21:34:09 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.04 seconds to create bucket default 2022-02-17 21:34:09 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.250 to accept set ops 2022-02-17 21:34:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.250:11210 default 2022-02-17 21:34:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.250:11210 default 2022-02-17 21:34:11 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 1760 2022-02-17 21:34:11 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_build_index_with_replica_failover_addback ============== 2022-02-17 21:34:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:34:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:34:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:34:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:34:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 2022-02-17 21:34:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.36 as root 2022-02-17 21:34:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:34:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:34:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 2022-02-17 21:34:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.42 as root 2022-02-17 21:34:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:34:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:34:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 2022-02-17 21:34:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.108 as root 2022-02-17 21:34:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:34:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:34:25 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-02-17 21:34:25 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.107.250:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.60686827268067, 'mem_free': 3092541440, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-02-17 21:34:25 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.108.108:8091 => {'services': ['index'], 'cpu_utilization': 2.576408183884819, 'mem_free': 3266875392, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-02-17 21:34:25 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.108.36:8091 => {'services': ['index'], 'cpu_utilization': 12.16489630753667, 'mem_free': 3272626176, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-02-17 21:34:25 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.108.42:8091 => {'services': ['index'], 'cpu_utilization': 13.10309801929914, 'mem_free': 3275931648, 'mem_total': 3945775104, 'swap_mem_used': 4980736, 'swap_mem_total': 3758092288} 2022-02-17 21:34:25 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-02-17 21:34:25 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 184.16901326179504 2022-02-17 21:34:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:34:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:34:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:34:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:34:26 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.107.250 port:8091 ssh_username:root] 2022-02-17 21:34:26 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.107.250 cluster is healthy 2022-02-17 21:34:26 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.107.250 is healthy 2022-02-17 21:34:27 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.108.36 cluster is healthy 2022-02-17 21:34:27 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.108.36 is healthy 2022-02-17 21:34:27 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.108.42 cluster is healthy 2022-02-17 21:34:27 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.108.42 is healthy 2022-02-17 21:34:27 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.108.108 cluster is healthy 2022-02-17 21:34:27 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.108.108 is healthy 2022-02-17 21:34:27 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.107.250 port:8091 ssh_username:root, ip:172.23.108.108 port:8091 ssh_username:root, ip:172.23.108.36 port:8091 ssh_username:root, ip:172.23.108.42 port:8091 ssh_username:root] 2022-02-17 21:34:27 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2022-02-17 21:34:27 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2022-02-17 21:34:27 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-02-17 21:34:32 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2022-02-17 21:34:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:34:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:34:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:34:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:34:33 | INFO | MainProcess | test_thread | [basetestcase.load] create 1000.0 to default documents... 2022-02-17 21:34:33 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.250:11210 default ./lib/log_scanner.py:14: DeprecationWarning: invalid escape sequence \s "Basic\s[a-zA-Z]\{10,\}==", ./lib/log_scanner.py:15: DeprecationWarning: invalid escape sequence \[ "Menelaus-Auth-User:\[", 2022-02-17 21:34:35 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2022-02-17 21:34:35 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.107.250 port:8091 ssh_username:root] 2022-02-17 21:34:35 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:172.23.107.250 port:8091 ssh_username:root 2022-02-17 21:34:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2022-02-17 21:35:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:35:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:35:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:35:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:35:06 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.107.250 port:8091 ssh_username:root, ip:172.23.108.108 port:8091 ssh_username:root, ip:172.23.108.36 port:8091 ssh_username:root, ip:172.23.108.42 port:8091 ssh_username:root] 2022-02-17 21:35:06 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=http://172.23.107.250:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False 2022-02-17 21:35:06 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2022-02-17 21:35:06 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=http://172.23.107.250:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False 2022-02-17 21:35:06 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2022-02-17 21:35:06 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.107.250 port:8091 ssh_username:root] 2022-02-17 21:35:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:35:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:35:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:35:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:35:07 | INFO | MainProcess | test_thread | [gsi_replica_indexes.test_build_index_with_replica_failover_addback] ['172.23.107.250:8091', '172.23.108.36:8091', '172.23.108.42:8091'] 2022-02-17 21:35:07 | INFO | MainProcess | test_thread | [gsi_replica_indexes.test_build_index_with_replica_failover_addback] CREATE INDEX random_index_255593 ON default(age) USING GSI WITH {'nodes': ['172.23.107.250:8091', '172.23.108.36:8091', '172.23.108.42:8091'], 'defer_build': true}; 2022-02-17 21:35:07 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX random_index_255593 ON default(age) USING GSI WITH {'nodes': ['172.23.107.250:8091', '172.23.108.36:8091', '172.23.108.42:8091'], 'defer_build': true}; 2022-02-17 21:35:07 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+random_index_255593+ON+default%28age%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27172.23.107.250%3A8091%27%2C+%27172.23.108.36%3A8091%27%2C+%27172.23.108.42%3A8091%27%5D%2C+%27defer_build%27%3A+true%7D%3B 2022-02-17 21:35:07 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 360.740474ms 2022-02-17 21:35:07 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=http://172.23.107.250:8091/indexStatus verb=GET params= client_cert=None verify=False 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [gsi_replica_indexes.test_build_index_with_replica_failover_addback] {'default': {'random_index_255593': {'status': 'Created', 'progress': '0', 'definition': 'CREATE INDEX `random_index_255593` ON `default`(`age`) WITH { "defer_build":true, "nodes":[ "172.23.107.250:8091","172.23.108.36:8091","172.23.108.42:8091" ], "num_replica":2 }', 'hosts': '172.23.108.42:8091', 'id': 1565091773902787641}, 'random_index_255593 (replica 1)': {'status': 'Created', 'progress': '0', 'definition': 'CREATE INDEX `random_index_255593` ON `default`(`age`) WITH { "defer_build":true, "nodes":[ "172.23.107.250:8091","172.23.108.36:8091","172.23.108.42:8091" ], "num_replica":2 }', 'hosts': '172.23.107.250:8091', 'id': 1565091773902787641}, 'random_index_255593 (replica 2)': {'status': 'Created', 'progress': '0', 'definition': 'CREATE INDEX `random_index_255593` ON `default`(`age`) WITH { "defer_build":true, "nodes":[ "172.23.107.250:8091","172.23.108.36:8091","172.23.108.42:8091" ], "num_replica":2 }', 'hosts': '172.23.108.36:8091', 'id': 1565091773902787641}}} 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes] Hostnames : 172.23.108.42:8091 , 172.23.107.250:8091 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes] Index IDs : 1565091773902787641, 1565091773902787641 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes] Hostnames : 172.23.108.42:8091 , 172.23.108.36:8091 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes] Index IDs : 1565091773902787641, 1565091773902787641 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY select distinct(name) from system:indexes where `using`='gsi' 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+distinct%28name%29+from+system%3Aindexes+where+%60using%60%3D%27gsi%27 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 28.245013ms 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes_build_status] index_name = random_index_255593, defer_build = True, index_status = Created 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes_build_status] index_name = random_index_255593 (replica 1), defer_build = True, index_replica_status = Created 2022-02-17 21:35:37 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes_build_status] index_name = random_index_255593 (replica 2), defer_build = True, index_replica_status = Created 2022-02-17 21:35:38 | INFO | MainProcess | Cluster_Thread | [task._failover_nodes] Failing over 172.23.108.42:8091 with graceful=False 2022-02-17 21:35:38 | INFO | MainProcess | Cluster_Thread | [rest_client.fail_over] fail_over node ns_1@172.23.108.42 successful 2022-02-17 21:35:38 | INFO | MainProcess | Cluster_Thread | [task.execute] 180 seconds sleep after failover, for nodes to go pending.... 2022-02-17 21:38:38 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on `default`(random_index_255593) 2022-02-17 21:38:38 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+%60default%60%28random_index_255593%29 2022-02-17 21:38:38 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 64.433884ms 2022-02-17 21:38:38 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2022-02-17 21:39:08 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.108.42 not part of cluster inactiveFailed 2022-02-17 21:39:08 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=full for node :: ns_1@172.23.108.42 2022-02-17 21:39:08 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.108.42 set successful 2022-02-17 21:39:08 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.108.42 successful 2022-02-17 21:39:09 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.108.42 not part of cluster inactiveAdded 2022-02-17 21:39:09 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.107.250,ns_1@172.23.108.108,ns_1@172.23.108.36,ns_1@172.23.108.42', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-02-17 21:39:09 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-02-17 21:39:11 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.006044387817383 seconds 2022-02-17 21:39:19 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-02-17 21:39:19 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.020385265350342 sec 2022-02-17 21:39:19 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 180 secs. ... 2022-02-17 21:42:19 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=http://172.23.107.250:8091/indexStatus verb=GET params= client_cert=None verify=False 2022-02-17 21:42:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY select distinct(name) from system:indexes where `using`='gsi' 2022-02-17 21:42:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+distinct%28name%29+from+system%3Aindexes+where+%60using%60%3D%27gsi%27 2022-02-17 21:42:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 28.765487ms 2022-02-17 21:42:19 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes_build_status] Expected random_index_255593 status to be Ready, but it is Created 2022-02-17 21:42:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.107.250 port:8091 ssh_username:root, ip:172.23.108.108 port:8091 ssh_username:root, ip:172.23.108.36 port:8091 ssh_username:root, ip:172.23.108.42 port:8091 ssh_username:root] 2022-02-17 21:42:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:42:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:42:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:42:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:42:20 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.250:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-02-17 21:42:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-02-17 21:42:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:42:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 2022-02-17 21:42:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.108 as root 2022-02-17 21:42:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:42:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:42:21 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.108.108:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-02-17 21:42:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.108: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-02-17 21:42:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:42:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 2022-02-17 21:42:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.42 as root 2022-02-17 21:42:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:42:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:42:22 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.108.42:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-02-17 21:42:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.108.42: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-02-17 21:42:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:42:23 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.107.250 port:8091 ssh_username:root] 2022-02-17 21:42:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:42:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:42:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:42:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:42:23 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.250:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-02-17 21:42:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.107.250 port:8091 ssh_username:root] 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 27.612488ms 2022-02-17 21:42:24 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list. List output: {'requestID': '36fdb829-3c2c-41b5-aa2d-c126fd5325be', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '27.612488ms', 'executionTime': '27.526963ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}} 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.107.250:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.81562099871959, 'mem_free': 2870317056, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.108.108:8091 => {'services': ['index'], 'cpu_utilization': 1.186569048220147, 'mem_free': 3224584192, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.108.36:8091 => {'services': ['index'], 'cpu_utilization': 1.898253606681853, 'mem_free': 3181924352, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.108.42:8091 => {'services': ['index'], 'cpu_utilization': 1.719777440566515, 'mem_free': 3212214272, 'mem_total': 3945775104, 'swap_mem_used': 4980736, 'swap_mem_total': 3758092288} 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:42:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 2022-02-17 21:42:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.36 as root 2022-02-17 21:42:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:42:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:42:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 2022-02-17 21:42:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.42 as root 2022-02-17 21:42:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:42:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:42:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 2022-02-17 21:42:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.108 as root 2022-02-17 21:42:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:42:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_build_index_with_replica_failover_addback ============== 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 172.23.107.250 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.107.250 existing buckets : [] 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.107.250 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.108.36, nothing to delete 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.108.42, nothing to delete 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.108.108, nothing to delete 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-02-17 21:42:39 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.107.250:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.107.250,ns_1@172.23.108.108,ns_1@172.23.108.36,ns_1@172.23.108.42', 'ejectedNodes': 'ns_1@172.23.108.108,ns_1@172.23.108.36,ns_1@172.23.108.42', 'user': 'Administrator', 'password': 'password'} 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2022-02-17 21:42:39 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-02-17 21:43:00 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 20.03 seconds 2022-02-17 21:43:00 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.107.250 port:8091 ssh_username:root ? [('ns_1@172.23.108.108', 8091), ('ns_1@172.23.108.36', 8091), ('ns_1@172.23.108.42', 8091)] 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.250:8091 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.250:8091 is running 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.108.36:8091 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.108.36:8091 is running 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.108.42:8091 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.108.42:8091 is running 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.108.108:8091 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.108.108:8091 is running 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_build_index_with_replica_failover_addback ============== 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-02-17 21:43:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-02-17 21:43:10 | INFO | MainProcess | Thread-127 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.42 with username:root, attempt#1 of 5 2022-02-17 21:43:10 | INFO | MainProcess | Thread-125 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.250 with username:root, attempt#1 of 5 2022-02-17 21:43:10 | INFO | MainProcess | Thread-128 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.108 with username:root, attempt#1 of 5 2022-02-17 21:43:10 | INFO | MainProcess | Thread-126 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.108.36 with username:root, attempt#1 of 5 2022-02-17 21:43:10 | INFO | MainProcess | Thread-125 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.250 as root 2022-02-17 21:43:10 | INFO | MainProcess | Thread-127 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.42 as root 2022-02-17 21:43:10 | INFO | MainProcess | Thread-128 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.108 as root 2022-02-17 21:43:10 | INFO | MainProcess | Thread-126 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.108.36 as root 2022-02-17 21:43:10 | INFO | MainProcess | Thread-125 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:43:10 | INFO | MainProcess | Thread-127 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:43:10 | INFO | MainProcess | Thread-126 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:43:10 | INFO | MainProcess | Thread-128 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 21:43:11 | INFO | MainProcess | Thread-125 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.250 2022-02-17 21:43:11 | INFO | MainProcess | Thread-125 | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: /opt/couchbase/bin/cbcollect_info 172.23.107.250-20220217-2143-diag.zip 2022-02-17 21:43:11 | INFO | MainProcess | Thread-127 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.108.42 2022-02-17 21:43:11 | INFO | MainProcess | Thread-127 | [remote_util.execute_command_raw] running command.raw on 172.23.108.42: /opt/couchbase/bin/cbcollect_info 172.23.108.42-20220217-2143-diag.zip 2022-02-17 21:43:11 | INFO | MainProcess | Thread-126 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.108.36 2022-02-17 21:43:11 | INFO | MainProcess | Thread-126 | [remote_util.execute_command_raw] running command.raw on 172.23.108.36: /opt/couchbase/bin/cbcollect_info 172.23.108.36-20220217-2143-diag.zip 2022-02-17 21:43:11 | INFO | MainProcess | Thread-128 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.108.108 2022-02-17 21:43:11 | INFO | MainProcess | Thread-128 | [remote_util.execute_command_raw] running command.raw on 172.23.108.108: /opt/couchbase/bin/cbcollect_info 172.23.108.108-20220217-2143-diag.zip 2022-02-17 21:45:08 | INFO | MainProcess | Thread-127 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.108.42 2022-02-17 21:45:08 | INFO | MainProcess | Thread-127 | [remote_util.execute_command_raw] running command.raw on 172.23.108.42: rm -f /root/172.23.108.42-20220217-2143-diag.zip 2022-02-17 21:45:08 | INFO | MainProcess | Thread-127 | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:45:09 | INFO | MainProcess | Thread-128 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.108.108 2022-02-17 21:45:09 | INFO | MainProcess | Thread-128 | [remote_util.execute_command_raw] running command.raw on 172.23.108.108: rm -f /root/172.23.108.108-20220217-2143-diag.zip 2022-02-17 21:45:09 | INFO | MainProcess | Thread-128 | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:45:10 | INFO | MainProcess | Thread-126 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.108.36 2022-02-17 21:45:10 | INFO | MainProcess | Thread-126 | [remote_util.execute_command_raw] running command.raw on 172.23.108.36: rm -f /root/172.23.108.36-20220217-2143-diag.zip 2022-02-17 21:45:11 | INFO | MainProcess | Thread-126 | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 21:46:41 | INFO | MainProcess | Thread-125 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.107.250 2022-02-17 21:46:41 | INFO | MainProcess | Thread-125 | [remote_util.execute_command_raw] running command.raw on 172.23.107.250: rm -f /root/172.23.107.250-20220217-2143-diag.zip 2022-02-17 21:46:41 | INFO | MainProcess | Thread-125 | [remote_util.execute_command_raw] command executed successfully with root ERROR ====================================================================== ERROR: test_build_index_with_replica_failover_addback (gsi.gsi_replica_indexes.GSIReplicaIndexesTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/gsi/gsi_replica_indexes.py", line 801, in test_build_index_with_replica_failover_addback defer_build=False) File "lib/couchbase_helper/tuq_helper.py", line 1228, in verify_replica_indexes_build_status raise Exception("Index status incorrect") Exception: Index status incorrect ---------------------------------------------------------------------- Ran 1 test in 708.294s FAILED (errors=1) suite_tearDown (unittest.loader._FailedTest) ... *** TestRunner *** workspace is /root/jenkins/workspace/centos-p0-2i-vset00-00-replica-indexes-plasma-P0 fails is 1 1 Desc1: 7.1.0-2284 - 2i replica - centos (0/1) python3 scripts/rerun_jobs.py 7.1.0-2284 --executor_jenkins_job --run_params=gsi_type=plasma,reset_services=True,EXCLUDE_GROUP=MOI_ONLY,GROUP=P0,get-cbcollect-info=True INFO:merge_reports:Merging of report files from ['Old_Report_mergedreport-22-Feb-12_09-41-45-gsi.gsi_replica_indexes.GSIReplicaIndexesTests.xml', 'logs/**/*.xml'] INFO:merge_reports:-- Old_Report_mergedreport-22-Feb-12_09-41-45-gsi.gsi_replica_indexes.GSIReplicaIndexesTests.xml -- INFO:merge_reports:-- logs/testrunner-22-Feb-17_21-31-21/report-22-Feb-17_21-31-21-gsi.gsi_replica_indexes.GSIReplicaIndexesTests.xml -- INFO:merge_reports: Number of TestSuites=1 INFO:merge_reports: TestSuite#1) gsi.gsi_replica_indexes.GSIReplicaIndexesTests, Number of Tests=46 INFO:merge_reports:Summary file is at /root/jenkins/workspace/centos-p0-2i-vset00-00-replica-indexes-plasma-P0/logs/testrunner-22-Feb-17_21-46-41/merged_summary/mergedreport-22-Feb-17_21-46-41-gsi.gsi_replica_indexes.GSIReplicaIndexesTests.xml Merging xmls summary so far suite gsi.gsi_replica_indexes.GSIReplicaIndexesTests , pass 45 , fail 1 failures so far... gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_build_index_with_replica_failover_addback merged xmls No more failed tests. Stopping reruns [description-setter] Description set: 7.1.0-2284 - 2i replica - centos (0/1) [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) Sending email to: mihir.kamdar@couchbase.com hemant.rajput@couchbase.com Triggering a new build of savejoblogs Triggering a new build of test-executor-cleanup Triggering a new build of post-to-gerrit Triggering a new build of test-executor-cleanup-aws Finished: UNSTABLE