Started by remote host 172.23.107.166 [EnvInject] - Loading node environment variables. Building remotely on slv-s63702-24g-12c (P0 jython_slave IPv6 bkrs3) in workspace /data/workspace/centos-p0-backup_recovery-vset00-00-1c [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Running Prebuild steps [centos-p0-backup_recovery-vset00-00-1c] $ /bin/sh -xe /tmp/jenkins1399956825929010849.sh ++ echo backup_recovery-1c-Feb-05-02:51:57-7.0.0-4347 ++ awk '{split($0,r,"-");print r[1],r[2]}' + desc='backup_recovery 1c' + echo Desc: 7.0.0-4347 - backup_recovery 1c - centos Desc: 7.0.0-4347 - backup_recovery 1c - centos + echo newState=available + newState=available Success build forhudson.tasks.Shell@203f1e07 [description-setter] Description set: 7.0.0-4347 - backup_recovery 1c - centos Success build forhudson.plugins.descriptionsetter.DescriptionSetterBuilder@2055c8ce [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Success build fororg.jenkinsci.plugins.envinject.EnvInjectBuilder@2f858cc9 Cloning the remote Git repository Using shallow clone Cloning repository git://github.com/couchbase/testrunner > /usr/bin/git init /data/workspace/centos-p0-backup_recovery-vset00-00-1c # timeout=10 Fetching upstream changes from git://github.com/couchbase/testrunner > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git config remote.origin.url git://github.com/couchbase/testrunner # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbase/testrunner # timeout=10 Fetching upstream changes from git://github.com/couchbase/testrunner > /usr/bin/git fetch --tags --progress git://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 84139279d3c723b15b7e035630fcfc17d36290fb (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 84139279d3c723b15b7e035630fcfc17d36290fb > /usr/bin/git rev-list 84139279d3c723b15b7e035630fcfc17d36290fb # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #310569 jenkins-test_suite_executor-310569 # timeout=10 No emails were triggered. [centos-p0-backup_recovery-vset00-00-1c] $ /bin/sh -xe /tmp/jenkins1112221511005870637.sh + echo Desc: backup_recovery-1c-Feb-05-02:51:57-7.0.0-4347 Desc: backup_recovery-1c-Feb-05-02:51:57-7.0.0-4347 [description-setter] Could not determine description. [centos-p0-backup_recovery-vset00-00-1c] $ /bin/sh -xe /tmp/jenkins6275514638079169642.sh [centos-p0-backup_recovery-vset00-00-1c] $ /bin/sh -xe /tmp/jenkins7954820135817084800.sh + py_executable=python2 + echo 7.0.0-4347 + grep '7\.0' 7.0.0-4347 + py_executable=python3 + [[ 7.0.0-4347 > 6.5 ]] + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From git://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + rerun_job=true + touch rerun_props_file + '[' true == false ']' [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'rerun_props_file' [EnvInject] - Variables injected successfully. [centos-p0-backup_recovery-vset00-00-1c] $ /bin/bash /tmp/jenkins7561682546838447648.sh 7.0.0-4347 Already on 'master' From git://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. Set ALLOW_HTP to False so test could run. HEAD is now at 8413927 move new upgrade with services tests from jenkins config to conf files. From http://review.couchbase.org/testrunner * branch refs/changes/72/145272/2 -> FETCH_HEAD [master 0435e94] WIP: Collect backup artifacts from 1c Author: Asad Zaidi 3 files changed, 11 insertions(+), 70 deletions(-) rewrite conf/entbackuprestore/py-entbackuprestore-3.conf (96%) the major release is 7 "172.23.100.15","172.23.100.16","172.23.100.17","172.23.100.18","172.23.98.15","172.23.108.239","172.23.105.79","172.23.121.22","172.23.106.18" Searching for httplib2 Best match: httplib2 0.17.0 Adding httplib2 0.17.0 to easy-install.pth file Using /usr/local/lib/python3.7/site-packages Processing dependencies for httplib2 Finished processing dependencies for httplib2 centos Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription- : manager This system is not registered with an entitlement server. You can use subscription-manager to register. Loading mirror speeds from cached hostfile * base: mirror.centos.lax1.serverforge.org * extras: la.mirrors.clouvider.net * updates: sjc.edge.kernel.org Package 2:docker-1.13.1-203.git0be3e21.el7.centos.x86_64 already installed and latest version Nothing to do Redirecting to /bin/systemctl start docker.service Using default tag: latest Trying to pull repository docker.io/jamesdbloom/mockserver ... latest: Pulling from docker.io/jamesdbloom/mockserver Digest: sha256:bc41010b9e920f5e1b75a226e8eda05eb4eb9843dad042e9e3950d1b7de823f4 Status: Image is up to date for docker.io/jamesdbloom/mockserver:latest [global] username:root password:couchbase port:8091 index_port:9102 n1ql_port:8093 fts_port:8094 eventing_port:8096 [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 9:_9 [cluster1] 1:_4 2:_5 3:_6 [cluster2] 1:_7 2:_8 3:_9 [_1] ip:dynamic services:kv,index,n1ql,fts,eventing [_2] ip:dynamic services:kv,index,n1ql,fts,eventing [_3] ip:dynamic [_4] ip:dynamic services:kv,index,n1ql,fts,eventing [_5] ip:dynamic [_6] ip:dynamic [_7] ip:dynamic services:kv,index,n1ql,fts,eventing [_8] ip:dynamic [_9] ip:dynamic [membase] rest_username:Administrator rest_password:password python3 scripts/populateIni.py -s "172.23.100.15","172.23.100.16","172.23.100.17","172.23.100.18","172.23.98.15","172.23.108.239","172.23.105.79","172.23.121.22","172.23.106.18" -d None -a None -i /tmp/testexec_reformat.16168.ini -p centos -o /tmp/testexec.16168.ini -k {} INFO:root:SSH Connecting to 172.23.100.15 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.98.15 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.100.16 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.106.18 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.79 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.108.239 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.98.15 as root INFO:root:SSH Connected to 172.23.100.17 as root INFO:root:SSH Connected to 172.23.100.15 as root INFO:root:SSH Connected to 172.23.100.18 as root INFO:root:SSH Connected to 172.23.100.16 as root INFO:root:SSH Connected to 172.23.108.239 as root INFO:root:SSH Connected to 172.23.106.18 as root INFO:root:SSH Connected to 172.23.121.22 as root INFO:root:SSH Connected to 172.23.105.79 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.98.15: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.239: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.18: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.18: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.16: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.17: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.15: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.121.22: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.79: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root in main the ini file is /tmp/testexec_reformat.16168.ini the given server info is "172.23.100.15","172.23.100.16","172.23.100.17","172.23.100.18","172.23.98.15","172.23.108.239","172.23.105.79","172.23.121.22","172.23.106.18" Collecting memory info from 172.23.98.15 Collecting memory info from 172.23.108.239 Collecting memory info from 172.23.100.18 Collecting memory info from 172.23.106.18 Collecting memory info from 172.23.100.16 Collecting memory info from 172.23.100.17 Collecting memory info from 172.23.100.15 Collecting memory info from 172.23.121.22 Collecting memory info from 172.23.105.79 the servers memory info is [('172.23.100.17', '4103208'), ('172.23.100.16', '4103208'), ('172.23.100.15', '4103208'), ('172.23.100.18', '4103212'), ('172.23.106.18', '4103360'), ('172.23.108.239', '4120544'), ('172.23.121.22', '4120548'), ('172.23.105.79', '4120548'), ('172.23.98.15', '4237412')] [global] username:root password:couchbase port:8091 index_port:9102 n1ql_port:8093 fts_port:8094 eventing_port:8096 [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 9:_9 [cluster1] 1:_4 2:_5 3:_6 [cluster2] 1:_7 2:_8 3:_9 [_1] ip:172.23.100.17 services:kv,index,n1ql,fts,eventing [_2] ip:172.23.100.16 services:kv,index,n1ql,fts,eventing [_3] ip:172.23.100.15 [_4] ip:172.23.100.18 services:kv,index,n1ql,fts,eventing [_5] ip:172.23.106.18 [_6] ip:172.23.108.239 [_7] ip:172.23.121.22 services:kv,index,n1ql,fts,eventing [_8] ip:172.23.105.79 [_9] ip:172.23.98.15 [membase] rest_username:Administrator rest_password:password extra install is Local time: Fri 2021-02-05 02:52:34 PST Universal time: Fri 2021-02-05 10:52:34 UTC RTC time: Fri 2021-02-05 10:52:34 Time zone: America/Los_Angeles (PST, -0800) NTP enabled: no NTP synchronized: yes RTC in local TZ: no DST active: no Last DST change: DST ended at Sun 2020-11-01 01:59:59 PDT Sun 2020-11-01 01:00:00 PST Next DST change: DST begins (the clock jumps one hour forward) at Sun 2021-03-14 01:59:59 PST Sun 2021-03-14 03:00:00 PDT python3 scripts/ssh.py -i /tmp/testexec_root.16168.ini iptables -F INFO:root:SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.100.16 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.100.15 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.106.18 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.108.239 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.79 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.98.15 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.100.16 as root INFO:root:SSH Connected to 172.23.100.18 as root INFO:root:SSH Connected to 172.23.98.15 as root INFO:root:SSH Connected to 172.23.100.17 as root INFO:root:SSH Connected to 172.23.100.15 as root INFO:root:SSH Connected to 172.23.106.18 as root INFO:root:SSH Connected to 172.23.108.239 as root INFO:root:SSH Connected to 172.23.105.79 as root INFO:root:SSH Connected to 172.23.121.22 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.98.15: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.18: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.239: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.15: iptables -F INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.16: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.18: iptables -F INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.17: iptables -F INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.121.22: iptables -F INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.79: iptables -F INFO:root:command executed successfully with root 172.23.108.239 172.23.98.15 172.23.100.15 172.23.100.18 172.23.100.16 172.23.106.18 172.23.100.17 172.23.121.22 172.23.105.79 python3 scripts/new_install.py -i /tmp/testexec.16168.ini -p timeout=1800,skip_local_download=False,get-cbcollect-info=True,version=7.0.0-4347,product=cb,debug_logs=True,url= 2021-02-05 02:52:36,016 - root - WARNING - URL: is not valid, will use version to locate build 2021-02-05 02:52:36,018 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 02:52:36,122 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 02:52:36,487 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:37,080 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:37,082 - root - INFO - SSH Connecting to 172.23.100.16 with username:root, attempt#1 of 5 2021-02-05 02:52:37,189 - root - INFO - SSH Connected to 172.23.100.16 as root 2021-02-05 02:52:37,454 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:37,746 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:37,748 - root - INFO - SSH Connecting to 172.23.100.15 with username:root, attempt#1 of 5 2021-02-05 02:52:37,853 - root - INFO - SSH Connected to 172.23.100.15 as root 2021-02-05 02:52:38,138 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:38,440 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:38,443 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 02:52:38,541 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 02:52:38,795 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:39,089 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:39,092 - root - INFO - SSH Connecting to 172.23.106.18 with username:root, attempt#1 of 5 2021-02-05 02:52:39,194 - root - INFO - SSH Connected to 172.23.106.18 as root 2021-02-05 02:52:39,453 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:39,751 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:39,754 - root - INFO - SSH Connecting to 172.23.108.239 with username:root, attempt#1 of 5 2021-02-05 02:52:39,848 - root - INFO - SSH Connected to 172.23.108.239 as root 2021-02-05 02:52:40,118 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:40,407 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:40,409 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 02:52:40,567 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 02:52:41,113 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:41,509 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:41,511 - root - INFO - SSH Connecting to 172.23.105.79 with username:root, attempt#1 of 5 2021-02-05 02:52:41,667 - root - INFO - SSH Connected to 172.23.105.79 as root 2021-02-05 02:52:42,275 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:42,664 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:42,667 - root - INFO - SSH Connecting to 172.23.98.15 with username:root, attempt#1 of 5 2021-02-05 02:52:42,757 - root - INFO - SSH Connected to 172.23.98.15 as root 2021-02-05 02:52:43,026 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:43,307 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:43,309 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 02:52:43,503 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 02:52:43,817 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:44,107 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:44,109 - root - INFO - SSH Connecting to 172.23.100.16 with username:root, attempt#1 of 5 2021-02-05 02:52:44,210 - root - INFO - SSH Connected to 172.23.100.16 as root 2021-02-05 02:52:44,472 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:44,769 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:44,771 - root - INFO - SSH Connecting to 172.23.100.15 with username:root, attempt#1 of 5 2021-02-05 02:52:44,871 - root - INFO - SSH Connected to 172.23.100.15 as root 2021-02-05 02:52:45,143 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:45,442 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:45,444 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 02:52:45,543 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 02:52:45,801 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:46,095 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:46,098 - root - INFO - SSH Connecting to 172.23.106.18 with username:root, attempt#1 of 5 2021-02-05 02:52:46,197 - root - INFO - SSH Connected to 172.23.106.18 as root 2021-02-05 02:52:46,445 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:46,739 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:46,741 - root - INFO - SSH Connecting to 172.23.108.239 with username:root, attempt#1 of 5 2021-02-05 02:52:46,837 - root - INFO - SSH Connected to 172.23.108.239 as root 2021-02-05 02:52:47,096 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:47,385 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:47,388 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 02:52:47,558 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 02:52:48,131 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:48,522 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:48,524 - root - INFO - SSH Connecting to 172.23.105.79 with username:root, attempt#1 of 5 2021-02-05 02:52:48,689 - root - INFO - SSH Connected to 172.23.105.79 as root 2021-02-05 02:52:49,320 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:49,712 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:49,714 - root - INFO - SSH Connecting to 172.23.98.15 with username:root, attempt#1 of 5 2021-02-05 02:52:49,802 - root - INFO - SSH Connected to 172.23.98.15 as root 2021-02-05 02:52:50,067 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:52:50,352 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:52:50,352 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:52:50,375 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm is live 2021-02-05 02:52:50,375 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:52:50,377 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm is live 2021-02-05 02:52:50,377 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:52:50,378 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm is live 2021-02-05 02:52:50,378 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:52:50,380 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm is live 2021-02-05 02:52:50,380 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:52:50,381 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm is live 2021-02-05 02:52:50,381 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:52:50,382 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm is live 2021-02-05 02:52:50,383 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:52:50,384 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm is live 2021-02-05 02:52:50,384 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:52:50,385 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm is live 2021-02-05 02:52:50,385 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:52:50,386 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm is live 2021-02-05 02:52:50,386 - root - INFO - Downloading build binary to /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm.. 2021-02-05 02:52:50,395 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm to 172.23.100.17 2021-02-05 02:52:50,396 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm to 172.23.100.16 2021-02-05 02:52:50,397 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm to 172.23.100.15 2021-02-05 02:52:50,397 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm to 172.23.100.18 2021-02-05 02:52:50,397 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm to 172.23.106.18 2021-02-05 02:52:50,400 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm to 172.23.108.239 2021-02-05 02:52:50,400 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm to 172.23.121.22 2021-02-05 02:52:50,403 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm to 172.23.105.79 2021-02-05 02:52:50,404 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm to 172.23.98.15 2021-02-05 02:55:33,100 - root - INFO - Done copying build to 172.23.108.239. 2021-02-05 02:55:33,316 - root - INFO - Done copying build to 172.23.121.22. 2021-02-05 02:55:33,323 - root - INFO - Done copying build to 172.23.105.79. 2021-02-05 02:55:33,335 - root - INFO - Done copying build to 172.23.98.15. 2021-02-05 02:55:33,430 - root - INFO - Done copying build to 172.23.100.16. 2021-02-05 02:55:33,469 - root - INFO - Done copying build to 172.23.100.17. 2021-02-05 02:55:33,606 - root - INFO - Done copying build to 172.23.100.15. 2021-02-05 02:55:33,769 - root - INFO - Done copying build to 172.23.106.18. 2021-02-05 02:55:33,810 - root - INFO - Done copying build to 172.23.100.18. 2021-02-05 02:55:33,811 - root - INFO - running command.raw on 172.23.100.17: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:33,928 - root - INFO - command executed successfully with root 2021-02-05 02:55:33,928 - root - INFO - running command.raw on 172.23.100.17: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-05 02:55:33,953 - root - INFO - command executed successfully with root 2021-02-05 02:55:33,953 - root - INFO - running command.raw on 172.23.100.17: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,007 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,007 - root - INFO - running command.raw on 172.23.100.16: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,025 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,025 - root - INFO - running command.raw on 172.23.100.16: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-05 02:55:34,087 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,088 - root - INFO - running command.raw on 172.23.100.16: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,141 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,141 - root - INFO - running command.raw on 172.23.100.15: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,159 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,159 - root - INFO - running command.raw on 172.23.100.15: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-05 02:55:34,221 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,222 - root - INFO - running command.raw on 172.23.100.15: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,275 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,275 - root - INFO - running command.raw on 172.23.100.18: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,292 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,292 - root - INFO - running command.raw on 172.23.100.18: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-05 02:55:34,354 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,354 - root - INFO - running command.raw on 172.23.100.18: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,451 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,452 - root - INFO - running command.raw on 172.23.106.18: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,475 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,475 - root - INFO - running command.raw on 172.23.106.18: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-05 02:55:34,542 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,543 - root - INFO - running command.raw on 172.23.106.18: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,597 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,597 - root - INFO - running command.raw on 172.23.108.239: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,616 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,616 - root - INFO - running command.raw on 172.23.108.239: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-05 02:55:34,674 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,675 - root - INFO - running command.raw on 172.23.108.239: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,727 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,727 - root - INFO - running command.raw on 172.23.121.22: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,759 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,759 - root - INFO - running command.raw on 172.23.121.22: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-05 02:55:34,834 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,835 - root - INFO - running command.raw on 172.23.121.22: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,900 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,900 - root - INFO - running command.raw on 172.23.105.79: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:34,936 - root - INFO - command executed successfully with root 2021-02-05 02:55:34,936 - root - INFO - running command.raw on 172.23.105.79: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-05 02:55:35,013 - root - INFO - command executed successfully with root 2021-02-05 02:55:35,013 - root - INFO - running command.raw on 172.23.105.79: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:35,076 - root - INFO - command executed successfully with root 2021-02-05 02:55:35,076 - root - INFO - running command.raw on 172.23.98.15: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:35,093 - root - INFO - command executed successfully with root 2021-02-05 02:55:35,094 - root - INFO - running command.raw on 172.23.98.15: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4347/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-05 02:55:35,155 - root - INFO - command executed successfully with root 2021-02-05 02:55:35,156 - root - INFO - running command.raw on 172.23.98.15: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm 2021-02-05 02:55:35,207 - root - INFO - command executed successfully with root 2021-02-05 02:55:35,208 - root - INFO - running command.raw on 172.23.100.17: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-05 02:55:35,208 - root - INFO - running command.raw on 172.23.100.16: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-05 02:55:35,209 - root - INFO - running command.raw on 172.23.100.15: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-05 02:55:35,210 - root - INFO - running command.raw on 172.23.100.18: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-05 02:55:35,212 - root - INFO - running command.raw on 172.23.106.18: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-05 02:55:35,212 - root - INFO - running command.raw on 172.23.108.239: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-05 02:55:35,212 - root - INFO - running command.raw on 172.23.121.22: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-05 02:55:35,214 - root - INFO - running command.raw on 172.23.105.79: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-05 02:55:35,215 - root - INFO - running command.raw on 172.23.98.15: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-05 02:55:38,210 - root - INFO - command executed with root but got an error ['warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave', 'warning: file /opt/couchbase/lib/python/pump_transfer.py: r ... 2021-02-05 02:55:38,603 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave', 'warning: file /opt/couchbase/lib/python/pump_transfer.py: remove ... 2021-02-05 02:55:39,105 - root - INFO - Done with uninstall on 172.23.98.15. 2021-02-05 02:55:39,105 - root - INFO - running command.raw on 172.23.98.15: yes | yum remove `rpm -qa | grep couchbase` 2021-02-05 02:55:39,616 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave', 'warning: file /opt/couchbase/lib/python/pump_transfer.py: remove ... 2021-02-05 02:55:39,627 - root - INFO - Done with uninstall on 172.23.108.239. 2021-02-05 02:55:39,627 - root - INFO - running command.raw on 172.23.108.239: yes | yum remove `rpm -qa | grep couchbase` 2021-02-05 02:55:39,818 - root - INFO - command executed with root but got an error ['warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave', 'warning: file /opt/couchbase/lib/python/pump_transfer.py: r ... 2021-02-05 02:55:39,872 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-05 02:55:39,872 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.98.15.. 2021-02-05 02:55:39,885 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave', 'warning: file /opt/couchbase/lib/python/pump_transfer.py: remove ... 2021-02-05 02:55:40,095 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave', 'warning: file /opt/couchbase/lib/python/pump_transfer.py: remove ... 2021-02-05 02:55:40,651 - root - INFO - Done with uninstall on 172.23.106.18. 2021-02-05 02:55:40,651 - root - INFO - running command.raw on 172.23.106.18: yes | yum remove `rpm -qa | grep couchbase` 2021-02-05 02:55:40,716 - root - INFO - command executed with root but got an error ['warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave', 'warning: file /opt/couchbase/lib/python/pump_transfer.py: r ... 2021-02-05 02:55:40,729 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-05 02:55:40,729 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.108.239.. 2021-02-05 02:55:40,846 - root - INFO - Done with uninstall on 172.23.100.17. 2021-02-05 02:55:40,846 - root - INFO - running command.raw on 172.23.100.17: yes | yum remove `rpm -qa | grep couchbase` 2021-02-05 02:55:40,915 - root - INFO - command executed with root but got an error ['warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave', 'warning: file /opt/couchbase/lib/python/pump_transfer.py: r ... 2021-02-05 02:55:40,938 - root - INFO - Done with uninstall on 172.23.105.79. 2021-02-05 02:55:40,938 - root - INFO - running command.raw on 172.23.105.79: yes | yum remove `rpm -qa | grep couchbase` 2021-02-05 02:55:41,007 - root - INFO - Done with uninstall on 172.23.121.22. 2021-02-05 02:55:41,007 - root - INFO - running command.raw on 172.23.121.22: yes | yum remove `rpm -qa | grep couchbase` 2021-02-05 02:55:41,715 - root - INFO - Done with uninstall on 172.23.100.18. 2021-02-05 02:55:41,716 - root - INFO - running command.raw on 172.23.100.18: yes | yum remove `rpm -qa | grep couchbase` 2021-02-05 02:55:41,909 - root - INFO - Done with uninstall on 172.23.100.16. 2021-02-05 02:55:41,910 - root - INFO - running command.raw on 172.23.100.16: yes | yum remove `rpm -qa | grep couchbase` 2021-02-05 02:55:41,986 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-05 02:55:41,986 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.106.18.. 2021-02-05 02:55:42,364 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-05 02:55:42,364 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.100.17.. 2021-02-05 02:55:43,064 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-05 02:55:43,065 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.100.18.. 2021-02-05 02:55:43,499 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-05 02:55:43,499 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.105.79.. 2021-02-05 02:55:43,542 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-05 02:55:43,543 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.121.22.. 2021-02-05 02:55:43,543 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-05 02:55:43,543 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.100.16.. 2021-02-05 02:55:46,615 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave', 'warning: file /opt/couchbase/lib/python/pump_transfer.py: remove ... 2021-02-05 02:55:47,588 - root - INFO - Done with uninstall on 172.23.100.15. 2021-02-05 02:55:47,588 - root - INFO - running command.raw on 172.23.100.15: yes | yum remove `rpm -qa | grep couchbase` 2021-02-05 02:55:48,938 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-05 02:55:48,938 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.100.15.. 2021-02-05 02:55:59,892 - root - INFO - running command.raw on 172.23.98.15: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-05 02:55:59,910 - root - INFO - command executed successfully with root 2021-02-05 02:55:59,911 - root - INFO - running command.raw on 172.23.98.15: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-05 02:56:00,749 - root - INFO - running command.raw on 172.23.108.239: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-05 02:56:00,765 - root - INFO - command executed successfully with root 2021-02-05 02:56:00,765 - root - INFO - running command.raw on 172.23.108.239: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-05 02:56:02,006 - root - INFO - running command.raw on 172.23.106.18: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-05 02:56:02,026 - root - INFO - command executed successfully with root 2021-02-05 02:56:02,026 - root - INFO - running command.raw on 172.23.106.18: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-05 02:56:02,384 - root - INFO - running command.raw on 172.23.100.17: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-05 02:56:02,401 - root - INFO - command executed successfully with root 2021-02-05 02:56:02,401 - root - INFO - running command.raw on 172.23.100.17: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-05 02:56:03,085 - root - INFO - running command.raw on 172.23.100.18: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-05 02:56:03,102 - root - INFO - command executed successfully with root 2021-02-05 02:56:03,102 - root - INFO - running command.raw on 172.23.100.18: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-05 02:56:03,519 - root - INFO - running command.raw on 172.23.105.79: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-05 02:56:03,549 - root - INFO - command executed successfully with root 2021-02-05 02:56:03,550 - root - INFO - running command.raw on 172.23.105.79: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-05 02:56:03,563 - root - INFO - running command.raw on 172.23.121.22: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-05 02:56:03,563 - root - INFO - running command.raw on 172.23.100.16: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-05 02:56:03,583 - root - INFO - command executed successfully with root 2021-02-05 02:56:03,583 - root - INFO - running command.raw on 172.23.100.16: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-05 02:56:03,594 - root - INFO - command executed successfully with root 2021-02-05 02:56:03,595 - root - INFO - running command.raw on 172.23.121.22: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-05 02:56:08,959 - root - INFO - running command.raw on 172.23.100.15: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-05 02:56:08,980 - root - INFO - command executed successfully with root 2021-02-05 02:56:08,980 - root - INFO - running command.raw on 172.23.100.15: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4347-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-05 02:56:52,334 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-05 02:56:52,335 - root - INFO - running command.raw on 172.23.98.15: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-05 02:56:52,356 - root - INFO - command executed successfully with root 2021-02-05 02:56:52,356 - root - INFO - Done with install on 172.23.98.15. 2021-02-05 02:56:52,356 - root - INFO - Waiting 60s for 172.23.98.15 to be initialized.. 2021-02-05 02:56:57,627 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-05 02:56:57,627 - root - INFO - running command.raw on 172.23.108.239: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-05 02:56:57,651 - root - INFO - command executed successfully with root 2021-02-05 02:56:57,651 - root - INFO - Done with install on 172.23.108.239. 2021-02-05 02:56:57,652 - root - INFO - Waiting 60s for 172.23.108.239 to be initialized.. 2021-02-05 02:57:04,486 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-05 02:57:04,486 - root - INFO - running command.raw on 172.23.100.17: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-05 02:57:04,507 - root - INFO - command executed successfully with root 2021-02-05 02:57:04,507 - root - INFO - Done with install on 172.23.100.17. 2021-02-05 02:57:04,508 - root - INFO - Waiting 60s for 172.23.100.17 to be initialized.. 2021-02-05 02:57:05,258 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-05 02:57:05,258 - root - INFO - running command.raw on 172.23.100.16: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-05 02:57:05,281 - root - INFO - command executed successfully with root 2021-02-05 02:57:05,281 - root - INFO - Done with install on 172.23.100.16. 2021-02-05 02:57:05,282 - root - INFO - Waiting 60s for 172.23.100.16 to be initialized.. 2021-02-05 02:57:05,721 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-05 02:57:05,722 - root - INFO - running command.raw on 172.23.100.18: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-05 02:57:05,745 - root - INFO - command executed successfully with root 2021-02-05 02:57:05,745 - root - INFO - Done with install on 172.23.100.18. 2021-02-05 02:57:05,745 - root - INFO - Waiting 60s for 172.23.100.18 to be initialized.. 2021-02-05 02:57:06,610 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-05 02:57:06,611 - root - INFO - running command.raw on 172.23.106.18: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-05 02:57:06,635 - root - INFO - command executed successfully with root 2021-02-05 02:57:06,635 - root - INFO - Done with install on 172.23.106.18. 2021-02-05 02:57:06,636 - root - INFO - Waiting 60s for 172.23.106.18 to be initialized.. 2021-02-05 02:57:12,576 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-05 02:57:12,576 - root - INFO - running command.raw on 172.23.100.15: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-05 02:57:12,600 - root - INFO - command executed successfully with root 2021-02-05 02:57:12,600 - root - INFO - Done with install on 172.23.100.15. 2021-02-05 02:57:12,600 - root - INFO - Waiting 60s for 172.23.100.15 to be initialized.. 2021-02-05 02:57:52,386 - root - INFO - running command.raw on 172.23.98.15: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.98.15 -u Administrator -p password 2021-02-05 02:57:52,596 - root - INFO - command executed successfully with root 2021-02-05 02:57:53,607 - root - INFO - Setting KV memory quota as 2217 MB on 172.23.98.15 2021-02-05 02:57:53,607 - root - INFO - pools/default params : memoryQuota=2217 2021-02-05 02:57:53,614 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-05 02:57:53,614 - root - INFO - /node/controller/setupServices params on 172.23.98.15: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-05 02:57:53,642 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:57:53,642 - root - INFO - settings/web params on 172.23.98.15:8091:port=8091&username=Administrator&password=password 2021-02-05 02:57:53,694 - root - INFO - --> status:True 2021-02-05 02:57:53,694 - root - INFO - Done with init on 172.23.98.15. 2021-02-05 02:57:53,694 - root - INFO - running command.raw on 172.23.98.15: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-05 02:57:53,715 - root - INFO - command executed successfully with root 2021-02-05 02:57:53,715 - root - INFO - Done with cleanup on 172.23.98.15. 2021-02-05 02:57:57,712 - root - INFO - running command.raw on 172.23.108.239: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.108.239 -u Administrator -p password 2021-02-05 02:57:57,967 - root - INFO - command executed successfully with root 2021-02-05 02:57:58,979 - root - INFO - Setting KV memory quota as 2156 MB on 172.23.108.239 2021-02-05 02:57:58,979 - root - INFO - pools/default params : memoryQuota=2156 2021-02-05 02:57:58,986 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-05 02:57:58,986 - root - INFO - /node/controller/setupServices params on 172.23.108.239: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-05 02:57:59,015 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:57:59,015 - root - INFO - settings/web params on 172.23.108.239:8091:port=8091&username=Administrator&password=password 2021-02-05 02:57:59,080 - root - INFO - --> status:True 2021-02-05 02:57:59,080 - root - INFO - Done with init on 172.23.108.239. 2021-02-05 02:57:59,080 - root - INFO - running command.raw on 172.23.108.239: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-05 02:57:59,096 - root - INFO - command executed successfully with root 2021-02-05 02:57:59,096 - root - INFO - Done with cleanup on 172.23.108.239. 2021-02-05 02:58:04,568 - root - INFO - running command.raw on 172.23.100.17: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.100.17 -u Administrator -p password 2021-02-05 02:58:04,812 - root - INFO - command executed successfully with root 2021-02-05 02:58:05,342 - root - INFO - running command.raw on 172.23.100.16: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.100.16 -u Administrator -p password 2021-02-05 02:58:05,589 - root - INFO - command executed successfully with root 2021-02-05 02:58:05,804 - root - INFO - running command.raw on 172.23.100.18: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.100.18 -u Administrator -p password 2021-02-05 02:58:05,824 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.100.17 2021-02-05 02:58:05,824 - root - INFO - pools/default params : indexMemoryQuota=256 2021-02-05 02:58:05,831 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.100.17 2021-02-05 02:58:05,831 - root - INFO - pools/default params : ftsMemoryQuota=512 2021-02-05 02:58:05,834 - root - INFO - Setting KV memory quota as 1379 MB on 172.23.100.17 2021-02-05 02:58:05,834 - root - INFO - pools/default params : memoryQuota=1379 2021-02-05 02:58:05,838 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts', 'eventing']) 2021-02-05 02:58:05,838 - root - INFO - /node/controller/setupServices params on 172.23.100.17: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Ceventing 2021-02-05 02:58:05,865 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:58:05,870 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:58:05,870 - root - INFO - settings/web params on 172.23.100.17:8091:port=8091&username=Administrator&password=password 2021-02-05 02:58:05,925 - root - INFO - --> status:True 2021-02-05 02:58:05,926 - root - INFO - Done with init on 172.23.100.17. 2021-02-05 02:58:05,926 - root - INFO - running command.raw on 172.23.100.17: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-05 02:58:05,955 - root - INFO - command executed successfully with root 2021-02-05 02:58:05,956 - root - INFO - Done with cleanup on 172.23.100.17. 2021-02-05 02:58:06,056 - root - INFO - command executed successfully with root 2021-02-05 02:58:06,599 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.100.16 2021-02-05 02:58:06,600 - root - INFO - pools/default params : indexMemoryQuota=256 2021-02-05 02:58:06,606 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.100.16 2021-02-05 02:58:06,606 - root - INFO - pools/default params : ftsMemoryQuota=512 2021-02-05 02:58:06,610 - root - INFO - Setting KV memory quota as 1379 MB on 172.23.100.16 2021-02-05 02:58:06,610 - root - INFO - pools/default params : memoryQuota=1379 2021-02-05 02:58:06,613 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts', 'eventing']) 2021-02-05 02:58:06,613 - root - INFO - /node/controller/setupServices params on 172.23.100.16: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Ceventing 2021-02-05 02:58:06,641 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:58:06,646 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:58:06,646 - root - INFO - settings/web params on 172.23.100.16:8091:port=8091&username=Administrator&password=password 2021-02-05 02:58:06,690 - root - INFO - running command.raw on 172.23.106.18: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.18 -u Administrator -p password 2021-02-05 02:58:06,700 - root - INFO - --> status:True 2021-02-05 02:58:06,701 - root - INFO - Done with init on 172.23.100.16. 2021-02-05 02:58:06,701 - root - INFO - running command.raw on 172.23.100.16: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-05 02:58:06,723 - root - INFO - command executed successfully with root 2021-02-05 02:58:06,723 - root - INFO - Done with cleanup on 172.23.100.16. 2021-02-05 02:58:06,948 - root - INFO - command executed successfully with root 2021-02-05 02:58:07,069 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.100.18 2021-02-05 02:58:07,069 - root - INFO - pools/default params : indexMemoryQuota=256 2021-02-05 02:58:07,076 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.100.18 2021-02-05 02:58:07,076 - root - INFO - pools/default params : ftsMemoryQuota=512 2021-02-05 02:58:07,079 - root - INFO - Setting KV memory quota as 1379 MB on 172.23.100.18 2021-02-05 02:58:07,079 - root - INFO - pools/default params : memoryQuota=1379 2021-02-05 02:58:07,084 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts', 'eventing']) 2021-02-05 02:58:07,084 - root - INFO - /node/controller/setupServices params on 172.23.100.18: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Ceventing 2021-02-05 02:58:07,112 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:58:07,118 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:58:07,118 - root - INFO - settings/web params on 172.23.100.18:8091:port=8091&username=Administrator&password=password 2021-02-05 02:58:07,180 - root - INFO - --> status:True 2021-02-05 02:58:07,180 - root - INFO - Done with init on 172.23.100.18. 2021-02-05 02:58:07,181 - root - INFO - running command.raw on 172.23.100.18: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-05 02:58:07,223 - root - INFO - command executed successfully with root 2021-02-05 02:58:07,223 - root - INFO - Done with cleanup on 172.23.100.18. 2021-02-05 02:58:07,959 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.106.18 2021-02-05 02:58:07,959 - root - INFO - pools/default params : memoryQuota=2147 2021-02-05 02:58:07,967 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-05 02:58:07,968 - root - INFO - /node/controller/setupServices params on 172.23.106.18: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-05 02:58:07,996 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:58:07,996 - root - INFO - settings/web params on 172.23.106.18:8091:port=8091&username=Administrator&password=password 2021-02-05 02:58:08,057 - root - INFO - --> status:True 2021-02-05 02:58:08,058 - root - INFO - Done with init on 172.23.106.18. 2021-02-05 02:58:08,058 - root - INFO - running command.raw on 172.23.106.18: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-05 02:58:08,081 - root - INFO - command executed successfully with root 2021-02-05 02:58:08,082 - root - INFO - Done with cleanup on 172.23.106.18. 2021-02-05 02:58:11,525 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-05 02:58:11,526 - root - INFO - running command.raw on 172.23.121.22: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-05 02:58:11,576 - root - INFO - command executed successfully with root 2021-02-05 02:58:11,577 - root - INFO - Done with install on 172.23.121.22. 2021-02-05 02:58:11,577 - root - INFO - Waiting 60s for 172.23.121.22 to be initialized.. 2021-02-05 02:58:12,664 - root - INFO - running command.raw on 172.23.100.15: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.100.15 -u Administrator -p password 2021-02-05 02:58:12,917 - root - INFO - command executed successfully with root 2021-02-05 02:58:13,948 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.100.15 2021-02-05 02:58:13,948 - root - INFO - pools/default params : memoryQuota=2147 2021-02-05 02:58:13,961 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-05 02:58:13,961 - root - INFO - /node/controller/setupServices params on 172.23.100.15: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-05 02:58:14,005 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:58:14,005 - root - INFO - settings/web params on 172.23.100.15:8091:port=8091&username=Administrator&password=password 2021-02-05 02:58:14,071 - root - INFO - --> status:True 2021-02-05 02:58:14,071 - root - INFO - Done with init on 172.23.100.15. 2021-02-05 02:58:14,071 - root - INFO - running command.raw on 172.23.100.15: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-05 02:58:14,101 - root - INFO - command executed successfully with root 2021-02-05 02:58:14,101 - root - INFO - Done with cleanup on 172.23.100.15. 2021-02-05 02:58:26,231 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-05 02:58:26,231 - root - INFO - running command.raw on 172.23.105.79: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-05 02:58:26,325 - root - INFO - command executed successfully with root 2021-02-05 02:58:26,325 - root - INFO - Done with install on 172.23.105.79. 2021-02-05 02:58:26,325 - root - INFO - Waiting 60s for 172.23.105.79 to be initialized.. 2021-02-05 02:59:11,637 - root - INFO - running command.raw on 172.23.121.22: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.22 -u Administrator -p password 2021-02-05 02:59:12,217 - root - INFO - command executed successfully with root 2021-02-05 02:59:13,236 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.121.22 2021-02-05 02:59:13,237 - root - INFO - pools/default params : indexMemoryQuota=256 2021-02-05 02:59:13,249 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.121.22 2021-02-05 02:59:13,249 - root - INFO - pools/default params : ftsMemoryQuota=512 2021-02-05 02:59:13,255 - root - INFO - Setting KV memory quota as 1388 MB on 172.23.121.22 2021-02-05 02:59:13,255 - root - INFO - pools/default params : memoryQuota=1388 2021-02-05 02:59:13,261 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts', 'eventing']) 2021-02-05 02:59:13,261 - root - INFO - /node/controller/setupServices params on 172.23.121.22: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Ceventing 2021-02-05 02:59:13,294 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:59:13,303 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:59:13,303 - root - INFO - settings/web params on 172.23.121.22:8091:port=8091&username=Administrator&password=password 2021-02-05 02:59:13,441 - root - INFO - --> status:True 2021-02-05 02:59:13,441 - root - INFO - Done with init on 172.23.121.22. 2021-02-05 02:59:13,442 - root - INFO - running command.raw on 172.23.121.22: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-05 02:59:13,501 - root - INFO - command executed successfully with root 2021-02-05 02:59:13,501 - root - INFO - Done with cleanup on 172.23.121.22. 2021-02-05 02:59:26,379 - root - INFO - running command.raw on 172.23.105.79: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.79 -u Administrator -p password 2021-02-05 02:59:26,994 - root - INFO - command executed successfully with root 2021-02-05 02:59:28,012 - root - INFO - Setting KV memory quota as 2156 MB on 172.23.105.79 2021-02-05 02:59:28,013 - root - INFO - pools/default params : memoryQuota=2156 2021-02-05 02:59:28,024 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-05 02:59:28,024 - root - INFO - /node/controller/setupServices params on 172.23.105.79: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-05 02:59:28,061 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:59:28,061 - root - INFO - settings/web params on 172.23.105.79:8091:port=8091&username=Administrator&password=password 2021-02-05 02:59:28,192 - root - INFO - --> status:True 2021-02-05 02:59:28,192 - root - INFO - Done with init on 172.23.105.79. 2021-02-05 02:59:28,192 - root - INFO - running command.raw on 172.23.105.79: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-05 02:59:28,417 - root - INFO - command executed successfully with root 2021-02-05 02:59:28,417 - root - INFO - Done with cleanup on 172.23.105.79. 2021-02-05 02:59:35,395 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-05 02:59:35,400 - root - INFO - node:172.23.100.17:8091 version:7.0.0-4347-enterprise aFamily:inet services:['eventing', 'fts', 'index', 'kv', 'n1ql'] 2021-02-05 02:59:35,405 - root - INFO - node:172.23.100.16:8091 version:7.0.0-4347-enterprise aFamily:inet services:['eventing', 'fts', 'index', 'kv', 'n1ql'] 2021-02-05 02:59:35,410 - root - INFO - node:172.23.100.15:8091 version:7.0.0-4347-enterprise aFamily:inet services:['kv'] 2021-02-05 02:59:35,414 - root - INFO - node:172.23.100.18:8091 version:7.0.0-4347-enterprise aFamily:inet services:['eventing', 'fts', 'index', 'kv', 'n1ql'] 2021-02-05 02:59:35,419 - root - INFO - node:172.23.106.18:8091 version:7.0.0-4347-enterprise aFamily:inet services:['kv'] 2021-02-05 02:59:35,424 - root - INFO - node:172.23.108.239:8091 version:7.0.0-4347-enterprise aFamily:inet services:['kv'] 2021-02-05 02:59:35,434 - root - INFO - node:172.23.121.22:8091 version:7.0.0-4347-enterprise aFamily:inet services:['eventing', 'fts', 'index', 'kv', 'n1ql'] 2021-02-05 02:59:35,444 - root - INFO - node:172.23.105.79:8091 version:7.0.0-4347-enterprise aFamily:inet services:['kv'] 2021-02-05 02:59:35,449 - root - INFO - node:172.23.98.15:8091 version:7.0.0-4347-enterprise aFamily:inet services:['kv'] 2021-02-05 02:59:35,449 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-05 02:59:35,449 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-05 02:59:35,449 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-05 02:59:35,449 - root - INFO - INSTALL COMPLETED ON: 172.23.100.17 2021-02-05 02:59:35,449 - root - INFO - INSTALL COMPLETED ON: 172.23.100.16 2021-02-05 02:59:35,449 - root - INFO - INSTALL COMPLETED ON: 172.23.100.15 2021-02-05 02:59:35,449 - root - INFO - INSTALL COMPLETED ON: 172.23.100.18 2021-02-05 02:59:35,449 - root - INFO - INSTALL COMPLETED ON: 172.23.106.18 2021-02-05 02:59:35,449 - root - INFO - INSTALL COMPLETED ON: 172.23.108.239 2021-02-05 02:59:35,449 - root - INFO - INSTALL COMPLETED ON: 172.23.121.22 2021-02-05 02:59:35,449 - root - INFO - INSTALL COMPLETED ON: 172.23.105.79 2021-02-05 02:59:35,449 - root - INFO - INSTALL COMPLETED ON: 172.23.98.15 2021-02-05 02:59:35,450 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-05 02:59:35,450 - root - INFO - TOTAL INSTALL TIME = 419 seconds success INFO:root:SSH Connecting to 172.23.100.15 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.100.16 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.108.239 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.106.18 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.98.15 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.79 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.100.15 as root INFO:root:SSH Connected to 172.23.98.15 as root INFO:root:SSH Connected to 172.23.108.239 as root INFO:root:SSH Connected to 172.23.100.18 as root INFO:root:SSH Connected to 172.23.100.17 as root INFO:root:SSH Connected to 172.23.100.16 as root INFO:root:SSH Connected to 172.23.106.18 as root INFO:root:SSH Connected to 172.23.121.22 as root INFO:root:SSH Connected to 172.23.105.79 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.18: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.16: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.108.239: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.17: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.18: iptables -F INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.98.15: iptables -F INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.100.15: iptables -F INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.121.22: iptables -F INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.79: iptables -F INFO:root:command executed successfully with root 172.23.106.18 172.23.100.16 172.23.108.239 172.23.100.17 172.23.100.18 172.23.98.15 172.23.100.15 172.23.121.22 172.23.105.79 Need to set ALLOW_HTP back to True to do git pull branch From http://review.couchbase.org/testrunner * branch refs/changes/72/145272/2 -> FETCH_HEAD The previous cherry-pick is now empty, possibly due to conflict resolution. If you wish to commit it anyway, use: git commit --allow-empty Otherwise, please use 'git reset' # On branch master # Your branch is ahead of 'origin/master' by 1 commit. # (use "git push" to publish your local commits) # # You are currently cherry-picking. # (all conflicts fixed: run "git commit") # # Untracked files: # (use "git add ..." to include in what will be committed) # # propfile # rerun_props_file # testexec.16168.ini nothing added to commit but untracked files present (use "git add" to track) Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client' Cloning into 'java_sdk_client'... Submodule path 'java_sdk_client': checked out '846cd1673c8dadde540d209bb1858ff13b39d62a' Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription- : manager This system is not registered with an entitlement server. You can use subscription-manager to register. Loading mirror speeds from cached hostfile * base: mirror.centos.lax1.serverforge.org * extras: la.mirrors.clouvider.net * updates: sjc.edge.kernel.org Package maven-3.0.5-17.el7.noarch already installed and latest version Nothing to do find: ‘/root/workspace/*/logs/*’: No such file or directory find: ‘/root/workspace/’: No such file or directory python3: no process found python3 testrunner.py -i /tmp/testexec.16168.ini -c entbackuprestore/py-entbackuprestore-3.conf -p get-coredumps=False,force_kill_memached=True,verify_unacked_bytes=True,get-cbcollect-info=True INFO:root:__main__ INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:Conf filename: entbackuprestore/py-entbackuprestore-3.conf INFO:__main__:Test prefix: ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest.test_backup_restore_with_fts,items=1000,GROUP=P0 INFO:__main__:Logs folder: /data/workspace/centos-p0-backup_recovery-vset00-00-1c/logs/testrunner-21-Feb-05_02-59-46/test_1 *** TestRunner *** {'cluster_name': 'testexec.16168', 'conf_file': 'entbackuprestore/py-entbackuprestore-3.conf', 'force_kill_memached': 'True', 'get-cbcollect-info': 'True', 'get-coredumps': 'False', 'ini': '/tmp/testexec.16168.ini', 'num_nodes': 9, 'spec': 'py-entbackuprestore-3', 'verify_unacked_bytes': 'True'} Logs will be stored at /data/workspace/centos-p0-backup_recovery-vset00-00-1c/logs/testrunner-21-Feb-05_02-59-46/test_1 ./testrunner -i /tmp/testexec.16168.ini -p get-coredumps=False,force_kill_memached=True,verify_unacked_bytes=True,get-cbcollect-info=True -t ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest.test_backup_restore_with_fts,items=1000,GROUP=P0 Test Input params: {'items': '1000', 'GROUP': 'P0', 'ini': '/tmp/testexec.16168.ini', 'cluster_name': 'testexec.16168', 'spec': 'py-entbackuprestore-3', 'conf_file': 'entbackuprestore/py-entbackuprestore-3.conf', 'get-coredumps': 'False', 'force_kill_memached': 'True', 'verify_unacked_bytes': 'True', 'get-cbcollect-info': 'True', 'num_nodes': 9, 'case_number': 1, 'logs_folder': '/data/workspace/centos-p0-backup_recovery-vset00-00-1c/logs/testrunner-21-Feb-05_02-59-46/test_1'} Run before suite setup for ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest.test_backup_restore_with_fts suite_setUp (unittest.loader._FailedTest) ... ERROR ====================================================================== ERROR: suite_setUp (unittest.loader._FailedTest) ---------------------------------------------------------------------- AttributeError: type object 'EnterpriseBackupRestoreTest' has no attribute 'suite_setUp' ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (errors=1) test_backup_restore_with_fts (ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest) ... -->before_suite_name:ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest.suite_setUp,suite: ]> -->result: 2021-02-05 02:59:47,428 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 02:59:47,528 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 02:59:47,793 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:59:48,083 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:59:48,088 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:48,097 - root - INFO - Node version in cluster 7.0.0-4347-enterprise 2021-02-05 02:59:48,103 - root - INFO - Node versions in cluster ['7.0.0-4347-enterprise'] 2021-02-05 02:59:48,103 - root - INFO - ============== basetestcase setup was started for test #1 test_backup_restore_with_fts============== 2021-02-05 02:59:48,114 - root - INFO - Could not find any buckets for node 172.23.100.17, nothing to delete 2021-02-05 02:59:48,120 - root - INFO - Could not find any buckets for node 172.23.100.16, nothing to delete 2021-02-05 02:59:48,127 - root - INFO - Could not find any buckets for node 172.23.100.15, nothing to delete 2021-02-05 02:59:48,133 - root - INFO - Could not find any buckets for node 172.23.100.18, nothing to delete 2021-02-05 02:59:48,139 - root - INFO - Could not find any buckets for node 172.23.106.18, nothing to delete 2021-02-05 02:59:48,145 - root - INFO - Could not find any buckets for node 172.23.108.239, nothing to delete 2021-02-05 02:59:48,156 - root - INFO - Could not find any buckets for node 172.23.121.22, nothing to delete 2021-02-05 02:59:48,167 - root - INFO - Could not find any buckets for node 172.23.105.79, nothing to delete 2021-02-05 02:59:48,173 - root - INFO - Could not find any buckets for node 172.23.98.15, nothing to delete 2021-02-05 02:59:48,176 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:48,190 - root - INFO - waiting for ns_server @ 172.23.100.17:8091 2021-02-05 02:59:48,190 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:48,193 - root - INFO - ns_server @ 172.23.100.17:8091 is running 2021-02-05 02:59:48,196 - root - INFO - waiting for ns_server @ 172.23.100.16:8091 2021-02-05 02:59:48,196 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:48,198 - root - INFO - ns_server @ 172.23.100.16:8091 is running 2021-02-05 02:59:48,202 - root - INFO - waiting for ns_server @ 172.23.100.15:8091 2021-02-05 02:59:48,202 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:48,204 - root - INFO - ns_server @ 172.23.100.15:8091 is running 2021-02-05 02:59:48,207 - root - INFO - waiting for ns_server @ 172.23.100.18:8091 2021-02-05 02:59:48,208 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:48,210 - root - INFO - ns_server @ 172.23.100.18:8091 is running 2021-02-05 02:59:48,214 - root - INFO - waiting for ns_server @ 172.23.106.18:8091 2021-02-05 02:59:48,214 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:48,217 - root - INFO - ns_server @ 172.23.106.18:8091 is running 2021-02-05 02:59:48,221 - root - INFO - waiting for ns_server @ 172.23.108.239:8091 2021-02-05 02:59:48,221 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:48,224 - root - INFO - ns_server @ 172.23.108.239:8091 is running 2021-02-05 02:59:48,230 - root - INFO - waiting for ns_server @ 172.23.121.22:8091 2021-02-05 02:59:48,230 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:48,235 - root - INFO - ns_server @ 172.23.121.22:8091 is running 2021-02-05 02:59:48,241 - root - INFO - waiting for ns_server @ 172.23.105.79:8091 2021-02-05 02:59:48,241 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:48,246 - root - INFO - ns_server @ 172.23.105.79:8091 is running 2021-02-05 02:59:48,249 - root - INFO - waiting for ns_server @ 172.23.98.15:8091 2021-02-05 02:59:48,249 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:48,251 - root - INFO - ns_server @ 172.23.98.15:8091 is running 2021-02-05 02:59:48,252 - root - INFO - ============== NewUpgradeBaseTest tearDown has started ============== 2021-02-05 02:59:48,253 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 02:59:48,415 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 02:59:48,974 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:59:49,366 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:59:49,367 - root - INFO - running command.raw on 172.23.121.22: rm -rf /tmp/entbackup_172.23.100.17 2021-02-05 02:59:49,391 - root - INFO - command executed successfully with root 2021-02-05 02:59:49,391 - root - INFO - running command.raw on 172.23.121.22: rm -rf /cbqe3043/entbackup 2021-02-05 02:59:49,454 - root - INFO - command executed successfully with root 2021-02-05 02:59:49,461 - root - INFO - Could not find any buckets for node 172.23.100.18, nothing to delete 2021-02-05 02:59:49,468 - root - INFO - Could not find any buckets for node 172.23.106.18, nothing to delete 2021-02-05 02:59:49,474 - root - INFO - Could not find any buckets for node 172.23.108.239, nothing to delete 2021-02-05 02:59:49,477 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:49,492 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:49,498 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:49,504 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:49,517 - root - INFO - Could not find any buckets for node 172.23.121.22, nothing to delete 2021-02-05 02:59:49,527 - root - INFO - Could not find any buckets for node 172.23.105.79, nothing to delete 2021-02-05 02:59:49,532 - root - INFO - Could not find any buckets for node 172.23.98.15, nothing to delete 2021-02-05 02:59:49,537 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:49,565 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:49,575 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:49,582 - root - INFO - -->is_ns_server_running? 2021-02-05 02:59:49,585 - root - INFO - initializing cluster 2021-02-05 02:59:50,589 - root - INFO - server: ip:172.23.100.17 port:8091 ssh_username:root, nodes/self 2021-02-05 02:59:50,592 - root - INFO - {'uptime': '160', 'memoryTotal': 4201684992, 'memoryFree': 3498614784, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.100.17:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4347-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1379, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.100.17', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['eventing', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-02-05 02:59:50,595 - root - INFO - quota for index service will be 256 MB 2021-02-05 02:59:50,596 - root - INFO - set index quota to node 172.23.100.17 2021-02-05 02:59:50,596 - root - INFO - pools/default params : indexMemoryQuota=256 2021-02-05 02:59:50,599 - root - INFO - quota for fts service will be 512 MB 2021-02-05 02:59:50,599 - root - INFO - set both index and fts quota at node 172.23.100.17 2021-02-05 02:59:50,600 - root - INFO - pools/default params : ftsMemoryQuota=512 2021-02-05 02:59:50,604 - root - INFO - pools/default params : memoryQuota=1379 2021-02-05 02:59:50,607 - root - INFO - --> init_node_services(Administrator,password,172.23.100.17,8091,['kv', 'index', 'n1ql', 'fts', 'eventing']) 2021-02-05 02:59:50,607 - root - INFO - /node/controller/setupServices params on 172.23.100.17: 8091:hostname=172.23.100.17&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Ceventing 2021-02-05 02:59:50,609 - root - ERROR - POST http://172.23.100.17:8091//node/controller/setupServices body: hostname=172.23.100.17&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Ceventing headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password 2021-02-05 02:59:50,612 - root - INFO - This node is already provisioned with services, we do not consider this as failure for test case 2021-02-05 02:59:50,612 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:59:50,612 - root - INFO - settings/web params on 172.23.100.17:8091:port=8091&username=Administrator&password=password 2021-02-05 02:59:50,663 - root - INFO - --> status:True 2021-02-05 02:59:50,664 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 02:59:50,764 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 02:59:51,024 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:59:51,318 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:59:51,325 - root - INFO - running command.raw on 172.23.100.17: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 02:59:51,380 - root - INFO - command executed successfully with root 2021-02-05 02:59:51,380 - root - INFO - ['ok'] 2021-02-05 02:59:51,382 - root - INFO - /diag/eval status on 172.23.100.17:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:51,384 - root - INFO - /diag/eval status on 172.23.100.17:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:51,389 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:59:51,404 - root - INFO - server: ip:172.23.100.16 port:8091 ssh_username:root, nodes/self 2021-02-05 02:59:51,407 - root - INFO - {'uptime': '160', 'memoryTotal': 4201684992, 'memoryFree': 3571769344, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.100.16:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4347-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1379, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.100.16', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['eventing', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-02-05 02:59:51,411 - root - INFO - pools/default params : memoryQuota=2147 2021-02-05 02:59:51,414 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:59:51,415 - root - INFO - settings/web params on 172.23.100.16:8091:port=8091&username=Administrator&password=password 2021-02-05 02:59:51,463 - root - INFO - --> status:True 2021-02-05 02:59:51,464 - root - INFO - SSH Connecting to 172.23.100.16 with username:root, attempt#1 of 5 2021-02-05 02:59:51,566 - root - INFO - SSH Connected to 172.23.100.16 as root 2021-02-05 02:59:51,814 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:59:52,105 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:59:52,113 - root - INFO - running command.raw on 172.23.100.16: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 02:59:52,168 - root - INFO - command executed successfully with root 2021-02-05 02:59:52,168 - root - INFO - ['ok'] 2021-02-05 02:59:52,171 - root - INFO - /diag/eval status on 172.23.100.16:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:52,173 - root - INFO - /diag/eval status on 172.23.100.16:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:52,177 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:59:52,190 - root - INFO - server: ip:172.23.100.15 port:8091 ssh_username:root, nodes/self 2021-02-05 02:59:52,193 - root - INFO - {'uptime': '155', 'memoryTotal': 4201684992, 'memoryFree': 3530641408, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.100.15:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4347-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.100.15', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-02-05 02:59:52,196 - root - INFO - pools/default params : memoryQuota=2147 2021-02-05 02:59:52,200 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:59:52,200 - root - INFO - settings/web params on 172.23.100.15:8091:port=8091&username=Administrator&password=password 2021-02-05 02:59:52,249 - root - INFO - --> status:True 2021-02-05 02:59:52,251 - root - INFO - SSH Connecting to 172.23.100.15 with username:root, attempt#1 of 5 2021-02-05 02:59:52,351 - root - INFO - SSH Connected to 172.23.100.15 as root 2021-02-05 02:59:52,625 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:59:52,920 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:59:52,928 - root - INFO - running command.raw on 172.23.100.15: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 02:59:52,986 - root - INFO - command executed successfully with root 2021-02-05 02:59:52,987 - root - INFO - ['ok'] 2021-02-05 02:59:52,989 - root - INFO - /diag/eval status on 172.23.100.15:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:52,990 - root - INFO - /diag/eval status on 172.23.100.15:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:52,995 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:59:53,013 - root - INFO - server: ip:172.23.100.18 port:8091 ssh_username:root, nodes/self 2021-02-05 02:59:53,016 - root - INFO - {'uptime': '165', 'memoryTotal': 4201689088, 'memoryFree': 3514826752, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.100.18:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4347-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1379, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.100.18', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['eventing', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-02-05 02:59:53,020 - root - INFO - pools/default params : memoryQuota=2147 2021-02-05 02:59:53,023 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:59:53,023 - root - INFO - settings/web params on 172.23.100.18:8091:port=8091&username=Administrator&password=password 2021-02-05 02:59:53,073 - root - INFO - --> status:True 2021-02-05 02:59:53,075 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 02:59:53,178 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 02:59:53,438 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:59:53,730 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:59:53,739 - root - INFO - running command.raw on 172.23.100.18: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 02:59:53,793 - root - INFO - command executed successfully with root 2021-02-05 02:59:53,794 - root - INFO - ['ok'] 2021-02-05 02:59:53,796 - root - INFO - /diag/eval status on 172.23.100.18:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:53,798 - root - INFO - /diag/eval status on 172.23.100.18:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:53,802 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:59:53,816 - root - INFO - server: ip:172.23.106.18 port:8091 ssh_username:root, nodes/self 2021-02-05 02:59:53,819 - root - INFO - {'uptime': '165', 'memoryTotal': 4201840640, 'memoryFree': 3576193024, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.106.18:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4347-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.106.18', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-02-05 02:59:53,822 - root - INFO - pools/default params : memoryQuota=2147 2021-02-05 02:59:53,826 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:59:53,826 - root - INFO - settings/web params on 172.23.106.18:8091:port=8091&username=Administrator&password=password 2021-02-05 02:59:53,880 - root - INFO - --> status:True 2021-02-05 02:59:53,882 - root - INFO - SSH Connecting to 172.23.106.18 with username:root, attempt#1 of 5 2021-02-05 02:59:53,986 - root - INFO - SSH Connected to 172.23.106.18 as root 2021-02-05 02:59:54,240 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:59:54,543 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:59:54,551 - root - INFO - running command.raw on 172.23.106.18: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 02:59:54,608 - root - INFO - command executed successfully with root 2021-02-05 02:59:54,608 - root - INFO - ['ok'] 2021-02-05 02:59:54,611 - root - INFO - /diag/eval status on 172.23.106.18:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:54,614 - root - INFO - /diag/eval status on 172.23.106.18:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:54,619 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:59:54,637 - root - INFO - server: ip:172.23.108.239 port:8091 ssh_username:root, nodes/self 2021-02-05 02:59:54,641 - root - INFO - {'uptime': '170', 'memoryTotal': 4219437056, 'memoryFree': 3679961088, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.108.239:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4347-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2156, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.108.239', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-05 02:59:54,645 - root - INFO - pools/default params : memoryQuota=2156 2021-02-05 02:59:54,649 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:59:54,649 - root - INFO - settings/web params on 172.23.108.239:8091:port=8091&username=Administrator&password=password 2021-02-05 02:59:54,709 - root - INFO - --> status:True 2021-02-05 02:59:54,710 - root - INFO - SSH Connecting to 172.23.108.239 with username:root, attempt#1 of 5 2021-02-05 02:59:54,806 - root - INFO - SSH Connected to 172.23.108.239 as root 2021-02-05 02:59:55,079 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:59:55,368 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:59:55,377 - root - INFO - running command.raw on 172.23.108.239: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 02:59:55,430 - root - INFO - command executed successfully with root 2021-02-05 02:59:55,430 - root - INFO - ['ok'] 2021-02-05 02:59:55,433 - root - INFO - /diag/eval status on 172.23.108.239:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:55,436 - root - INFO - /diag/eval status on 172.23.108.239:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:55,440 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:59:55,459 - root - INFO - server: ip:172.23.121.22 port:8091 ssh_username:root, nodes/self 2021-02-05 02:59:55,464 - root - INFO - {'uptime': '101', 'memoryTotal': 4219441152, 'memoryFree': 3592560640, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.121.22:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4347-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1388, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.121.22', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['eventing', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-05 02:59:55,471 - root - INFO - pools/default params : memoryQuota=2156 2021-02-05 02:59:55,478 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:59:55,478 - root - INFO - settings/web params on 172.23.121.22:8091:port=8091&username=Administrator&password=password 2021-02-05 02:59:55,607 - root - INFO - --> status:True 2021-02-05 02:59:55,609 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 02:59:55,773 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 02:59:56,322 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:59:56,703 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:59:56,716 - root - INFO - running command.raw on 172.23.121.22: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 02:59:56,784 - root - INFO - command executed successfully with root 2021-02-05 02:59:56,784 - root - INFO - ['ok'] 2021-02-05 02:59:56,788 - root - INFO - /diag/eval status on 172.23.121.22:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:56,791 - root - INFO - /diag/eval status on 172.23.121.22:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:56,798 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:59:56,826 - root - INFO - server: ip:172.23.105.79 port:8091 ssh_username:root, nodes/self 2021-02-05 02:59:56,832 - root - INFO - {'uptime': '86', 'memoryTotal': 4219441152, 'memoryFree': 3668738048, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.105.79:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4347-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2156, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.79', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-05 02:59:56,838 - root - INFO - pools/default params : memoryQuota=2156 2021-02-05 02:59:56,844 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:59:56,844 - root - INFO - settings/web params on 172.23.105.79:8091:port=8091&username=Administrator&password=password 2021-02-05 02:59:56,964 - root - INFO - --> status:True 2021-02-05 02:59:56,966 - root - INFO - SSH Connecting to 172.23.105.79 with username:root, attempt#1 of 5 2021-02-05 02:59:57,124 - root - INFO - SSH Connected to 172.23.105.79 as root 2021-02-05 02:59:57,745 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:59:58,133 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:59:58,147 - root - INFO - running command.raw on 172.23.105.79: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 02:59:58,214 - root - INFO - command executed successfully with root 2021-02-05 02:59:58,215 - root - INFO - ['ok'] 2021-02-05 02:59:58,218 - root - INFO - /diag/eval status on 172.23.105.79:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:58,221 - root - INFO - /diag/eval status on 172.23.105.79:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:58,228 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:59:58,257 - root - INFO - server: ip:172.23.98.15 port:8091 ssh_username:root, nodes/self 2021-02-05 02:59:58,260 - root - INFO - {'uptime': '179', 'memoryTotal': 4339109888, 'memoryFree': 3715805184, 'mcdMemoryReserved': 3310, 'mcdMemoryAllocated': 3310, 'status': 'healthy', 'hostname': '172.23.98.15:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4347-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2217, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.98.15', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4138, 'curr_items': 0} 2021-02-05 02:59:58,263 - root - INFO - pools/default params : memoryQuota=2217 2021-02-05 02:59:58,266 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 02:59:58,266 - root - INFO - settings/web params on 172.23.98.15:8091:port=8091&username=Administrator&password=password 2021-02-05 02:59:58,311 - root - INFO - --> status:True 2021-02-05 02:59:58,312 - root - INFO - SSH Connecting to 172.23.98.15 with username:root, attempt#1 of 5 2021-02-05 02:59:58,401 - root - INFO - SSH Connected to 172.23.98.15 as root 2021-02-05 02:59:58,677 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 02:59:58,957 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 02:59:58,965 - root - INFO - running command.raw on 172.23.98.15: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 02:59:59,017 - root - INFO - command executed successfully with root 2021-02-05 02:59:59,017 - root - INFO - ['ok'] 2021-02-05 02:59:59,020 - root - INFO - /diag/eval status on 172.23.98.15:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:59,022 - root - INFO - /diag/eval status on 172.23.98.15:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-05 02:59:59,028 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 02:59:59,046 - root - INFO - **** add built-in 'cbadminbucket' user to node 172.23.100.17 **** 2021-02-05 02:59:59,056 - root - ERROR - DELETE http://172.23.100.17:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-02-05 02:59:59,060 - root - INFO - Exception while deleting user. Exception is -b'"User was not found."' 2021-02-05 02:59:59,123 - root - INFO - sleep for 5 secs. ... 2021-02-05 03:00:04,126 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2021-02-05 03:00:04,138 - root - INFO - done initializing cluster 2021-02-05 03:00:04,139 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 03:00:04,243 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 03:00:04,511 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:04,803 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:04,809 - root - INFO - running command.raw on 172.23.100.17: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 03:00:04,866 - root - INFO - command executed successfully with root 2021-02-05 03:00:04,866 - root - INFO - ['ok'] 2021-02-05 03:00:04,867 - root - INFO - Enabled diag/eval for non-local hosts from 172.23.100.17 2021-02-05 03:00:05,060 - root - INFO - http://172.23.100.17:8091/pools/default/buckets with param: name=default&authType=sasl&saslPassword=None&ramQuotaMB=1379&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-02-05 03:00:05,085 - root - INFO - 0.03 seconds to create bucket default 2021-02-05 03:00:05,085 - root - INFO - waiting for memcached bucket : default in 172.23.100.17 to accept set ops 2021-02-05 03:00:05,762 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:05,890 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:05,951 - root - WARNING - vbucket map not ready after try 0 2021-02-05 03:00:05,951 - root - INFO - waiting for memcached bucket : default in 172.23.100.17 to accept set ops 2021-02-05 03:00:06,093 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:06,226 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:06,339 - root - WARNING - vbucket map not ready after try 1 2021-02-05 03:00:06,339 - root - INFO - waiting for memcached bucket : default in 172.23.100.17 to accept set ops 2021-02-05 03:00:06,467 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:06,587 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:06,656 - root - WARNING - vbucket map not ready after try 2 2021-02-05 03:00:06,656 - root - INFO - waiting for memcached bucket : default in 172.23.100.17 to accept set ops 2021-02-05 03:00:06,795 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:06,912 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:06,972 - root - WARNING - vbucket map not ready after try 3 2021-02-05 03:00:06,973 - root - INFO - waiting for memcached bucket : default in 172.23.100.17 to accept set ops 2021-02-05 03:00:07,087 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:07,187 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:07,263 - root - WARNING - vbucket map not ready after try 4 2021-02-05 03:00:07,263 - root - INFO - waiting for memcached bucket : default in 172.23.100.17 to accept set ops 2021-02-05 03:00:07,365 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:07,486 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:07,548 - root - WARNING - vbucket map not ready after try 5 2021-02-05 03:00:07,550 - root - INFO - ============== basetestcase setup was finished for test #1 test_backup_restore_with_fts ============== 2021-02-05 03:00:07,561 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 03:00:07,662 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 03:00:07,923 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:08,217 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:08,220 - root - INFO - SSH Connecting to 172.23.100.16 with username:root, attempt#1 of 5 2021-02-05 03:00:08,318 - root - INFO - SSH Connected to 172.23.100.16 as root 2021-02-05 03:00:08,579 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:08,871 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:08,874 - root - INFO - SSH Connecting to 172.23.100.15 with username:root, attempt#1 of 5 2021-02-05 03:00:08,978 - root - INFO - SSH Connected to 172.23.100.15 as root 2021-02-05 03:00:09,259 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:09,591 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:09,594 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 03:00:09,700 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 03:00:09,964 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:10,259 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:10,262 - root - INFO - SSH Connecting to 172.23.106.18 with username:root, attempt#1 of 5 2021-02-05 03:00:10,365 - root - INFO - SSH Connected to 172.23.106.18 as root 2021-02-05 03:00:10,626 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:10,933 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:10,935 - root - INFO - SSH Connecting to 172.23.108.239 with username:root, attempt#1 of 5 2021-02-05 03:00:11,031 - root - INFO - SSH Connected to 172.23.108.239 as root 2021-02-05 03:00:11,295 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:11,590 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:11,592 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:00:11,752 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:00:12,322 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:12,721 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:12,724 - root - INFO - SSH Connecting to 172.23.105.79 with username:root, attempt#1 of 5 2021-02-05 03:00:12,904 - root - INFO - SSH Connected to 172.23.105.79 as root 2021-02-05 03:00:13,539 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:13,940 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:13,942 - root - INFO - SSH Connecting to 172.23.98.15 with username:root, attempt#1 of 5 2021-02-05 03:00:14,034 - root - INFO - SSH Connected to 172.23.98.15 as root 2021-02-05 03:00:14,313 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:14,594 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:22,950 - root - INFO - ------- Cluster statistics ------- 2021-02-05 03:00:22,950 - root - INFO - 172.23.100.17:8091 => {'services': ['eventing', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 16.35124905374716, 'mem_free': 3379879936, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-05 03:00:22,950 - root - INFO - --- End of cluster statistics --- 2021-02-05 03:00:22,950 - root - INFO - ============== NewUpgradeBaseTest setup has started ============== 2021-02-05 03:00:22,952 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 03:00:23,051 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 03:00:23,318 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:23,612 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:23,613 - root - INFO - ============== NewUpgradeBaseTest setup has completed ============== % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 52 100 20 100 32 12944 20711 --:--:-- --:--:-- --:--:-- 32000 2021-02-05 03:00:23,629 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 03:00:23,725 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 03:00:23,989 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:24,284 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:24,297 - root - INFO - Node version in cluster 7.0.0-4347-enterprise 2021-02-05 03:00:24,299 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:00:24,456 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:00:24,995 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:25,378 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:25,378 - root - INFO - running command.raw on 172.23.121.22: nproc 2021-02-05 03:00:25,404 - root - INFO - command executed successfully with root 2021-02-05 03:00:25,420 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:00:25,575 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:00:26,150 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:26,546 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:26,645 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 03:00:26,744 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 03:00:27,005 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:27,301 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:27,386 - root - INFO - adding remote node @172.23.100.16:8091 to this cluster @172.23.100.17:8091 2021-02-05 03:00:31,963 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.16,ns_1@172.23.100.17', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-02-05 03:00:32,194 - root - INFO - rebalance operation started 2021-02-05 03:00:32,198 - root - INFO - rebalance percentage : 0.00 % 2021-02-05 03:00:32,199 - root - INFO - Rebalance - status: running, progress: 0.00% 2021-02-05 03:00:42,216 - root - INFO - rebalance percentage : 24.00 % 2021-02-05 03:00:42,216 - root - INFO - Rebalance - status: running, progress: 24.00% 2021-02-05 03:00:52,235 - root - INFO - Rebalance - status: none, progress: 100.00% 2021-02-05 03:00:52,241 - root - INFO - rebalancing was completed with progress: 100% in 20.04659366607666 sec 2021-02-05 03:00:52,305 - root - INFO - creating direct client 172.23.100.16:11210 default 2021-02-05 03:00:52,353 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:53,348 - root - INFO - creating direct client 172.23.100.16:11210 default 2021-02-05 03:00:53,354 - root - INFO - creating direct client 172.23.100.16:11210 default 2021-02-05 03:00:53,358 - root - INFO - creating direct client 172.23.100.16:11210 default 2021-02-05 03:00:53,361 - root - INFO - creating direct client 172.23.100.16:11210 default 2021-02-05 03:00:53,411 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:53,413 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:53,413 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:53,420 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:00:53,642 - root - INFO - Node version in cluster 7.0.0-4347-enterprise 2021-02-05 03:00:53,644 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:00:53,805 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:00:54,377 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:00:54,775 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:00:54,776 - root - INFO - running command.raw on 172.23.121.22: rm -rf /tmp/entbackup_172.23.100.17 2021-02-05 03:00:54,806 - root - INFO - command executed successfully with root 2021-02-05 03:00:54,806 - root - INFO - running command.raw on 172.23.121.22: /opt/couchbase/bin//cbbackupmgr config --archive /tmp/entbackup_172.23.100.17 --repo backup 2021-02-05 03:00:54,926 - root - INFO - command executed successfully with root 2021-02-05 03:00:54,926 - root - INFO - Backup repository `backup` created successfully in archive `/tmp/entbackup_172.23.100.17` 2021-02-05 03:00:54,986 - root - INFO - Checking if index already exists ... 2021-02-05 03:00:54,989 - root - ERROR - GET http://172.23.100.17:8094/api/index/index_default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: rest_auth: preparePerms, err: index not found b'{"error":"rest_auth: preparePerms, err: index not found","request":"","status":"fail"}\n' auth: Administrator:password 2021-02-05 03:00:54,995 - root - ERROR - DELETE http://172.23.100.17:8094/api/index/index_default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: rest_auth: preparePerms, err: index not found b'{"error":"rest_auth: preparePerms, err: index not found","request":"","status":"fail"}\n' auth: Administrator:password 2021-02-05 03:00:54,996 - root - INFO - Creating fulltext-index index_default on 172.23.100.17 2021-02-05 03:00:54,996 - root - INFO - {"type": "fulltext-index", "name": "index_default", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {}, "sourceParams": {}} 2021-02-05 03:00:55,043 - root - INFO - Index index_default created 2021-02-05 03:00:55,083 - root - ERROR - GET http://172.23.100.17:8094/api/index/index_default/count body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: rest_index: Count, indexName: index_default, err: bleve: CountBleve indexAlias error, indexName: index_default, indexUUID: , err: bleve: bleveIndexTargets, err: pindex: no planPIndexes for indexName: index_default b'{"error":"rest_index: Count, indexName: index_default, err: bleve: CountBleve indexAlias error, indexName: index_default, indexUUID: , err: bleve: bleveIndexTargets, err: pindex: no planPIndexes for indexName: index_default","request":"","status":"fail"}\n' auth: Administrator:password 2021-02-05 03:00:55,084 - root - INFO - 'count' 2021-02-05 03:01:01,117 - root - INFO - http://172.23.100.17:8091/pools/default/buckets/default/stats?zoom=minute 2021-02-05 03:01:01,571 - root - INFO - Docs in bucket = 0, docs in FTS index 'index_default': 1000 2021-02-05 03:01:07,600 - root - INFO - http://172.23.100.17:8091/pools/default/buckets/default/stats?zoom=minute 2021-02-05 03:01:08,089 - root - INFO - Docs in bucket = 1000, docs in FTS index 'index_default': 1000 2021-02-05 03:01:08,103 - root - INFO - Checking if index already exists ... 2021-02-05 03:01:08,113 - root - ERROR - GET http://172.23.100.17:8094/api/index/alias_1612522868 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: rest_auth: preparePerms, err: index not found b'{"error":"rest_auth: preparePerms, err: index not found","request":"","status":"fail"}\n' auth: Administrator:password 2021-02-05 03:01:08,115 - root - ERROR - DELETE http://172.23.100.17:8094/api/index/alias_1612522868 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: rest_auth: preparePerms, err: index not found b'{"error":"rest_auth: preparePerms, err: index not found","request":"","status":"fail"}\n' auth: Administrator:password 2021-02-05 03:01:08,115 - root - INFO - Creating fulltext-alias alias_1612522868 on 172.23.100.17 2021-02-05 03:01:08,115 - root - INFO - {"type": "fulltext-alias", "name": "alias_1612522868", "uuid": "", "params": {"targets": {"index_default": {}}, "store": {"kvStoreName": "mossStore", "mossStoreOptions": {}}}, "sourceType": "nil", "sourceName": "", "sourceUUID": "", "planParams": {}, "sourceParams": {}} 2021-02-05 03:01:08,127 - root - INFO - Index alias_1612522868 created 2021-02-05 03:01:08,135 - root - INFO - -->is_ns_server_running? 2021-02-05 03:01:08,149 - root - INFO - Node version in cluster 7.0.0-4347-enterprise 2021-02-05 03:01:08,151 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:01:08,317 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:01:08,874 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:09,264 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:09,265 - root - INFO - running command.raw on 172.23.121.22: /opt/couchbase/bin//cbbackupmgr backup --archive /tmp/entbackup_172.23.100.17 --repo backup --cluster http://172.23.100.17:8091 --username Administrator --password password --no-progress-bar 2021-02-05 03:01:13,638 - root - INFO - command executed successfully with root 2021-02-05 03:01:13,638 - root - INFO - Found 'Backup completed successfully in CLI output 2021-02-05 03:01:13,639 - root - INFO - Found 'Backed up bucket "default" succeeded in CLI output 2021-02-05 03:01:13,639 - root - INFO - running command.raw on 172.23.121.22: ls -tr /tmp/entbackup_172.23.100.17/backup/ | grep -P '^\d{4}-\d{2}-\d{2}' | tail -1 2021-02-05 03:01:13,668 - root - INFO - command executed successfully with root 2021-02-05 03:01:13,668 - root - INFO - Finished taking backup with args: backup --archive /tmp/entbackup_172.23.100.17 --repo backup --cluster http://172.23.100.17:8091 --username Administrator --password password --no-progress-bar 2021-02-05 03:01:13,669 - root - INFO - Found 'Backup completed successfully in CLI output 2021-02-05 03:01:13,669 - root - INFO - Found 'Backed up bucket "default" succeeded in CLI output 2021-02-05 03:01:13,680 - root - INFO - Node version in cluster 7.0.0-4347-enterprise 2021-02-05 03:01:13,682 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:01:13,838 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:01:14,400 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:14,789 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:14,790 - root - INFO - running command.raw on 172.23.121.22: /opt/couchbase/bin//cbbackupmgr info --archive /tmp/entbackup_172.23.100.17 --all --json 2021-02-05 03:01:14,882 - root - INFO - command executed successfully with root 2021-02-05 03:01:14,885 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:01:15,043 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:01:15,597 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:15,980 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:15,981 - root - INFO - running command.raw on 172.23.121.22: ls -lR /tmp/entbackup_172.23.100.17/backup 2021-02-05 03:01:16,131 - root - INFO - command executed successfully with root 2021-02-05 03:01:16,158 - root - INFO - Expected Directory Structure and Actual Directory Structure are equal 2021-02-05 03:01:16,177 - root - INFO - creating direct client 172.23.100.16:11210 default 2021-02-05 03:01:16,497 - root - INFO - creating direct client 172.23.100.17:11210 default 2021-02-05 03:01:16,887 - root - INFO - SSH Connecting to 172.23.100.16 with username:root, attempt#1 of 5 2021-02-05 03:01:16,989 - root - INFO - SSH Connected to 172.23.100.16 as root 2021-02-05 03:01:17,247 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:17,539 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:17,540 - root - INFO - Run cbtransfer to get data map 2021-02-05 03:01:18,614 - root - INFO - done execute cbtransfer 2021-02-05 03:01:18,713 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 03:01:18,812 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 03:01:19,082 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:19,379 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:19,380 - root - INFO - Run cbtransfer to get data map 2021-02-05 03:01:20,438 - root - INFO - done execute cbtransfer 2021-02-05 03:01:20,547 - root - INFO - SSH Connecting to 172.23.100.16 with username:root, attempt#1 of 5 2021-02-05 03:01:20,649 - root - INFO - SSH Connected to 172.23.100.16 as root 2021-02-05 03:01:20,902 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:21,195 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:21,195 - root - INFO - Run cbtransfer to get data map 2021-02-05 03:01:22,197 - root - INFO - done execute cbtransfer 2021-02-05 03:01:22,299 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 03:01:22,397 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 03:01:22,656 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:22,950 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:22,950 - root - INFO - Run cbtransfer to get data map 2021-02-05 03:01:23,963 - root - INFO - done execute cbtransfer 2021-02-05 03:01:24,071 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:01:24,238 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:01:24,805 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:25,199 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:25,207 - root - INFO - running command.raw on 172.23.121.22: cat /tmp/entbackup_172.23.100.17/backup/2021-02-05T03_01_09.37091013-08_00/default-d90d8d4126c4d85fcd6796f227a0c9db/range.json 2021-02-05 03:01:25,289 - root - INFO - command executed successfully with root 2021-02-05 03:01:25,305 - root - INFO - -->is_ns_server_running? 2021-02-05 03:01:25,333 - root - INFO - /diag/eval status on 172.23.100.18:8091: True content: ok command: gen_server:cast(ns_cluster, leave). 2021-02-05 03:01:25,337 - root - INFO - couchbase server is up but down soon. 2021-02-05 03:01:26,339 - root - INFO - couchbase server is down. Waiting for couchbase server up 2021-02-05 03:01:28,345 - root - INFO - couchbase server is up again in few seconds 2021-02-05 03:01:35,352 - root - INFO - -->is_ns_server_running? 2021-02-05 03:01:35,357 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 03:01:35,457 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 03:01:35,714 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:36,007 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:36,014 - root - ERROR - GET http://172.23.100.18:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-05 03:01:36,016 - root - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases 2021-02-05 03:01:37,023 - root - INFO - quota for index service will be 256 MB 2021-02-05 03:01:37,023 - root - INFO - set index quota to node 172.23.100.18 2021-02-05 03:01:37,023 - root - INFO - pools/default params : indexMemoryQuota=256 2021-02-05 03:01:37,026 - root - INFO - quota for fts service will be 512 MB 2021-02-05 03:01:37,027 - root - INFO - set both index and fts quota at node 172.23.100.18 2021-02-05 03:01:37,027 - root - INFO - pools/default params : ftsMemoryQuota=512 2021-02-05 03:01:37,030 - root - INFO - quota for kv: 1915 MB 2021-02-05 03:01:37,030 - root - INFO - pools/default params : memoryQuota=1915 2021-02-05 03:01:37,033 - root - INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['eventing', 'fts', 'index', 'kv', 'n1ql']) 2021-02-05 03:01:37,033 - root - INFO - /node/controller/setupServices params on 172.23.100.18: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=eventing%2Cfts%2Cindex%2Ckv%2Cn1ql 2021-02-05 03:01:37,058 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 03:01:37,058 - root - INFO - settings/web params on 172.23.100.18:8091:port=8091&username=Administrator&password=password 2021-02-05 03:01:37,107 - root - INFO - --> status:True 2021-02-05 03:01:37,108 - root - INFO - adding remote node @172.23.106.18:8091 to this cluster @172.23.100.18:8091 2021-02-05 03:01:41,305 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.18,ns_1@172.23.106.18', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-02-05 03:01:41,332 - root - INFO - rebalance operation started 2021-02-05 03:01:41,334 - root - INFO - rebalance percentage : 0.00 % 2021-02-05 03:01:41,334 - root - INFO - Rebalance - status: running, progress: 0.00% 2021-02-05 03:01:51,350 - root - INFO - Rebalance - status: none, progress: 100.00% 2021-02-05 03:01:51,355 - root - INFO - rebalancing was completed with progress: 100% in 10.022752046585083 sec 2021-02-05 03:01:51,361 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 03:01:51,463 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 03:01:51,727 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:52,023 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:52,030 - root - INFO - running command.raw on 172.23.100.18: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 03:01:52,085 - root - INFO - command executed successfully with root 2021-02-05 03:01:52,085 - root - INFO - ['ok'] 2021-02-05 03:01:52,087 - root - INFO - SSH Connecting to 172.23.106.18 with username:root, attempt#1 of 5 2021-02-05 03:01:52,192 - root - INFO - SSH Connected to 172.23.106.18 as root 2021-02-05 03:01:52,448 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:52,751 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:52,761 - root - INFO - running command.raw on 172.23.106.18: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 03:01:52,816 - root - INFO - command executed successfully with root 2021-02-05 03:01:52,817 - root - INFO - ['ok'] 2021-02-05 03:01:52,817 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 03:01:52,823 - root - INFO - Done reset node 2021-02-05 03:01:53,831 - root - INFO - quota for index service will be 256 MB 2021-02-05 03:01:53,831 - root - INFO - set index quota to node 172.23.100.18 2021-02-05 03:01:53,831 - root - INFO - pools/default params : indexMemoryQuota=256 2021-02-05 03:01:53,835 - root - INFO - quota for fts service will be 512 MB 2021-02-05 03:01:53,835 - root - INFO - set both index and fts quota at node 172.23.100.18 2021-02-05 03:01:53,835 - root - INFO - pools/default params : ftsMemoryQuota=512 2021-02-05 03:01:53,839 - root - INFO - quota for kv: 1378 MB 2021-02-05 03:01:53,839 - root - INFO - pools/default params : memoryQuota=1378 2021-02-05 03:01:53,844 - root - INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['eventing', 'fts', 'index', 'kv', 'n1ql']) 2021-02-05 03:01:53,844 - root - INFO - /node/controller/setupServices params on 172.23.100.18: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=eventing%2Cfts%2Cindex%2Ckv%2Cn1ql 2021-02-05 03:01:53,847 - root - ERROR - POST http://172.23.100.18:8091//node/controller/setupServices body: hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=eventing%2Cfts%2Cindex%2Ckv%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password 2021-02-05 03:01:53,847 - root - INFO - This node is already provisioned with services, we do not consider this as failure for test case 2021-02-05 03:01:53,847 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 03:01:53,847 - root - INFO - settings/web params on 172.23.100.18:8091:port=8091&username=Administrator&password=password 2021-02-05 03:01:53,901 - root - INFO - --> status:True 2021-02-05 03:01:53,914 - root - INFO - Node version in cluster 7.0.0-4347-enterprise 2021-02-05 03:01:53,942 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 03:01:54,041 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 03:01:54,304 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:01:54,597 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:01:54,603 - root - INFO - running command.raw on 172.23.100.18: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 03:01:54,659 - root - INFO - command executed successfully with root 2021-02-05 03:01:54,659 - root - INFO - ['ok'] 2021-02-05 03:01:54,660 - root - INFO - Services in restore cluster: ['eventing', 'fts', 'index', 'kv', 'n1ql'] 2021-02-05 03:01:54,670 - root - INFO - node 172.23.100.18 existing buckets : [] 2021-02-05 03:01:54,671 - root - INFO - Creating bucket default in restore host 172.23.100.18 2021-02-05 03:01:54,679 - root - INFO - -->is_ns_server_running? 2021-02-05 03:01:54,687 - root - INFO - rebalancing all nodes in order to remove nodes 2021-02-05 03:01:54,689 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.18,ns_1@172.23.106.18', 'ejectedNodes': 'ns_1@172.23.106.18', 'user': 'Administrator', 'password': 'password'} 2021-02-05 03:01:54,715 - root - INFO - rebalance operation started 2021-02-05 03:01:54,717 - root - INFO - rebalance percentage : 0.00 % 2021-02-05 03:02:04,730 - root - INFO - rebalance percentage : 45.00 % 2021-02-05 03:02:24,756 - root - INFO - rebalance progress took 30.04 seconds 2021-02-05 03:02:24,756 - root - INFO - sleep for 10 seconds after rebalance... 2021-02-05 03:02:34,774 - root - INFO - removed all the nodes from cluster associated with ip:172.23.100.18 port:8091 ssh_username:root ? [('ns_1@172.23.106.18', 8091)] 2021-02-05 03:02:34,796 - root - INFO - /diag/eval status on 172.23.100.18:8091: True content: ok command: gen_server:cast(ns_cluster, leave). 2021-02-05 03:02:34,799 - root - INFO - couchbase server is up but down soon. 2021-02-05 03:02:35,801 - root - INFO - couchbase server is down. Waiting for couchbase server up 2021-02-05 03:02:37,807 - root - INFO - couchbase server is up again in few seconds 2021-02-05 03:02:44,814 - root - INFO - -->is_ns_server_running? 2021-02-05 03:02:44,820 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 03:02:44,919 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 03:02:45,176 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:02:45,469 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:02:45,476 - root - ERROR - GET http://172.23.100.18:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-05 03:02:45,476 - root - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases 2021-02-05 03:02:46,484 - root - INFO - quota for index service will be 256 MB 2021-02-05 03:02:46,484 - root - INFO - set index quota to node 172.23.100.18 2021-02-05 03:02:46,484 - root - INFO - pools/default params : indexMemoryQuota=256 2021-02-05 03:02:46,488 - root - INFO - quota for fts service will be 512 MB 2021-02-05 03:02:46,488 - root - INFO - set both index and fts quota at node 172.23.100.18 2021-02-05 03:02:46,488 - root - INFO - pools/default params : ftsMemoryQuota=512 2021-02-05 03:02:46,492 - root - INFO - quota for kv: 1915 MB 2021-02-05 03:02:46,492 - root - INFO - pools/default params : memoryQuota=1915 2021-02-05 03:02:46,496 - root - INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['eventing', 'fts', 'index', 'kv', 'n1ql']) 2021-02-05 03:02:46,496 - root - INFO - /node/controller/setupServices params on 172.23.100.18: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=eventing%2Cfts%2Cindex%2Ckv%2Cn1ql 2021-02-05 03:02:46,521 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 03:02:46,521 - root - INFO - settings/web params on 172.23.100.18:8091:port=8091&username=Administrator&password=password 2021-02-05 03:02:46,572 - root - INFO - --> status:True 2021-02-05 03:02:46,572 - root - INFO - adding remote node @172.23.106.18:8091 to this cluster @172.23.100.18:8091 2021-02-05 03:02:49,424 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.18,ns_1@172.23.106.18', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-02-05 03:02:49,450 - root - INFO - rebalance operation started 2021-02-05 03:02:49,452 - root - INFO - rebalance percentage : 0.00 % 2021-02-05 03:02:49,452 - root - INFO - Rebalance - status: running, progress: 0.00% 2021-02-05 03:02:59,471 - root - INFO - Rebalance - status: none, progress: 100.00% 2021-02-05 03:02:59,475 - root - INFO - rebalancing was completed with progress: 100% in 10.024897336959839 sec 2021-02-05 03:02:59,481 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 03:02:59,581 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 03:02:59,840 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:03:00,133 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:03:00,139 - root - INFO - running command.raw on 172.23.100.18: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 03:03:00,196 - root - INFO - command executed successfully with root 2021-02-05 03:03:00,196 - root - INFO - ['ok'] 2021-02-05 03:03:00,198 - root - INFO - SSH Connecting to 172.23.106.18 with username:root, attempt#1 of 5 2021-02-05 03:03:00,299 - root - INFO - SSH Connected to 172.23.106.18 as root 2021-02-05 03:03:00,556 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:03:00,857 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:03:00,867 - root - INFO - running command.raw on 172.23.106.18: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-05 03:03:00,922 - root - INFO - command executed successfully with root 2021-02-05 03:03:00,922 - root - INFO - ['ok'] 2021-02-05 03:03:00,922 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-05 03:03:00,926 - root - INFO - Done reset node 2021-02-05 03:03:01,935 - root - INFO - quota for index service will be 256 MB 2021-02-05 03:03:01,935 - root - INFO - set index quota to node 172.23.100.18 2021-02-05 03:03:01,935 - root - INFO - pools/default params : indexMemoryQuota=256 2021-02-05 03:03:01,940 - root - INFO - quota for fts service will be 512 MB 2021-02-05 03:03:01,940 - root - INFO - set both index and fts quota at node 172.23.100.18 2021-02-05 03:03:01,940 - root - INFO - pools/default params : ftsMemoryQuota=512 2021-02-05 03:03:01,946 - root - INFO - quota for kv: 1378 MB 2021-02-05 03:03:01,946 - root - INFO - pools/default params : memoryQuota=1378 2021-02-05 03:03:01,950 - root - INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['eventing', 'fts', 'index', 'kv', 'n1ql']) 2021-02-05 03:03:01,950 - root - INFO - /node/controller/setupServices params on 172.23.100.18: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=eventing%2Cfts%2Cindex%2Ckv%2Cn1ql 2021-02-05 03:03:01,952 - root - ERROR - POST http://172.23.100.18:8091//node/controller/setupServices body: hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=eventing%2Cfts%2Cindex%2Ckv%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password 2021-02-05 03:03:01,952 - root - INFO - This node is already provisioned with services, we do not consider this as failure for test case 2021-02-05 03:03:01,952 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-05 03:03:01,952 - root - INFO - settings/web params on 172.23.100.18:8091:port=8091&username=Administrator&password=password 2021-02-05 03:03:02,005 - root - INFO - --> status:True 2021-02-05 03:03:02,005 - root - INFO - replica in bucket default is 1 2021-02-05 03:03:02,022 - root - INFO - http://172.23.100.18:8091/pools/default/buckets with param: name=default&authType=sasl&saslPassword=&ramQuotaMB=964&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-02-05 03:03:02,062 - root - INFO - 0.04 seconds to create bucket default 2021-02-05 03:03:02,605 - root - INFO - sleep for 5 secs. wait for bucket is up ... 2021-02-05 03:03:07,628 - root - INFO - sleep for 15 secs. wait for index service ready ... 2021-02-05 03:03:22,645 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:03:22,817 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:03:23,379 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:03:23,763 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:03:23,763 - root - INFO - running command.raw on 172.23.121.22: /opt/couchbase/bin//cbbackupmgr restore --archive /tmp/entbackup_172.23.100.17 --repo backup --cluster http://172.23.100.18:8091 --username Administrator --password password --start 2021-02-05T03_01_09.37091013-08_00 --end 2021-02-05T03_01_09.37091013-08_00 --no-progress-bar 2021-02-05 03:03:26,314 - root - INFO - command executed successfully with root 2021-02-05 03:03:26,314 - root - INFO - Restore bucket 'default' succeeded 2021-02-05 03:03:26,314 - root - INFO - Mutations restored: 1000, Mutations failed to restore: 0 2021-02-05 03:03:26,314 - root - INFO - Deletions restored: 0, Deletions failed to restore: 0 2021-02-05 03:03:26,315 - root - INFO - Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 2021-02-05 03:03:26,315 - root - INFO - Restore completed successfully 2021-02-05 03:03:26,315 - root - INFO - Restored 1 backups 2021-02-05 03:03:26,315 - root - INFO - Restored all data in 2.443109862s (Avg. 67.77KB/Sec) 2021-02-05 03:03:26,315 - root - INFO - 1000 items / 165.58KB 2021-02-05 03:03:26,325 - root - INFO - Node version in cluster 7.0.0-4347-enterprise 2021-02-05 03:03:26,357 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:03:26,513 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:03:27,060 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:03:27,442 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:03:27,455 - root - INFO - Node version in cluster 7.0.0-4347-enterprise 2021-02-05 03:03:27,455 - root - INFO - running command.raw on 172.23.121.22: grep 'Restore completed successfully' /tmp/entbackup_172.23.100.17/logs/backup-*.log 2021-02-05 03:03:27,510 - root - INFO - command executed successfully with root 2021-02-05 03:03:27,511 - root - INFO - running command.raw on 172.23.121.22: grep 'Transfer failed' /tmp/entbackup_172.23.100.17/logs/backup-*.log 2021-02-05 03:03:27,576 - root - INFO - command executed successfully with root 2021-02-05 03:03:27,576 - root - INFO - Finished restoring backup 2021-02-05 03:03:27,586 - root - INFO - Get current vseqno on node 172.23.100.18 2021-02-05 03:03:27,597 - root - INFO - **** add built-in 'cbadminbucket' user to node 172.23.100.18 **** 2021-02-05 03:03:27,609 - root - ERROR - DELETE http://172.23.100.18:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-02-05 03:03:27,609 - root - INFO - Exception while deleting user. Exception is -b'"User was not found."' 2021-02-05 03:03:27,689 - root - INFO - sleep for 5 secs. ... 2021-02-05 03:03:32,695 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2021-02-05 03:03:32,724 - root - INFO - creating direct client 172.23.100.18:11210 default 2021-02-05 03:03:33,023 - root - INFO - creating direct client 172.23.106.18:11210 default 2021-02-05 03:03:33,384 - root - INFO - *** Start to validate the restore 2021-02-05 03:03:33,384 - root - INFO - backup start: 1 2021-02-05 03:03:33,384 - root - INFO - backup end: 1 2021-02-05 03:03:33,394 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 03:03:33,492 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 03:03:33,752 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:03:34,044 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:03:34,045 - root - INFO - Run cbtransfer to get data map 2021-02-05 03:03:35,125 - root - INFO - done execute cbtransfer 2021-02-05 03:03:35,226 - root - INFO - Compare backup data in bucket default 2021-02-05 03:03:35,228 - root - INFO - End Verification for vbucket sequence numbers comparison Expected default Data and Actual default Data are equal Data validation success 2021-02-05 03:03:35,228 - root - INFO - Collecting the backup artifacts 2021-02-05 03:03:35,230 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:03:35,381 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:03:35,869 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:03:36,246 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:03:36,246 - root - INFO - running command.raw on 172.23.121.22: yum -y install zip 2021-02-05 03:03:51,386 - root - INFO - command executed successfully with root 2021-02-05 03:03:51,387 - root - INFO - running command.raw on 172.23.121.22: zip -r /tmp/archive.zip /tmp/entbackup_172.23.100.17 2021-02-05 03:03:53,476 - root - INFO - command executed successfully with root 2021-02-05 03:03:53,684 - root - INFO - FTS index found in restore cluster as expected 2021-02-05 03:03:53,693 - root - INFO - FTS alias found in restore cluster as expected 2021-02-05 03:03:53,722 - root - ERROR - DELETE http://172.23.106.18:8094/api/index/index_default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: rest_delete_index: error deleting index, err: manager_api: could not delete index, indexDefs.ImplVersion: > mgr.version: 5.5.0 b'{"error":"rest_delete_index: error deleting index, err: manager_api: could not delete index, indexDefs.ImplVersion: \\u003e mgr.version: 5.5.0","request":"","status":"fail"}\n' auth: Administrator:password 2021-02-05 03:03:53,725 - root - ERROR - DELETE http://172.23.106.18:8094/api/index/alias_1612522868 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: rest_delete_index: error deleting index, err: manager_api: could not delete index, indexDefs.ImplVersion: > mgr.version: 5.5.0 b'{"error":"rest_delete_index: error deleting index, err: manager_api: could not delete index, indexDefs.ImplVersion: \\u003e mgr.version: 5.5.0","request":"","status":"fail"}\n' auth: Administrator:password 2021-02-05 03:03:53,725 - root - INFO - ============== NewUpgradeBaseTest tearDown has started ============== 2021-02-05 03:03:53,749 - root - INFO - ------- Cluster statistics ------- 2021-02-05 03:03:53,749 - root - INFO - 172.23.100.16:8091 => {'services': ['fts', 'kv'], 'cpu_utilization': 1.510193808205386, 'mem_free': 3470589952, 'mem_total': 4201684992, 'swap_mem_used': 61341696, 'swap_mem_total': 3758092288} 2021-02-05 03:03:53,749 - root - INFO - 172.23.100.17:8091 => {'services': ['eventing', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.859292701279157, 'mem_free': 3204149248, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-05 03:03:53,749 - root - INFO - --- End of cluster statistics --- 2021-02-05 03:03:53,751 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 03:03:53,853 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 03:03:54,120 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:03:54,416 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:03:54,418 - root - INFO - SSH Connecting to 172.23.100.16 with username:root, attempt#1 of 5 2021-02-05 03:03:54,521 - root - INFO - SSH Connected to 172.23.100.16 as root 2021-02-05 03:03:54,779 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:03:55,076 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:03:58,598 - root - INFO - ============== basetestcase cleanup was started for test #1 test_backup_restore_with_fts ============== 2021-02-05 03:03:58,626 - root - INFO - deleting existing buckets ['default'] on 172.23.100.17 2021-02-05 03:04:00,054 - root - INFO - waiting for bucket deletion to complete.... 2021-02-05 03:04:00,057 - root - INFO - node 172.23.100.17 existing buckets : [] 2021-02-05 03:04:00,057 - root - INFO - deleted bucket : default from 172.23.100.17 2021-02-05 03:04:00,063 - root - INFO - Could not find any buckets for node 172.23.100.16, nothing to delete 2021-02-05 03:04:00,067 - root - INFO - -->is_ns_server_running? 2021-02-05 03:04:00,077 - root - INFO - rebalancing all nodes in order to remove nodes 2021-02-05 03:04:00,079 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.16,ns_1@172.23.100.17', 'ejectedNodes': 'ns_1@172.23.100.16', 'user': 'Administrator', 'password': 'password'} 2021-02-05 03:04:00,110 - root - INFO - rebalance operation started 2021-02-05 03:04:00,113 - root - INFO - rebalance percentage : 0.00 % 2021-02-05 03:04:10,126 - root - INFO - rebalance percentage : 55.00 % 2021-02-05 03:04:30,145 - root - INFO - rebalance progress took 30.03 seconds 2021-02-05 03:04:30,145 - root - INFO - sleep for 10 seconds after rebalance... 2021-02-05 03:04:40,162 - root - INFO - removed all the nodes from cluster associated with ip:172.23.100.17 port:8091 ssh_username:root ? [('ns_1@172.23.100.16', 8091)] 2021-02-05 03:04:40,166 - root - INFO - waiting for ns_server @ 172.23.100.17:8091 2021-02-05 03:04:40,166 - root - INFO - -->is_ns_server_running? 2021-02-05 03:04:40,170 - root - INFO - ns_server @ 172.23.100.17:8091 is running 2021-02-05 03:04:40,173 - root - INFO - waiting for ns_server @ 172.23.100.16:8091 2021-02-05 03:04:40,173 - root - INFO - -->is_ns_server_running? 2021-02-05 03:04:40,175 - root - INFO - ns_server @ 172.23.100.16:8091 is running 2021-02-05 03:04:40,175 - root - INFO - ============== basetestcase cleanup was finished for test #1 test_backup_restore_with_fts ============== 2021-02-05 03:04:40,175 - root - INFO - closing all ssh connections 2021-02-05 03:04:40,176 - root - INFO - closing all memcached connections Cluster instance shutdown with force 2021-02-05 03:04:40,181 - root - INFO - sleep for 20 secs. sleep 20 seconds before run next test ... 2021-02-05 03:05:00,201 - root - INFO - ============== NewUpgradeBaseTest tearDown has completed ============== 2021-02-05 03:05:00,203 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:05:00,471 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:05:01,168 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:05:01,732 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-05 03:05:01,733 - root - INFO - running command.raw on 172.23.121.22: rm -rf /tmp/entbackup_172.23.100.17 2021-02-05 03:05:02,111 - root - INFO - command executed successfully with root 2021-02-05 03:05:02,111 - root - INFO - running command.raw on 172.23.121.22: rm -rf /cbqe3043/entbackup 2021-02-05 03:05:02,200 - root - INFO - command executed successfully with root 2021-02-05 03:05:02,215 - root - INFO - deleting existing buckets ['default'] on 172.23.100.18 2021-02-05 03:05:02,792 - root - INFO - waiting for bucket deletion to complete.... 2021-02-05 03:05:02,795 - root - INFO - node 172.23.100.18 existing buckets : [] 2021-02-05 03:05:02,795 - root - INFO - deleted bucket : default from 172.23.100.18 2021-02-05 03:05:02,802 - root - INFO - Could not find any buckets for node 172.23.106.18, nothing to delete 2021-02-05 03:05:02,842 - root - INFO - Could not find any buckets for node 172.23.108.239, nothing to delete 2021-02-05 03:05:02,847 - root - INFO - -->is_ns_server_running? 2021-02-05 03:05:02,856 - root - INFO - rebalancing all nodes in order to remove nodes 2021-02-05 03:05:02,858 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.18,ns_1@172.23.106.18', 'ejectedNodes': 'ns_1@172.23.106.18', 'user': 'Administrator', 'password': 'password'} 2021-02-05 03:05:02,890 - root - INFO - rebalance operation started 2021-02-05 03:05:02,892 - root - INFO - rebalance percentage : 0.00 % 2021-02-05 03:05:12,905 - root - ERROR - {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2021-02-05 03:05:12,909 - root - INFO - Latest logs from UI on 172.23.100.18: 2021-02-05 03:05:12,910 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1612523102980, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,fts,\n {worker_died,\n {\'EXIT\',<0.29384.0>,\n {rebalance_failed,\n {service_error,\n <<"planner: indexDefs.ImplVersion: > version: 5.5.0">>}}}}}.\nRebalance Operation Id = 80be7d36f7d7e482b83fa640f72b66d1', 'serverTime': '2021-02-05T03:05:02.980Z'} 2021-02-05 03:05:12,910 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1612523102860, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.100.18'], EjectNodes = ['ns_1@172.23.106.18'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 80be7d36f7d7e482b83fa640f72b66d1", 'serverTime': '2021-02-05T03:05:02.860Z'} 2021-02-05 03:05:12,910 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'warning', 'code': 102, 'module': 'menelaus_web', 'tstamp': 1612523102858, 'shortText': 'client-side error report', 'text': 'Client-side error-report for user "Administrator" on node \'ns_1@172.23.100.18\':\nUser-Agent:Python-httplib2/0.13.1 (gzip)\nStarting rebalance from test, ejected nodes [\'ns_1@172.23.106.18\']', 'serverTime': '2021-02-05T03:05:02.858Z'} 2021-02-05 03:05:12,910 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'info', 'code': 11, 'module': 'menelaus_web', 'tstamp': 1612523102792, 'shortText': 'message', 'text': 'Deleted bucket "default"\n', 'serverTime': '2021-02-05T03:05:02.792Z'} 2021-02-05 03:05:12,910 - root - ERROR - {'node': 'ns_1@172.23.106.18', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1612523102389, 'shortText': 'message', 'text': 'Shutting down bucket "default" on \'ns_1@172.23.106.18\' for deletion', 'serverTime': '2021-02-05T03:05:02.389Z'} 2021-02-05 03:05:12,910 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1612523102246, 'shortText': 'message', 'text': 'Shutting down bucket "default" on \'ns_1@172.23.100.18\' for deletion', 'serverTime': '2021-02-05T03:05:02.246Z'} 2021-02-05 03:05:12,910 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1612522982799, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.100.18\' in 0 seconds.', 'serverTime': '2021-02-05T03:03:02.799Z'} 2021-02-05 03:05:12,910 - root - ERROR - {'node': 'ns_1@172.23.106.18', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1612522982217, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.106.18\' in 0 seconds.', 'serverTime': '2021-02-05T03:03:02.217Z'} 2021-02-05 03:05:12,910 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'info', 'code': 12, 'module': 'menelaus_web', 'tstamp': 1612522982062, 'shortText': 'message', 'text': 'Created bucket "default" of type: couchbase\n[{num_replicas,1},\n {replica_index,true},\n {ram_quota,1010827264},\n {durability_min_level,none},\n {pitr_enabled,false},\n {pitr_granularity,600},\n {pitr_max_history_age,86400},\n {flush_enabled,true},\n {num_threads,3},\n {eviction_policy,value_only},\n {conflict_resolution_type,seqno},\n {storage_mode,couchstore},\n {max_ttl,0},\n {compression_mode,passive}]', 'serverTime': '2021-02-05T03:03:02.062Z'} 2021-02-05 03:05:12,910 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1612522973522, 'shortText': 'message', 'text': 'Rebalance completed successfully.\nRebalance Operation Id = 6a629ec286cba8585ccb3d8e105451dc', 'serverTime': '2021-02-05T03:02:53.522Z'} 2021-02-05 03:05:12,910 - root - ERROR - was not able to cleanup cluster the first time 2021-02-05 03:05:12,916 - root - INFO - Could not find any buckets for node 172.23.100.18, nothing to delete 2021-02-05 03:05:12,922 - root - INFO - Could not find any buckets for node 172.23.106.18, nothing to delete 2021-02-05 03:05:12,965 - root - INFO - Could not find any buckets for node 172.23.108.239, nothing to delete 2021-02-05 03:05:12,969 - root - INFO - -->is_ns_server_running? 2021-02-05 03:05:12,978 - root - INFO - rebalancing all nodes in order to remove nodes 2021-02-05 03:05:12,980 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.18,ns_1@172.23.106.18', 'ejectedNodes': 'ns_1@172.23.106.18', 'user': 'Administrator', 'password': 'password'} 2021-02-05 03:05:13,006 - root - INFO - rebalance operation started 2021-02-05 03:05:13,008 - root - INFO - rebalance percentage : 0.00 % 2021-02-05 03:05:23,021 - root - ERROR - {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2021-02-05 03:05:23,024 - root - INFO - Latest logs from UI on 172.23.100.18: 2021-02-05 03:05:23,025 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1612523113146, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,fts,\n {worker_died,\n {\'EXIT\',<0.30069.0>,\n {rebalance_failed,\n {service_error,\n <<"planner: indexDefs.ImplVersion: > version: 5.5.0">>}}}}}.\nRebalance Operation Id = 2da1ce2ce990d778791ca7e67931109d', 'serverTime': '2021-02-05T03:05:13.146Z'} 2021-02-05 03:05:23,025 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1612523112983, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.100.18'], EjectNodes = ['ns_1@172.23.106.18'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 2da1ce2ce990d778791ca7e67931109d", 'serverTime': '2021-02-05T03:05:12.983Z'} 2021-02-05 03:05:23,025 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1612523102980, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,fts,\n {worker_died,\n {\'EXIT\',<0.29384.0>,\n {rebalance_failed,\n {service_error,\n <<"planner: indexDefs.ImplVersion: > version: 5.5.0">>}}}}}.\nRebalance Operation Id = 80be7d36f7d7e482b83fa640f72b66d1', 'serverTime': '2021-02-05T03:05:02.980Z'} 2021-02-05 03:05:23,025 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1612523102860, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.100.18'], EjectNodes = ['ns_1@172.23.106.18'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 80be7d36f7d7e482b83fa640f72b66d1", 'serverTime': '2021-02-05T03:05:02.860Z'} 2021-02-05 03:05:23,025 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'warning', 'code': 102, 'module': 'menelaus_web', 'tstamp': 1612523102858, 'shortText': 'client-side error report', 'text': 'Client-side error-report for user "Administrator" on node \'ns_1@172.23.100.18\':\nUser-Agent:Python-httplib2/0.13.1 (gzip)\nStarting rebalance from test, ejected nodes [\'ns_1@172.23.106.18\']', 'serverTime': '2021-02-05T03:05:02.858Z'} 2021-02-05 03:05:23,025 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'info', 'code': 11, 'module': 'menelaus_web', 'tstamp': 1612523102792, 'shortText': 'message', 'text': 'Deleted bucket "default"\n', 'serverTime': '2021-02-05T03:05:02.792Z'} 2021-02-05 03:05:23,025 - root - ERROR - {'node': 'ns_1@172.23.106.18', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1612523102389, 'shortText': 'message', 'text': 'Shutting down bucket "default" on \'ns_1@172.23.106.18\' for deletion', 'serverTime': '2021-02-05T03:05:02.389Z'} 2021-02-05 03:05:23,025 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1612523102246, 'shortText': 'message', 'text': 'Shutting down bucket "default" on \'ns_1@172.23.100.18\' for deletion', 'serverTime': '2021-02-05T03:05:02.246Z'} 2021-02-05 03:05:23,025 - root - ERROR - {'node': 'ns_1@172.23.100.18', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1612522982799, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.100.18\' in 0 seconds.', 'serverTime': '2021-02-05T03:03:02.799Z'} 2021-02-05 03:05:23,025 - root - ERROR - {'node': 'ns_1@172.23.106.18', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1612522982217, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.106.18\' in 0 seconds.', 'serverTime': '2021-02-05T03:03:02.217Z'} 2021-02-05 03:05:23,028 - root - INFO - SSH Connecting to 172.23.100.17 with username:root, attempt#1 of 5 2021-02-05 03:05:23,030 - root - INFO - SSH Connecting to 172.23.100.15 with username:root, attempt#1 of 5 2021-02-05 03:05:23,032 - root - INFO - SSH Connecting to 172.23.100.16 with username:root, attempt#1 of 5 2021-02-05 03:05:23,034 - root - INFO - SSH Connecting to 172.23.100.18 with username:root, attempt#1 of 5 2021-02-05 03:05:23,036 - root - INFO - SSH Connecting to 172.23.106.18 with username:root, attempt#1 of 5 2021-02-05 03:05:23,038 - root - INFO - SSH Connecting to 172.23.108.239 with username:root, attempt#1 of 5 2021-02-05 03:05:23,040 - root - INFO - SSH Connecting to 172.23.121.22 with username:root, attempt#1 of 5 2021-02-05 03:05:23,041 - root - INFO - SSH Connecting to 172.23.105.79 with username:root, attempt#1 of 5 2021-02-05 03:05:23,045 - root - INFO - SSH Connecting to 172.23.98.15 with username:root, attempt#1 of 5 2021-02-05 03:05:23,132 - root - INFO - SSH Connected to 172.23.100.17 as root 2021-02-05 03:05:23,133 - root - INFO - SSH Connected to 172.23.98.15 as root 2021-02-05 03:05:23,142 - root - INFO - SSH Connected to 172.23.100.16 as root 2021-02-05 03:05:23,143 - root - INFO - SSH Connected to 172.23.100.15 as root 2021-02-05 03:05:23,145 - root - INFO - SSH Connected to 172.23.106.18 as root 2021-02-05 03:05:23,147 - root - INFO - SSH Connected to 172.23.100.18 as root 2021-02-05 03:05:23,241 - root - INFO - SSH Connected to 172.23.108.239 as root 2021-02-05 03:05:23,303 - root - INFO - SSH Connected to 172.23.121.22 as root 2021-02-05 03:05:23,315 - root - INFO - SSH Connected to 172.23.105.79 as root 2021-02-05 03:05:23,415 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:05:23,424 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:05:23,433 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:05:23,439 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:05:23,440 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:05:23,448 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:05:23,652 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:05:23,732 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.100.16 2021-02-05 03:05:23,733 - root - INFO - running command.raw on 172.23.100.16: /opt/couchbase/bin/cbcollect_info 172.23.100.16-20210205-0305-diag.zip 2021-02-05 03:05:23,736 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.100.18 2021-02-05 03:05:23,737 - root - INFO - running command.raw on 172.23.100.18: /opt/couchbase/bin/cbcollect_info 172.23.100.18-20210205-0305-diag.zip 2021-02-05 03:05:23,739 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.106.18 2021-02-05 03:05:23,739 - root - INFO - running command.raw on 172.23.106.18: /opt/couchbase/bin/cbcollect_info 172.23.106.18-20210205-0305-diag.zip 2021-02-05 03:05:23,741 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.98.15 2021-02-05 03:05:23,742 - root - INFO - running command.raw on 172.23.98.15: /opt/couchbase/bin/cbcollect_info 172.23.98.15-20210205-0305-diag.zip 2021-02-05 03:05:23,747 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.100.15 2021-02-05 03:05:23,747 - root - INFO - running command.raw on 172.23.100.15: /opt/couchbase/bin/cbcollect_info 172.23.100.15-20210205-0305-diag.zip 2021-02-05 03:05:23,784 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.100.17 2021-02-05 03:05:23,786 - root - INFO - running command.raw on 172.23.100.17: /opt/couchbase/bin/cbcollect_info 172.23.100.17-20210205-0305-diag.zip 2021-02-05 03:05:24,015 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:05:24,095 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-05 03:05:24,121 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.108.239 2021-02-05 03:05:24,122 - root - INFO - running command.raw on 172.23.108.239: /opt/couchbase/bin/cbcollect_info 172.23.108.239-20210205-0305-diag.zip 2021-02-05 03:05:24,595 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.121.22 2021-02-05 03:05:24,596 - root - INFO - running command.raw on 172.23.121.22: /opt/couchbase/bin/cbcollect_info 172.23.121.22-20210205-0305-diag.zip 2021-02-05 03:05:24,668 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.105.79 2021-02-05 03:05:24,669 - root - INFO - running command.raw on 172.23.105.79: /opt/couchbase/bin/cbcollect_info 172.23.105.79-20210205-0305-diag.zip 2021-02-05 03:06:49,549 - root - INFO - command executed successfully with root 2021-02-05 03:06:49,564 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.108.239 2021-02-05 03:06:49,985 - root - INFO - running command.raw on 172.23.108.239: rm -f /root/172.23.108.239-20210205-0305-diag.zip 2021-02-05 03:06:50,038 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.98.15 2021-02-05 03:06:50,111 - root - INFO - running command.raw on 172.23.98.15: rm -f /root/172.23.98.15-20210205-0305-diag.zip 2021-02-05 03:06:50,130 - root - INFO - command executed successfully with root 2021-02-05 03:06:52,732 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.100.16 2021-02-05 03:06:53,054 - root - INFO - running command.raw on 172.23.100.16: rm -f /root/172.23.100.16-20210205-0305-diag.zip 2021-02-05 03:06:53,071 - root - INFO - command executed successfully with root 2021-02-05 03:06:55,062 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.100.15 2021-02-05 03:06:55,392 - root - INFO - running command.raw on 172.23.100.15: rm -f /root/172.23.100.15-20210205-0305-diag.zip 2021-02-05 03:06:55,408 - root - INFO - command executed successfully with root 2021-02-05 03:06:57,791 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.106.18 2021-02-05 03:06:58,185 - root - INFO - running command.raw on 172.23.106.18: rm -f /root/172.23.106.18-20210205-0305-diag.zip 2021-02-05 03:06:58,204 - root - INFO - command executed successfully with root 2021-02-05 03:07:18,506 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.105.79 2021-02-05 03:07:18,870 - root - INFO - running command.raw on 172.23.105.79: rm -f /root/172.23.105.79-20210205-0305-diag.zip 2021-02-05 03:07:18,902 - root - INFO - command executed successfully with root 2021-02-05 03:07:55,113 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.100.18 2021-02-05 03:07:55,419 - root - INFO - running command.raw on 172.23.100.18: rm -f /root/172.23.100.18-20210205-0305-diag.zip 2021-02-05 03:07:55,435 - root - INFO - command executed successfully with root 2021-02-05 03:08:02,092 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.100.17 2021-02-05 03:08:02,388 - root - INFO - running command.raw on 172.23.100.17: rm -f /root/172.23.100.17-20210205-0305-diag.zip 2021-02-05 03:08:02,404 - root - INFO - command executed successfully with root 2021-02-05 03:08:41,950 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.121.22 2021-02-05 03:08:42,397 - root - INFO - running command.raw on 172.23.121.22: rm -f /root/172.23.121.22-20210205-0305-diag.zip 2021-02-05 03:08:42,430 - root - INFO - command executed successfully with root ERROR ====================================================================== ERROR: test_backup_restore_with_fts (ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 375, in clean_up self.backup_reset_clusters(servers) File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 430, in backup_reset_clusters ClusterOperationHelper.cleanup_cluster(servers, master=servers[0]) File "lib/membase/helper/cluster_helper.py", line 267, in cleanup_cluster wait_for_rebalance=wait_for_rebalance) File "lib/membase/api/rest_client.py", line 116, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1783, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1882, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1859, in _rebalance_status_and_progress raise RebalanceFailedException(msg) membase.api.exception.RebalanceFailedException: Rebalance Failed: {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "pytests/ent_backup_restore/enterprise_backup_restore_test.py", line 70, in tearDown super(EnterpriseBackupRestoreTest, self).tearDown() File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 396, in tearDown self.clean_up(self.input.clusters[1][0]) File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 378, in clean_up self.backup_reset_clusters(servers) File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 430, in backup_reset_clusters ClusterOperationHelper.cleanup_cluster(servers, master=servers[0]) File "lib/membase/helper/cluster_helper.py", line 267, in cleanup_cluster wait_for_rebalance=wait_for_rebalance) File "lib/membase/api/rest_client.py", line 116, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1783, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1882, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1859, in _rebalance_status_and_progress raise RebalanceFailedException(msg) membase.api.exception.RebalanceFailedException: Rebalance Failed: {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 335.602s FAILED (errors=1) suite_tearDown (unittest.loader._FailedTest) ... ERROR ====================================================================== ERROR: suite_tearDown (unittest.loader._FailedTest) ---------------------------------------------------------------------- AttributeError: type object 'EnterpriseBackupRestoreTest' has no attribute 'suite_tearDown' ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (errors=1) summary so far suite ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest , pass 0 , fail 1 failures so far... ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest.test_backup_restore_with_fts testrunner logs, diags and results are available under /data/workspace/centos-p0-backup_recovery-vset00-00-1c/logs/testrunner-21-Feb-05_02-59-46/test_1 *** Tests executed count: 1 Run after suite setup for ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest.test_backup_restore_with_fts During the test, Remote Connections: 0, Disconnections: 94 ('Thread', , 'was not properly terminated, will be terminated now.') Shutting down the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Shutting down the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Stopping the thread... *** TestRunner *** workspace is /data/workspace/centos-p0-backup_recovery-vset00-00-1c fails is 1 1 Desc1: 7.0.0-4347 - backup_recovery 1c - centos (0/1) python3 scripts/rerun_jobs.py 7.0.0-4347 --executor_jenkins_job --run_params=get-coredumps=False,force_kill_memached=True,verify_unacked_bytes=True,get-cbcollect-info=True INFO:merge_reports:Merging of report files from logs/**/*.xml INFO:merge_reports:-- logs/testrunner-21-Feb-05_02-59-46/report-21-Feb-05_02-59-46-ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest.xml -- INFO:merge_reports: Number of TestSuites=1 INFO:merge_reports: TestSuite#1) ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest, Number of Tests=1 INFO:merge_reports:Summary file is at /data/workspace/centos-p0-backup_recovery-vset00-00-1c/logs/testrunner-21-Feb-05_03-08-43/merged_summary/mergedreport-21-Feb-05_03-08-43-ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest.xml summary so far suite ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest , pass 0 , fail 1 failures so far... ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest.test_backup_restore_with_fts No more failed tests. Stopping reruns [description-setter] Description set: 7.0.0-4347 - backup_recovery 1c - centos (0/1) [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) Sending email to: arunkumar.senthilnathan@couchbase.com thuan@couchbase.com Triggering a new build of savejoblogs Triggering a new build of test-executor-cleanup Finished: UNSTABLE