Started by user Balakumaran G [EnvInject] - Loading node environment variables. Building remotely on slv-s63803-32g-12c (jython_slave) in workspace /data/workspace/temp_rebalance_even [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbase/testrunner/ > /usr/bin/git init /data/workspace/temp_rebalance_even # timeout=10 Fetching upstream changes from git://github.com/couchbase/testrunner/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbase/testrunner/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbase/testrunner/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbase/testrunner/ # timeout=10 Fetching upstream changes from git://github.com/couchbase/testrunner/ > /usr/bin/git fetch --tags --progress git://github.com/couchbase/testrunner/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision a871fd112de599abb15d3191f474a4946731af6d (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f a871fd112de599abb15d3191f474a4946731af6d First time build. Skipping changelog. [temp_rebalance_even] $ /bin/sh -xe /tmp/jenkins1015817740304023518.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 #7:_7 [cluster1] 1:_1 2:_2 [cluster2] 1:_3 2:_4 #[_7] #ip:172.23.104.109 [_2] ip:172.23.120.206 [_3] ip:172.23.98.195 [_4] ip:172.23.121.10 [_5] ip:172.23.104.186 [_1] ip:172.23.98.196 [_6] ip:172.23.120.201' + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From git://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + python scripts/install.py -i /tmp/win10-bucket-ops.ini -p version=6.0.3-2879,product=cb,parallel=True 2019-09-19 01:35:31,651 - root - INFO - *****Starting the complete install process **** 2019-09-19 01:35:31,655 - root - INFO - Doing parallel install**** 2019-09-19 01:35:31,662 - root - INFO - connecting to 172.23.98.196 with username:root 2019-09-19 01:35:31,663 - root - INFO - connecting to 172.23.98.195 with username:root 2019-09-19 01:35:31,666 - root - INFO - connecting to 172.23.120.206 with username:root 2019-09-19 01:35:31,669 - root - INFO - connecting to 172.23.121.10 with username:root 2019-09-19 01:35:31,670 - root - INFO - connecting to 172.23.104.186 with username:root 2019-09-19 01:35:31,677 - root - INFO - connecting to 172.23.120.201 with username:root 2019-09-19 01:35:31,804 - root - INFO - Connected to 172.23.120.206 2019-09-19 01:35:31,813 - root - INFO - Connected to 172.23.120.201 2019-09-19 01:35:31,845 - root - INFO - Connected to 172.23.98.196 2019-09-19 01:35:31,846 - root - INFO - Connected to 172.23.121.10 2019-09-19 01:35:31,846 - root - INFO - Connected to 172.23.104.186 2019-09-19 01:35:31,877 - root - INFO - Connected to 172.23.98.195 2019-09-19 01:35:31,904 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:35:32,026 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:35:32,303 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:35:32,311 - root - INFO - 172.23.120.206 *****In couchbase uninstall**** 2019-09-19 01:35:32,312 - root - INFO - CentOS 2019-09-19 01:35:32,389 - root - INFO - running command.raw on 172.23.120.206: killall -9 rpm 2019-09-19 01:35:32,425 - root - INFO - command executed successfully 2019-09-19 01:35:32,425 - root - ERROR - rpm: no process found 2019-09-19 01:35:32,426 - root - INFO - running command.raw on 172.23.120.206: rm -f /var/lib/rpm/.rpm.lock 2019-09-19 01:35:32,436 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:35:32,469 - root - INFO - 172.23.120.201 *****In couchbase uninstall**** 2019-09-19 01:35:32,469 - root - INFO - CentOS 2019-09-19 01:35:32,475 - root - INFO - command executed successfully 2019-09-19 01:35:32,546 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:35:32,573 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:35:32,631 - root - INFO - running rpm -e to remove couchbase-server 2019-09-19 01:35:32,631 - root - INFO - running command.raw on 172.23.120.206: rpm -e couchbase-server 2019-09-19 01:35:32,654 - root - INFO - running command.raw on 172.23.120.201: killall -9 rpm 2019-09-19 01:35:32,688 - root - INFO - command executed successfully 2019-09-19 01:35:32,688 - root - ERROR - rpm: no process found 2019-09-19 01:35:32,688 - root - INFO - running command.raw on 172.23.120.201: rm -f /var/lib/rpm/.rpm.lock 2019-09-19 01:35:32,742 - root - INFO - 172.23.121.10 *****In couchbase uninstall**** 2019-09-19 01:35:32,742 - root - INFO - CentOS 2019-09-19 01:35:32,769 - root - INFO - command executed successfully 2019-09-19 01:35:32,883 - root - INFO - running rpm -e to remove couchbase-server 2019-09-19 01:35:32,883 - root - INFO - running command.raw on 172.23.120.201: rpm -e couchbase-server 2019-09-19 01:35:32,894 - root - INFO - 172.23.104.186 *****In couchbase uninstall**** 2019-09-19 01:35:32,894 - root - INFO - CentOS 2019-09-19 01:35:32,907 - root - INFO - running command.raw on 172.23.121.10: killall -9 rpm 2019-09-19 01:35:32,942 - root - INFO - command executed successfully 2019-09-19 01:35:32,942 - root - ERROR - bash: killall: command not found 2019-09-19 01:35:32,943 - root - INFO - running command.raw on 172.23.121.10: rm -f /var/lib/rpm/.rpm.lock 2019-09-19 01:35:32,989 - root - INFO - 172.23.98.196 *****In couchbase uninstall**** 2019-09-19 01:35:32,989 - root - INFO - CentOS 2019-09-19 01:35:33,009 - root - INFO - command executed successfully 2019-09-19 01:35:33,056 - root - INFO - 172.23.98.195 *****In couchbase uninstall**** 2019-09-19 01:35:33,057 - root - INFO - CentOS 2019-09-19 01:35:33,083 - root - INFO - running command.raw on 172.23.104.186: killall -9 rpm 2019-09-19 01:35:33,149 - root - INFO - running command.raw on 172.23.98.196: killall -9 rpm 2019-09-19 01:35:33,151 - root - INFO - command executed successfully 2019-09-19 01:35:33,152 - root - ERROR - rpm: no process found 2019-09-19 01:35:33,152 - root - INFO - running command.raw on 172.23.104.186: rm -f /var/lib/rpm/.rpm.lock 2019-09-19 01:35:33,158 - root - INFO - running rpm -e to remove couchbase-server 2019-09-19 01:35:33,158 - root - INFO - running command.raw on 172.23.121.10: rpm -e couchbase-server 2019-09-19 01:35:33,171 - root - INFO - running command.raw on 172.23.98.195: killall -9 rpm 2019-09-19 01:35:33,188 - root - INFO - command executed successfully 2019-09-19 01:35:33,267 - root - INFO - command executed successfully 2019-09-19 01:35:33,267 - root - ERROR - rpm: no process found 2019-09-19 01:35:33,267 - root - INFO - running command.raw on 172.23.98.196: rm -f /var/lib/rpm/.rpm.lock 2019-09-19 01:35:33,288 - root - INFO - command executed successfully 2019-09-19 01:35:33,288 - root - ERROR - rpm: no process found 2019-09-19 01:35:33,288 - root - INFO - running command.raw on 172.23.98.195: rm -f /var/lib/rpm/.rpm.lock 2019-09-19 01:35:33,302 - root - INFO - command executed successfully 2019-09-19 01:35:33,312 - root - INFO - running rpm -e to remove couchbase-server 2019-09-19 01:35:33,312 - root - INFO - running command.raw on 172.23.104.186: rpm -e couchbase-server 2019-09-19 01:35:33,323 - root - INFO - command executed successfully 2019-09-19 01:35:33,448 - root - INFO - running rpm -e to remove couchbase-server 2019-09-19 01:35:33,448 - root - INFO - running command.raw on 172.23.98.195: rpm -e couchbase-server 2019-09-19 01:35:33,457 - root - INFO - running rpm -e to remove couchbase-server 2019-09-19 01:35:33,458 - root - INFO - running command.raw on 172.23.98.196: rpm -e couchbase-server 2019-09-19 01:35:36,457 - root - INFO - command executed successfully 2019-09-19 01:35:36,457 - root - ERROR - **** Uninstalling on server: 172.23.120.201 **** 2019-09-19 01:35:36,457 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave 2019-09-19 01:35:36,457 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory 2019-09-19 01:35:36,458 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory 2019-09-19 01:35:36,458 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2019-09-19 01:35:36,458 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2019-09-19 01:35:36,458 - root - INFO - running command.raw on 172.23.120.201: pkill -u couchbase 2019-09-19 01:35:36,725 - root - INFO - command executed successfully 2019-09-19 01:35:36,725 - root - INFO - running command.raw on 172.23.120.201: ps -ef | grep couchbase 2019-09-19 01:35:36,761 - root - INFO - command executed successfully 2019-09-19 01:35:36,761 - root - INFO - root 9142 8944 0 01:35 ? 00:00:00 bash -c ps -ef | grep couchbase 2019-09-19 01:35:36,761 - root - INFO - root 9150 9142 0 01:35 ? 00:00:00 grep couchbase 2019-09-19 01:35:38,487 - root - INFO - running command.raw on 172.23.120.201: ipcrm 2019-09-19 01:35:38,568 - root - INFO - command executed successfully 2019-09-19 01:35:38,725 - root - INFO - command executed successfully 2019-09-19 01:35:38,726 - root - ERROR - **** Uninstalling on server: 172.23.98.195 **** 2019-09-19 01:35:38,726 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave 2019-09-19 01:35:38,726 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory 2019-09-19 01:35:38,726 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory 2019-09-19 01:35:38,726 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2019-09-19 01:35:38,727 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2019-09-19 01:35:38,727 - root - INFO - running command.raw on 172.23.98.195: pkill -u couchbase 2019-09-19 01:35:38,785 - root - INFO - command executed successfully 2019-09-19 01:35:38,785 - root - ERROR - **** Uninstalling on server: 172.23.98.196 **** 2019-09-19 01:35:38,786 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory 2019-09-19 01:35:38,786 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2019-09-19 01:35:38,786 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory 2019-09-19 01:35:38,786 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2019-09-19 01:35:38,786 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2019-09-19 01:35:38,786 - root - INFO - running command.raw on 172.23.98.196: pkill -u couchbase 2019-09-19 01:35:38,794 - root - INFO - command executed successfully 2019-09-19 01:35:38,794 - root - INFO - running command.raw on 172.23.98.195: ps -ef | grep couchbase 2019-09-19 01:35:38,853 - root - INFO - command executed successfully 2019-09-19 01:35:38,853 - root - INFO - running command.raw on 172.23.98.196: ps -ef | grep couchbase 2019-09-19 01:35:38,876 - root - INFO - command executed successfully 2019-09-19 01:35:38,876 - root - INFO - root 7479 7298 0 01:35 ? 00:00:00 bash -c ps -ef | grep couchbase 2019-09-19 01:35:38,876 - root - INFO - root 7487 7479 0 01:35 ? 00:00:00 grep couchbase 2019-09-19 01:35:38,936 - root - INFO - command executed successfully 2019-09-19 01:35:38,937 - root - ERROR - **** Uninstalling on server: 172.23.121.10 **** 2019-09-19 01:35:38,937 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave 2019-09-19 01:35:38,937 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory 2019-09-19 01:35:38,937 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory 2019-09-19 01:35:38,937 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2019-09-19 01:35:38,937 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2019-09-19 01:35:38,937 - root - INFO - running command.raw on 172.23.121.10: pkill -u couchbase 2019-09-19 01:35:38,950 - root - INFO - command executed successfully 2019-09-19 01:35:38,951 - root - INFO - root 28532 28350 0 01:35 ? 00:00:00 bash -c ps -ef | grep couchbase 2019-09-19 01:35:38,951 - root - INFO - root 28539 28532 0 01:35 ? 00:00:00 grep couchbase 2019-09-19 01:35:39,005 - root - INFO - command executed successfully 2019-09-19 01:35:39,005 - root - INFO - running command.raw on 172.23.121.10: ps -ef | grep couchbase 2019-09-19 01:35:39,102 - root - INFO - command executed successfully 2019-09-19 01:35:39,103 - root - INFO - root 30341 30175 0 01:35 ? 00:00:00 bash -c ps -ef | grep couchbase 2019-09-19 01:35:39,103 - root - INFO - root 30347 30341 0 01:35 ? 00:00:00 grep couchbase 2019-09-19 01:35:41,114 - root - INFO - running command.raw on 172.23.98.196: ipcrm 2019-09-19 01:35:41,212 - root - INFO - command executed successfully 2019-09-19 01:35:41,221 - root - INFO - running command.raw on 172.23.98.195: ipcrm 2019-09-19 01:35:41,271 - root - INFO - command executed successfully thread uninstaller-thread-172.23.98.196 finished 2019-09-19 01:35:41,603 - root - INFO - running command.raw on 172.23.121.10: ipcrm 2019-09-19 01:35:41,669 - root - INFO - command executed successfully 2019-09-19 01:37:06,918 - root - INFO - command executed successfully 2019-09-19 01:37:06,919 - root - ERROR - **** Uninstalling on server: 172.23.104.186 **** 2019-09-19 01:37:06,919 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave 2019-09-19 01:37:06,919 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory 2019-09-19 01:37:06,919 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory 2019-09-19 01:37:06,919 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2019-09-19 01:37:06,919 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2019-09-19 01:37:06,919 - root - INFO - running command.raw on 172.23.104.186: pkill -u couchbase 2019-09-19 01:37:06,988 - root - INFO - command executed successfully 2019-09-19 01:37:06,988 - root - INFO - running command.raw on 172.23.104.186: ps -ef | grep couchbase 2019-09-19 01:37:07,085 - root - INFO - command executed successfully 2019-09-19 01:37:07,086 - root - INFO - root 29892 29574 0 01:37 ? 00:00:00 bash -c ps -ef | grep couchbase 2019-09-19 01:37:07,086 - root - INFO - root 29898 29892 0 01:37 ? 00:00:00 grep couchbase 2019-09-19 01:37:09,593 - root - INFO - running command.raw on 172.23.104.186: ipcrm 2019-09-19 01:37:09,643 - root - INFO - command executed successfully 2019-09-19 01:37:10,548 - root - INFO - command executed successfully 2019-09-19 01:37:10,548 - root - ERROR - **** Uninstalling on server: 172.23.120.206 **** 2019-09-19 01:37:10,548 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave 2019-09-19 01:37:10,548 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory 2019-09-19 01:37:10,548 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory 2019-09-19 01:37:10,548 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2019-09-19 01:37:10,549 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2019-09-19 01:37:10,549 - root - INFO - running command.raw on 172.23.120.206: pkill -u couchbase 2019-09-19 01:37:10,583 - root - INFO - command executed successfully 2019-09-19 01:37:10,583 - root - INFO - running command.raw on 172.23.120.206: ps -ef | grep couchbase 2019-09-19 01:37:10,631 - root - INFO - command executed successfully 2019-09-19 01:37:10,632 - root - INFO - root 1064 1 0 Sep18 ? 00:00:34 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver overlay2 --insecure-registry sdk-s435.sc.couchbase.com 2019-09-19 01:37:10,632 - root - INFO - root 1677 1260 0 01:37 ? 00:00:00 bash -c ps -ef | grep couchbase 2019-09-19 01:37:10,632 - root - INFO - root 1683 1677 0 01:37 ? 00:00:00 grep couchbase 2019-09-19 01:37:12,631 - root - INFO - running command.raw on 172.23.120.206: ipcrm 2019-09-19 01:37:12,712 - root - INFO - command executed successfully thread uninstaller-thread-172.23.120.206 finished thread uninstaller-thread-172.23.98.195 finished thread uninstaller-thread-172.23.121.10 finished thread uninstaller-thread-172.23.104.186 finished thread uninstaller-thread-172.23.120.201 finished 2019-09-19 01:37:12,827 - root - INFO - connecting to 172.23.98.196 with username:root 2019-09-19 01:37:12,962 - root - INFO - Connected to 172.23.98.196 2019-09-19 01:37:13,415 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:14,057 - root - INFO - connecting to 172.23.120.206 with username:root 2019-09-19 01:37:14,138 - root - INFO - Connected to 172.23.120.206 2019-09-19 01:37:14,235 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:14,822 - root - INFO - connecting to 172.23.98.195 with username:root 2019-09-19 01:37:14,988 - root - INFO - Connected to 172.23.98.195 2019-09-19 01:37:15,466 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:16,089 - root - INFO - connecting to 172.23.121.10 with username:root 2019-09-19 01:37:16,255 - root - INFO - Connected to 172.23.121.10 2019-09-19 01:37:16,668 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:17,307 - root - INFO - connecting to 172.23.104.186 with username:root 2019-09-19 01:37:17,473 - root - INFO - Connected to 172.23.104.186 2019-09-19 01:37:18,081 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:18,681 - root - INFO - connecting to 172.23.120.201 with username:root 2019-09-19 01:37:18,778 - root - INFO - Connected to 172.23.120.201 2019-09-19 01:37:18,925 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:19,530 - root - INFO - ********CouchbaseServerInstaller:install 2019-09-19 01:37:19,531 - root - INFO - ********CouchbaseServerInstaller:install 2019-09-19 01:37:19,532 - root - INFO - ********CouchbaseServerInstaller:install 2019-09-19 01:37:19,533 - root - INFO - ********CouchbaseServerInstaller:install 2019-09-19 01:37:19,535 - root - INFO - ********CouchbaseServerInstaller:install 2019-09-19 01:37:19,538 - root - INFO - ********CouchbaseServerInstaller:install 2019-09-19 01:37:19,540 - root - INFO - connecting to 172.23.98.196 with username:root 2019-09-19 01:37:19,541 - root - INFO - connecting to 172.23.120.206 with username:root 2019-09-19 01:37:19,546 - root - INFO - connecting to 172.23.98.195 with username:root 2019-09-19 01:37:19,553 - root - INFO - connecting to 172.23.121.10 with username:root 2019-09-19 01:37:19,555 - root - INFO - connecting to 172.23.104.186 with username:root 2019-09-19 01:37:19,556 - root - INFO - connecting to 172.23.120.201 with username:root 2019-09-19 01:37:19,638 - root - INFO - Connected to 172.23.120.201 2019-09-19 01:37:19,678 - root - INFO - Connected to 172.23.120.206 2019-09-19 01:37:19,682 - root - INFO - Connected to 172.23.98.196 2019-09-19 01:37:19,714 - root - INFO - Connected to 172.23.98.195 2019-09-19 01:37:19,721 - root - INFO - Connected to 172.23.121.10 2019-09-19 01:37:19,724 - root - INFO - Connected to 172.23.104.186 2019-09-19 01:37:19,777 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:19,848 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:20,112 - root - INFO - It could be other operating system. Go to check at other location *** OS version of this server 172.23.120.206 is CentOS 7 *** 2019-09-19 01:37:20,217 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:20,220 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:20,227 - root - INFO - connecting to 172.23.120.206 with username:root 2019-09-19 01:37:20,264 - root - INFO - It could be other operating system. Go to check at other location *** OS version of this server 172.23.120.201 is CentOS 7 *** 2019-09-19 01:37:20,332 - root - INFO - connecting to 172.23.120.201 with username:root 2019-09-19 01:37:20,359 - root - INFO - Connected to 172.23.120.206 2019-09-19 01:37:20,413 - root - INFO - Connected to 172.23.120.201 2019-09-19 01:37:20,457 - root - INFO - It could be other operating system. Go to check at other location *** OS version of this server 172.23.121.10 is CentOS 7 *** 2019-09-19 01:37:20,618 - root - INFO - connecting to 172.23.121.10 with username:root 2019-09-19 01:37:20,628 - root - INFO - It could be other operating system. Go to check at other location *** OS version of this server 172.23.98.196 is CentOS 7 *** *** OS version of this server 172.23.98.195 is CentOS 7 *** 2019-09-19 01:37:20,691 - root - INFO - connecting to 172.23.98.196 with username:root *** OS version of this server 172.23.104.186 is CentOS 7 *** 2019-09-19 01:37:20,726 - root - INFO - connecting to 172.23.98.195 with username:root 2019-09-19 01:37:20,739 - root - INFO - connecting to 172.23.104.186 with username:root 2019-09-19 01:37:20,784 - root - INFO - Connected to 172.23.121.10 2019-09-19 01:37:20,824 - root - INFO - Connected to 172.23.98.196 2019-09-19 01:37:20,859 - root - INFO - Connected to 172.23.98.195 2019-09-19 01:37:20,904 - root - INFO - Connected to 172.23.104.186 2019-09-19 01:37:21,028 - root - INFO - This url http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm is live 2019-09-19 01:37:21,048 - root - INFO - This url http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm is live 2019-09-19 01:37:21,061 - root - INFO - connecting to 172.23.120.206 with username:root 2019-09-19 01:37:21,114 - root - INFO - connecting to 172.23.120.201 with username:root 2019-09-19 01:37:21,142 - root - INFO - Connected to 172.23.120.206 2019-09-19 01:37:21,185 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:21,231 - root - INFO - Connected to 172.23.120.201 2019-09-19 01:37:21,242 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:21,309 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:21,399 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:21,400 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:21,460 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:21,674 - root - INFO - This url http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm is live 2019-09-19 01:37:21,749 - root - INFO - This url http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm is live 2019-09-19 01:37:21,790 - root - INFO - connecting to 172.23.121.10 with username:root 2019-09-19 01:37:21,816 - root - INFO - connecting to 172.23.98.195 with username:root 2019-09-19 01:37:21,859 - root - INFO - This url http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm is live 2019-09-19 01:37:21,867 - root - INFO - Connect to node: 172.23.120.206 as user: root 2019-09-19 01:37:21,892 - root - INFO - connecting to 172.23.98.196 with username:root 2019-09-19 01:37:21,898 - root - INFO - This url http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm is live 2019-09-19 01:37:21,931 - root - INFO - connecting to 172.23.104.186 with username:root 2019-09-19 01:37:21,947 - root - INFO - Connected to 172.23.120.206 as root 2019-09-19 01:37:21,951 - root - INFO - Connected to 172.23.98.195 2019-09-19 01:37:21,955 - root - INFO - Connected to 172.23.121.10 2019-09-19 01:37:21,982 - root - INFO - running command.raw on 172.23.120.206: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm;cd /tmp;ls -lh 2019-09-19 01:37:21,987 - root - INFO - Connect to node: 172.23.120.201 as user: root 2019-09-19 01:37:22,029 - root - INFO - Connected to 172.23.98.196 2019-09-19 01:37:22,097 - root - INFO - Connected to 172.23.104.186 2019-09-19 01:37:22,104 - root - INFO - Connected to 172.23.120.201 as root 2019-09-19 01:37:22,286 - root - INFO - running command.raw on 172.23.120.201: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm;cd /tmp;ls -lh 2019-09-19 01:37:22,370 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:22,450 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:22,610 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:22,635 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:37:22,966 - root - INFO - Connect to node: 172.23.121.10 as user: root 2019-09-19 01:37:23,010 - root - INFO - Connect to node: 172.23.98.195 as user: root 2019-09-19 01:37:23,131 - root - INFO - Connected to 172.23.121.10 as root 2019-09-19 01:37:23,143 - root - INFO - Connected to 172.23.98.195 as root 2019-09-19 01:37:23,223 - root - INFO - Connect to node: 172.23.104.186 as user: root 2019-09-19 01:37:23,230 - root - INFO - Connect to node: 172.23.98.196 as user: root 2019-09-19 01:37:23,363 - root - INFO - Connected to 172.23.98.196 as root 2019-09-19 01:37:23,389 - root - INFO - Connected to 172.23.104.186 as root 2019-09-19 01:37:23,566 - root - INFO - running command.raw on 172.23.121.10: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm;cd /tmp;ls -lh 2019-09-19 01:37:23,615 - root - INFO - running command.raw on 172.23.98.195: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm;cd /tmp;ls -lh 2019-09-19 01:37:23,862 - root - INFO - running command.raw on 172.23.104.186: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm;cd /tmp;ls -lh 2019-09-19 01:37:24,036 - root - INFO - running command.raw on 172.23.98.196: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/2879/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm;cd /tmp;ls -lh 2019-09-19 01:37:43,248 - root - INFO - command executed successfully 2019-09-19 01:37:43,248 - root - INFO - total 354M 2019-09-19 01:37:43,248 - root - INFO - -rw-r--r--. 1 root root 354M Sep 18 05:26 couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:37:43,248 - root - INFO - drwx------. 3 root root 16 Sep 16 05:59 systemd-private-1a943a2ecb854315a3e75650c6b8a494-ntpd.service-ZP2SsM 2019-09-19 01:37:43,248 - root - INFO - drwx------. 2 root root 4.0K Sep 15 23:56 tmpvhjtl8xr 2019-09-19 01:37:43,342 - root - INFO - *****install server *** 2019-09-19 01:37:43,342 - root - INFO - deliverable_type : rpm 2019-09-19 01:37:43,343 - root - INFO - /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm or /tmp/ 2019-09-19 01:37:43,343 - root - INFO - running command.raw on 172.23.98.196: /sbin/sysctl vm.swappiness=0 2019-09-19 01:37:43,377 - root - INFO - command executed successfully 2019-09-19 01:37:43,377 - root - INFO - vm.swappiness = 0 2019-09-19 01:37:43,378 - root - INFO - running command.raw on 172.23.98.196: cat /etc/redhat-release 2019-09-19 01:37:43,443 - root - INFO - command executed successfully 2019-09-19 01:37:43,443 - root - INFO - CentOS Linux release 7.1.1503 (Core) 2019-09-19 01:37:43,443 - root - INFO - no need to install pkgconfig 2019-09-19 01:37:43,443 - root - INFO - running command.raw on 172.23.98.196: yes | yum localinstall -y /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:37:48,364 - root - INFO - command executed successfully 2019-09-19 01:37:48,364 - root - INFO - total 354M 2019-09-19 01:37:48,364 - root - INFO - drwx------ 2 root root 22 Sep 12 16:47 atop.d 2019-09-19 01:37:48,364 - root - INFO - -rw-r--r-- 1 root root 354M Sep 18 05:26 couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:37:48,364 - root - INFO - drwx------ 3 root root 16 Sep 18 07:33 systemd-private-8ed3b86f3ebb48ea8c9e487eae5342e3-ntpd.service-aCH1YV 2019-09-19 01:37:48,365 - root - INFO - drwx------ 2 root root 6 Sep 19 01:37 tmp.0I7tpmu8BV 2019-09-19 01:37:48,365 - root - INFO - drwx------ 2 root root 6 Sep 15 23:05 tmp.0Z2zOFqvLY 2019-09-19 01:37:48,365 - root - INFO - drwx------ 2 root root 6 Sep 15 16:55 tmp.1iQIP6GsUf 2019-09-19 01:37:48,365 - root - INFO - drwx------ 2 root root 6 Sep 12 16:46 tmp.48BboGbimO 2019-09-19 01:37:48,365 - root - INFO - drwx------ 2 root root 6 Sep 16 03:25 tmp.5U77RBYwzI 2019-09-19 01:37:48,365 - root - INFO - drwx------ 2 root root 6 Sep 12 02:13 tmp.6gvePKMDys 2019-09-19 01:37:48,365 - root - INFO - drwx------ 2 root root 6 Sep 15 20:33 tmp.6Xo5dNKGWe 2019-09-19 01:37:48,366 - root - INFO - drwx------ 2 root root 6 Sep 15 15:50 tmp.Anbv0Db9Qv 2019-09-19 01:37:48,366 - root - INFO - drwx------ 2 root root 6 Sep 15 23:17 tmp.AUSC19bBEn 2019-09-19 01:37:48,366 - root - INFO - drwx------ 2 root root 6 Sep 15 21:50 tmp.biJkvnceDq 2019-09-19 01:37:48,366 - root - INFO - drwx------ 2 root root 6 Sep 15 18:12 tmp.dgPDHyDVJc 2019-09-19 01:37:48,366 - root - INFO - drwx------ 2 root root 6 Sep 14 00:56 tmp.EC37FQHq4H 2019-09-19 01:37:48,366 - root - INFO - drwx------ 2 root root 6 Sep 15 17:20 tmp.eecExbnf1h 2019-09-19 01:37:48,366 - root - INFO - drwx------ 2 root root 6 Sep 15 16:20 tmp.gzhRWhMn0m 2019-09-19 01:37:48,367 - root - INFO - drwx------ 2 root root 6 Sep 18 07:33 tmp.hMQdSyVYOx 2019-09-19 01:37:48,367 - root - INFO - drwx------ 2 root root 6 Sep 15 16:02 tmp.I2JLr6F0ho 2019-09-19 01:37:48,367 - root - INFO - drwx------ 2 root root 6 Sep 15 19:37 tmp.kUi19jbzeB 2019-09-19 01:37:48,367 - root - INFO - drwx------ 2 root root 6 Sep 18 06:25 tmp.lqpb2IxYZe 2019-09-19 01:37:48,367 - root - INFO - drwx------ 2 root root 6 Sep 11 17:01 tmp.lZzfTxf8S8 2019-09-19 01:37:48,367 - root - INFO - drwx------ 2 root root 6 Sep 15 21:17 tmp.m0ODulP6bK 2019-09-19 01:37:48,367 - root - INFO - drwx------ 2 root root 6 Sep 11 19:53 tmp.M8AZ8B7qFO 2019-09-19 01:37:48,368 - root - INFO - drwx------ 2 root root 6 Sep 15 17:04 tmp.mAzQfSHLVX 2019-09-19 01:37:48,368 - root - INFO - drwx------ 2 root root 6 Sep 15 21:06 tmp.MJAfxSbqxh 2019-09-19 01:37:48,368 - root - INFO - drwx------ 2 root root 6 Sep 15 20:55 tmp.NDaKnTaDZx 2019-09-19 01:37:48,368 - root - INFO - drwx------ 2 root root 6 Sep 15 17:37 tmp.nOO8rengTD 2019-09-19 01:37:48,368 - root - INFO - drwx------ 2 root root 6 Sep 15 20:44 tmp.OaoJBU7DTA 2019-09-19 01:37:48,368 - root - INFO - drwx------ 2 root root 6 Sep 15 19:59 tmp.pK1N5OLWEU 2019-09-19 01:37:48,368 - root - INFO - drwx------ 2 root root 6 Sep 15 17:29 tmp.pzbnt3xtha 2019-09-19 01:37:48,369 - root - INFO - drwx------ 2 root root 6 Sep 18 06:46 tmp.qfzVYgs4dc 2019-09-19 01:37:48,369 - root - INFO - drwx------ 2 root root 6 Sep 15 17:56 tmp.RhrqoON9Qg 2019-09-19 01:37:48,369 - root - INFO - drwx------ 2 root root 6 Sep 15 15:11 tmp.sqUpDZaMPa 2019-09-19 01:37:48,369 - root - INFO - drwx------ 2 root root 6 Sep 18 07:01 tmp.suKd05V5CS 2019-09-19 01:37:48,369 - root - INFO - drwx------ 2 root root 6 Sep 11 02:45 tmp.tBqAshRHyu 2019-09-19 01:37:48,369 - root - INFO - drwx------ 2 root root 6 Sep 15 20:21 tmp.tegOsJ3lZh 2019-09-19 01:37:48,369 - root - INFO - drwx------ 2 root root 6 Sep 15 22:01 tmp.THtBYnQ5gG 2019-09-19 01:37:48,370 - root - INFO - drwx------ 2 root root 6 Sep 18 06:33 tmp.UOJamISsRz 2019-09-19 01:37:48,370 - root - INFO - drwx------ 2 root root 6 Sep 15 21:39 tmp.UrCdZ3w7Pu 2019-09-19 01:37:48,370 - root - INFO - drwx------ 2 root root 6 Sep 11 19:14 tmp.UWp9Wwia7S 2019-09-19 01:37:48,370 - root - INFO - drwx------ 2 root root 6 Sep 15 19:48 tmp.V7UrZZYKDe 2019-09-19 01:37:48,370 - root - INFO - drwx------ 2 root root 6 Sep 11 01:36 tmp.VBEBgQBFTi 2019-09-19 01:37:48,370 - root - INFO - drwx------ 2 root root 6 Sep 12 15:41 tmp.vFUOK4JzpV 2019-09-19 01:37:48,370 - root - INFO - drwx------ 2 root root 6 Sep 15 15:32 tmp.VSK9sgezri 2019-09-19 01:37:48,371 - root - INFO - drwx------ 2 root root 6 Sep 11 15:58 tmp.WjT7qlfDow 2019-09-19 01:37:48,371 - root - INFO - drwx------ 2 root root 6 Sep 15 23:56 tmp.X1bZ0CxciJ 2019-09-19 01:37:48,371 - root - INFO - drwx------ 2 root root 6 Sep 15 16:38 tmp.yMH64Q107A 2019-09-19 01:37:48,371 - root - INFO - drwx------ 2 root root 6 Sep 15 15:23 tmp.ZELiZzVKJQ 2019-09-19 01:37:48,371 - root - INFO - drwx------ 2 root root 6 Sep 15 17:12 tmp.ZsXhlBHUV7 2019-09-19 01:37:48,371 - root - INFO - drwx------ 2 root root 6 Sep 15 18:04 tmp.zwwZWoGAch 2019-09-19 01:37:48,419 - root - INFO - *****install server *** 2019-09-19 01:37:48,419 - root - INFO - deliverable_type : rpm 2019-09-19 01:37:48,419 - root - INFO - /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm or /tmp/ 2019-09-19 01:37:48,419 - root - INFO - running command.raw on 172.23.120.206: /sbin/sysctl vm.swappiness=0 2019-09-19 01:37:48,501 - root - INFO - command executed successfully 2019-09-19 01:37:48,501 - root - INFO - vm.swappiness = 0 2019-09-19 01:37:48,501 - root - INFO - running command.raw on 172.23.120.206: cat /etc/redhat-release 2019-09-19 01:37:48,574 - root - INFO - command executed successfully 2019-09-19 01:37:48,574 - root - INFO - CentOS Linux release 7.2.1511 (Core) 2019-09-19 01:37:48,574 - root - INFO - no need to install pkgconfig 2019-09-19 01:37:48,574 - root - INFO - running command.raw on 172.23.120.206: yes | yum localinstall -y /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:37:55,276 - root - INFO - command executed successfully 2019-09-19 01:37:55,276 - root - INFO - total 354M 2019-09-19 01:37:55,277 - root - INFO - -rw-r--r--. 1 root root 354M Sep 18 05:26 couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:37:55,277 - root - INFO - drwx------. 3 root root 16 Sep 7 06:29 systemd-private-ba0ef561bd4543bca5e6b458f002a7cc-ntpd.service-TIHPFn 2019-09-19 01:37:55,310 - root - INFO - *****install server *** 2019-09-19 01:37:55,310 - root - INFO - deliverable_type : rpm 2019-09-19 01:37:55,311 - root - INFO - /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm or /tmp/ 2019-09-19 01:37:55,311 - root - INFO - running command.raw on 172.23.121.10: /sbin/sysctl vm.swappiness=0 2019-09-19 01:37:55,409 - root - INFO - command executed successfully 2019-09-19 01:37:55,409 - root - INFO - vm.swappiness = 0 2019-09-19 01:37:55,409 - root - INFO - running command.raw on 172.23.121.10: cat /etc/redhat-release 2019-09-19 01:37:55,507 - root - INFO - command executed successfully 2019-09-19 01:37:55,507 - root - INFO - CentOS Linux release 7.6.1810 (Core) 2019-09-19 01:37:55,507 - root - INFO - no need to install pkgconfig 2019-09-19 01:37:55,507 - root - INFO - running command.raw on 172.23.121.10: yes | yum localinstall -y /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:37:59,013 - root - INFO - command executed successfully 2019-09-19 01:37:59,013 - root - INFO - total 354M 2019-09-19 01:37:59,013 - root - INFO - -rw-r--r--. 1 root root 354M Sep 18 05:26 couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:37:59,013 - root - INFO - drwxr-xr-x. 2 root root 6 Sep 11 23:54 hsperfdata_root 2019-09-19 01:37:59,013 - root - INFO - drwxr-xr-x. 3 root root 17 Sep 8 22:28 ignite 2019-09-19 01:37:59,014 - root - INFO - drwxr-xr-x. 2 root root 6 Sep 9 09:45 jna-3506402 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 8 22:32 miniSolrCloudCluster1340791392436158511 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 9 03:22 miniSolrCloudCluster1685071661261395436 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 9 09:47 miniSolrCloudCluster4063534636374041650 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 9 03:21 miniSolrCloudCluster4608310840660905038 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 9 09:47 miniSolrCloudCluster4993775148971301538 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 9 09:46 miniSolrCloudCluster5035235467368332788 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 9 03:21 miniSolrCloudCluster5180133124933440629 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 8 22:33 miniSolrCloudCluster569370505899734597 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 9 03:20 miniSolrCloudCluster6238068195924560504 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 8 22:32 miniSolrCloudCluster6384939698091619988 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 8 22:31 miniSolrCloudCluster848465179683809309 2019-09-19 01:37:59,014 - root - INFO - drwx------. 4 root root 34 Sep 9 09:46 miniSolrCloudCluster8764639386207908750 2019-09-19 01:37:59,015 - root - INFO - drwx------. 3 root root 16 Sep 7 17:44 systemd-private-23d6f4ebcf3d4ed9bbc0ee01b14cef9b-httpd.service-fFgHRZ 2019-09-19 01:37:59,015 - root - INFO - drwx------. 3 root root 16 Sep 7 17:44 systemd-private-23d6f4ebcf3d4ed9bbc0ee01b14cef9b-ntpd.service-XZCmqH 2019-09-19 01:37:59,050 - root - INFO - *****install server *** 2019-09-19 01:37:59,050 - root - INFO - deliverable_type : rpm 2019-09-19 01:37:59,050 - root - INFO - /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm or /tmp/ 2019-09-19 01:37:59,050 - root - INFO - running command.raw on 172.23.98.195: /sbin/sysctl vm.swappiness=0 2019-09-19 01:37:59,084 - root - INFO - command executed successfully 2019-09-19 01:37:59,084 - root - INFO - vm.swappiness = 0 2019-09-19 01:37:59,085 - root - INFO - running command.raw on 172.23.98.195: cat /etc/redhat-release 2019-09-19 01:37:59,150 - root - INFO - command executed successfully 2019-09-19 01:37:59,150 - root - INFO - CentOS Linux release 7.1.1503 (Core) 2019-09-19 01:37:59,150 - root - INFO - no need to install pkgconfig 2019-09-19 01:37:59,150 - root - INFO - running command.raw on 172.23.98.195: yes | yum localinstall -y /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:38:04,893 - root - INFO - command executed successfully 2019-09-19 01:38:04,893 - root - INFO - total 354M 2019-09-19 01:38:04,894 - root - INFO - -rw-r--r--. 1 root root 354M Sep 18 05:26 couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:38:04,894 - root - INFO - drwx------. 3 root root 16 Sep 18 07:33 systemd-private-6383ea4cca9749828e6110f911e3eb1b-ntpd.service-9mrRZM 2019-09-19 01:38:04,988 - root - INFO - *****install server *** 2019-09-19 01:38:04,988 - root - INFO - deliverable_type : rpm 2019-09-19 01:38:04,989 - root - INFO - /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm or /tmp/ 2019-09-19 01:38:04,989 - root - INFO - running command.raw on 172.23.104.186: /sbin/sysctl vm.swappiness=0 2019-09-19 01:38:05,023 - root - INFO - command executed successfully 2019-09-19 01:38:05,024 - root - INFO - vm.swappiness = 0 2019-09-19 01:38:05,024 - root - INFO - running command.raw on 172.23.104.186: cat /etc/redhat-release 2019-09-19 01:38:05,122 - root - INFO - command executed successfully 2019-09-19 01:38:05,122 - root - INFO - CentOS Linux release 7.4.1708 (Core) 2019-09-19 01:38:05,122 - root - INFO - no need to install pkgconfig 2019-09-19 01:38:05,122 - root - INFO - running command.raw on 172.23.104.186: yes | yum localinstall -y /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:38:13,459 - root - INFO - command executed successfully 2019-09-19 01:38:13,459 - root - INFO - total 354M 2019-09-19 01:38:13,459 - root - INFO - -rw-r--r--. 1 root root 354M Sep 18 05:26 couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:38:13,460 - root - INFO - drwx------. 2 root root 24 Sep 6 02:38 ssh-quOkyRXFkn4p 2019-09-19 01:38:13,460 - root - INFO - drwx------. 3 root root 16 Aug 6 15:15 systemd-private-53e8b80f9a724a7c828be2bac579dca4-ntpd.service-N0dEcK 2019-09-19 01:38:13,478 - root - INFO - *****install server *** 2019-09-19 01:38:13,478 - root - INFO - deliverable_type : rpm 2019-09-19 01:38:13,478 - root - INFO - /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm or /tmp/ 2019-09-19 01:38:13,478 - root - INFO - running command.raw on 172.23.120.201: /sbin/sysctl vm.swappiness=0 2019-09-19 01:38:13,559 - root - INFO - command executed successfully 2019-09-19 01:38:13,559 - root - INFO - vm.swappiness = 0 2019-09-19 01:38:13,559 - root - INFO - running command.raw on 172.23.120.201: cat /etc/redhat-release 2019-09-19 01:38:13,640 - root - INFO - command executed successfully 2019-09-19 01:38:13,640 - root - INFO - CentOS Linux release 7.2.1511 (Core) 2019-09-19 01:38:13,640 - root - INFO - no need to install pkgconfig 2019-09-19 01:38:13,640 - root - INFO - running command.raw on 172.23.120.201: yes | yum localinstall -y /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm 2019-09-19 01:38:35,202 - root - INFO - command executed successfully 2019-09-19 01:38:35,202 - root - ERROR - Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast 2019-09-19 01:38:35,202 - root - ERROR - Warning: RPMDB altered outside of yum. 2019-09-19 01:38:35,202 - root - INFO - Loaded plugins: fastestmirror 2019-09-19 01:38:35,202 - root - INFO - Examining /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm: couchbase-server-6.0.3-2879.x86_64 2019-09-19 01:38:35,202 - root - INFO - Marking /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm to be installed 2019-09-19 01:38:35,202 - root - INFO - Resolving Dependencies 2019-09-19 01:38:35,202 - root - INFO - --> Running transaction check 2019-09-19 01:38:35,202 - root - INFO - ---> Package couchbase-server.x86_64 0:6.0.3-2879 will be installed 2019-09-19 01:38:35,203 - root - INFO - --> Finished Dependency Resolution 2019-09-19 01:38:35,203 - root - INFO - 2019-09-19 01:38:35,203 - root - INFO - Dependencies Resolved 2019-09-19 01:38:35,203 - root - INFO - 2019-09-19 01:38:35,203 - root - INFO - ================================================================================ 2019-09-19 01:38:35,203 - root - INFO - Package 2019-09-19 01:38:35,203 - root - INFO - Arch Version 2019-09-19 01:38:35,203 - root - INFO - Repository Size 2019-09-19 01:38:35,203 - root - INFO - ================================================================================ 2019-09-19 01:38:35,203 - root - INFO - Installing: 2019-09-19 01:38:35,203 - root - INFO - couchbase-server 2019-09-19 01:38:35,203 - root - INFO - x86_64 6.0.3-2879 2019-09-19 01:38:35,204 - root - INFO - /couchbase-server-enterprise-6.0.3-2879-centos7.x86_64 762 M 2019-09-19 01:38:35,204 - root - INFO - 2019-09-19 01:38:35,204 - root - INFO - Transaction Summary 2019-09-19 01:38:35,204 - root - INFO - ================================================================================ 2019-09-19 01:38:35,204 - root - INFO - Install 1 Package 2019-09-19 01:38:35,204 - root - INFO - 2019-09-19 01:38:35,204 - root - INFO - Total size: 762 M 2019-09-19 01:38:35,204 - root - INFO - Installed size: 762 M 2019-09-19 01:38:35,204 - root - INFO - Downloading packages: 2019-09-19 01:38:35,204 - root - INFO - Running transaction check 2019-09-19 01:38:35,204 - root - INFO - Running transaction test 2019-09-19 01:38:35,205 - root - INFO - Transaction test succeeded 2019-09-19 01:38:35,205 - root - INFO - Running transaction 2019-09-19 01:38:35,205 - root - INFO - Warning: Transparent hugepages looks to be active and should not be. 2019-09-19 01:38:35,205 - root - INFO - Please look at https://developer.couchbase.com/documentation/server/current/install/thp-disable.html as for how to PERMANENTLY alter this setting. 2019-09-19 01:38:35,205 - root - INFO - Minimum RAM required : 4 GB 2019-09-19 01:38:35,205 - root - INFO - System RAM configured : 3.45 GB 2019-09-19 01:38:35,205 - root - INFO - 2019-09-19 01:38:35,205 - root - INFO - Minimum number of processors required : 4 cores 2019-09-19 01:38:35,205 - root - INFO - Number of processors on the system : 4 cores 2019-09-19 01:38:35,205 - root - INFO - 2019-09-19 01:38:35,206 - root - INFO - Installing : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:38:35,206 - root - INFO - 2019-09-19 01:38:35,206 - root - INFO - You have successfully installed Couchbase Server. 2019-09-19 01:38:35,206 - root - INFO - Please browse to http://localhost.localdomain:8091/ to configure your server. 2019-09-19 01:38:35,206 - root - INFO - Please refer to http://couchbase.com for additional resources. 2019-09-19 01:38:35,206 - root - INFO - 2019-09-19 01:38:35,206 - root - INFO - Please note that you have to update your firewall configuration to 2019-09-19 01:38:35,206 - root - INFO - allow connections to the following ports: 2019-09-19 01:38:35,206 - root - INFO - 4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211, 2019-09-19 01:38:35,206 - root - INFO - 11214, 11215, 18091 to 18093, and from 21100 to 21299. 2019-09-19 01:38:35,207 - root - INFO - 2019-09-19 01:38:35,207 - root - INFO - By using this software you agree to the End User License Agreement. 2019-09-19 01:38:35,207 - root - INFO - See /opt/couchbase/LICENSE.txt. 2019-09-19 01:38:35,207 - root - INFO - 2019-09-19 01:38:35,207 - root - INFO - Verifying : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:38:35,207 - root - INFO - 2019-09-19 01:38:35,207 - root - INFO - Installed: 2019-09-19 01:38:35,207 - root - INFO - couchbase-server.x86_64 0:6.0.3-2879 2019-09-19 01:38:35,207 - root - INFO - 2019-09-19 01:38:35,207 - root - INFO - Complete! 2019-09-19 01:38:35,207 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2019-09-19 01:38:35,207 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2019-09-19 01:38:35,208 - root - INFO - Warming: RPMDB altered outside of yum 2019-09-19 01:38:35,208 - root - INFO - There is a warning about transparent_hugepage may be in used when install cb server. So we will disable transparent_hugepage in this vm 2019-09-19 01:38:35,208 - root - INFO - running command.raw on 172.23.120.206: echo never > /sys/kernel/mm/transparent_hugepage/enabled 2019-09-19 01:38:35,248 - root - INFO - command executed successfully 2019-09-19 01:38:35,323 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2019-09-19 01:38:35,444 - root - INFO - Running systemd command on this server 2019-09-19 01:38:35,444 - root - INFO - running command.raw on 172.23.120.206: systemctl start couchbase-server.service 2019-09-19 01:38:35,462 - root - INFO - command executed successfully 2019-09-19 01:38:35,543 - root - INFO - wait 5 seconds for Couchbase server to start 2019-09-19 01:39:00,873 - root - INFO - command executed successfully 2019-09-19 01:39:00,873 - root - ERROR - Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast 2019-09-19 01:39:00,873 - root - ERROR - Warning: RPMDB altered outside of yum. 2019-09-19 01:39:00,873 - root - INFO - Loaded plugins: fastestmirror 2019-09-19 01:39:00,873 - root - INFO - Examining /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm: couchbase-server-6.0.3-2879.x86_64 2019-09-19 01:39:00,874 - root - INFO - Marking /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm to be installed 2019-09-19 01:39:00,874 - root - INFO - Resolving Dependencies 2019-09-19 01:39:00,874 - root - INFO - --> Running transaction check 2019-09-19 01:39:00,874 - root - INFO - ---> Package couchbase-server.x86_64 0:6.0.3-2879 will be installed 2019-09-19 01:39:00,874 - root - INFO - --> Finished Dependency Resolution 2019-09-19 01:39:00,874 - root - INFO - 2019-09-19 01:39:00,874 - root - INFO - Dependencies Resolved 2019-09-19 01:39:00,874 - root - INFO - 2019-09-19 01:39:00,874 - root - INFO - ================================================================================ 2019-09-19 01:39:00,874 - root - INFO - Package 2019-09-19 01:39:00,875 - root - INFO - Arch Version 2019-09-19 01:39:00,875 - root - INFO - Repository Size 2019-09-19 01:39:00,875 - root - INFO - ================================================================================ 2019-09-19 01:39:00,875 - root - INFO - Installing: 2019-09-19 01:39:00,875 - root - INFO - couchbase-server 2019-09-19 01:39:00,875 - root - INFO - x86_64 6.0.3-2879 2019-09-19 01:39:00,875 - root - INFO - /couchbase-server-enterprise-6.0.3-2879-centos7.x86_64 762 M 2019-09-19 01:39:00,875 - root - INFO - 2019-09-19 01:39:00,875 - root - INFO - Transaction Summary 2019-09-19 01:39:00,875 - root - INFO - ================================================================================ 2019-09-19 01:39:00,875 - root - INFO - Install 1 Package 2019-09-19 01:39:00,876 - root - INFO - 2019-09-19 01:39:00,876 - root - INFO - Total size: 762 M 2019-09-19 01:39:00,876 - root - INFO - Installed size: 762 M 2019-09-19 01:39:00,876 - root - INFO - Downloading packages: 2019-09-19 01:39:00,876 - root - INFO - Running transaction check 2019-09-19 01:39:00,876 - root - INFO - Running transaction test 2019-09-19 01:39:00,876 - root - INFO - Transaction test succeeded 2019-09-19 01:39:00,876 - root - INFO - Running transaction 2019-09-19 01:39:00,876 - root - INFO - Minimum RAM required : 4 GB 2019-09-19 01:39:00,876 - root - INFO - System RAM configured : 3.45 GB 2019-09-19 01:39:00,876 - root - INFO - 2019-09-19 01:39:00,877 - root - INFO - Minimum number of processors required : 4 cores 2019-09-19 01:39:00,877 - root - INFO - Number of processors on the system : 4 cores 2019-09-19 01:39:00,877 - root - INFO - 2019-09-19 01:39:00,877 - root - INFO - Installing : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:39:00,877 - root - INFO - 2019-09-19 01:39:00,877 - root - INFO - You have successfully installed Couchbase Server. 2019-09-19 01:39:00,877 - root - INFO - Please browse to http://localhost.localdomain:8091/ to configure your server. 2019-09-19 01:39:00,877 - root - INFO - Please refer to http://couchbase.com for additional resources. 2019-09-19 01:39:00,877 - root - INFO - 2019-09-19 01:39:00,877 - root - INFO - Please note that you have to update your firewall configuration to 2019-09-19 01:39:00,877 - root - INFO - allow connections to the following ports: 2019-09-19 01:39:00,877 - root - INFO - 4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211, 2019-09-19 01:39:00,877 - root - INFO - 11214, 11215, 18091 to 18093, and from 21100 to 21299. 2019-09-19 01:39:00,878 - root - INFO - 2019-09-19 01:39:00,878 - root - INFO - By using this software you agree to the End User License Agreement. 2019-09-19 01:39:00,878 - root - INFO - See /opt/couchbase/LICENSE.txt. 2019-09-19 01:39:00,878 - root - INFO - 2019-09-19 01:39:00,878 - root - INFO - Verifying : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:39:00,878 - root - INFO - 2019-09-19 01:39:00,878 - root - INFO - Installed: 2019-09-19 01:39:00,878 - root - INFO - couchbase-server.x86_64 0:6.0.3-2879 2019-09-19 01:39:00,878 - root - INFO - 2019-09-19 01:39:00,878 - root - INFO - Complete! 2019-09-19 01:39:00,879 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2019-09-19 01:39:00,879 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2019-09-19 01:39:00,879 - root - INFO - Warming: RPMDB altered outside of yum 2019-09-19 01:39:00,909 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2019-09-19 01:39:01,041 - root - INFO - Running systemd command on this server 2019-09-19 01:39:01,041 - root - INFO - running command.raw on 172.23.120.201: systemctl start couchbase-server.service 2019-09-19 01:39:01,075 - root - INFO - command executed successfully 2019-09-19 01:39:01,124 - root - INFO - wait 5 seconds for Couchbase server to start 2019-09-19 01:39:28,416 - root - INFO - command executed successfully 2019-09-19 01:39:28,416 - root - ERROR - Warning: RPMDB altered outside of yum. 2019-09-19 01:39:28,416 - root - INFO - Loaded plugins: fastestmirror, langpacks 2019-09-19 01:39:28,417 - root - INFO - Examining /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm: couchbase-server-6.0.3-2879.x86_64 2019-09-19 01:39:28,417 - root - INFO - Marking /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm to be installed 2019-09-19 01:39:28,417 - root - INFO - Resolving Dependencies 2019-09-19 01:39:28,417 - root - INFO - --> Running transaction check 2019-09-19 01:39:28,417 - root - INFO - ---> Package couchbase-server.x86_64 0:6.0.3-2879 will be installed 2019-09-19 01:39:28,417 - root - INFO - --> Finished Dependency Resolution 2019-09-19 01:39:28,417 - root - INFO - 2019-09-19 01:39:28,417 - root - INFO - Dependencies Resolved 2019-09-19 01:39:28,417 - root - INFO - 2019-09-19 01:39:28,417 - root - INFO - ================================================================================ 2019-09-19 01:39:28,417 - root - INFO - Package 2019-09-19 01:39:28,417 - root - INFO - Arch Version 2019-09-19 01:39:28,418 - root - INFO - Repository Size 2019-09-19 01:39:28,418 - root - INFO - ================================================================================ 2019-09-19 01:39:28,418 - root - INFO - Installing: 2019-09-19 01:39:28,418 - root - INFO - couchbase-server 2019-09-19 01:39:28,418 - root - INFO - x86_64 6.0.3-2879 2019-09-19 01:39:28,418 - root - INFO - /couchbase-server-enterprise-6.0.3-2879-centos7.x86_64 762 M 2019-09-19 01:39:28,418 - root - INFO - 2019-09-19 01:39:28,418 - root - INFO - Transaction Summary 2019-09-19 01:39:28,418 - root - INFO - ================================================================================ 2019-09-19 01:39:28,418 - root - INFO - Install 1 Package 2019-09-19 01:39:28,418 - root - INFO - 2019-09-19 01:39:28,419 - root - INFO - Total size: 762 M 2019-09-19 01:39:28,419 - root - INFO - Installed size: 762 M 2019-09-19 01:39:28,419 - root - INFO - Downloading packages: 2019-09-19 01:39:28,419 - root - INFO - Running transaction check 2019-09-19 01:39:28,419 - root - INFO - Running transaction test 2019-09-19 01:39:28,419 - root - INFO - Transaction test succeeded 2019-09-19 01:39:28,419 - root - INFO - Running transaction 2019-09-19 01:39:28,420 - root - INFO - Minimum RAM required : 4 GB 2019-09-19 01:39:28,420 - root - INFO - System RAM configured : 3.45 GB 2019-09-19 01:39:28,420 - root - INFO - 2019-09-19 01:39:28,420 - root - INFO - Minimum number of processors required : 4 cores 2019-09-19 01:39:28,420 - root - INFO - Number of processors on the system : 4 cores 2019-09-19 01:39:28,420 - root - INFO - 2019-09-19 01:39:28,420 - root - INFO - Installing : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:39:28,421 - root - INFO - 2019-09-19 01:39:28,421 - root - INFO - You have successfully installed Couchbase Server. 2019-09-19 01:39:28,421 - root - INFO - Please browse to http://s81705:8091/ to configure your server. 2019-09-19 01:39:28,421 - root - INFO - Please refer to http://couchbase.com for additional resources. 2019-09-19 01:39:28,421 - root - INFO - 2019-09-19 01:39:28,421 - root - INFO - Please note that you have to update your firewall configuration to 2019-09-19 01:39:28,421 - root - INFO - allow connections to the following ports: 2019-09-19 01:39:28,421 - root - INFO - 4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211, 2019-09-19 01:39:28,422 - root - INFO - 11214, 11215, 18091 to 18093, and from 21100 to 21299. 2019-09-19 01:39:28,422 - root - INFO - 2019-09-19 01:39:28,422 - root - INFO - By using this software you agree to the End User License Agreement. 2019-09-19 01:39:28,422 - root - INFO - See /opt/couchbase/LICENSE.txt. 2019-09-19 01:39:28,422 - root - INFO - 2019-09-19 01:39:28,422 - root - INFO - Verifying : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:39:28,422 - root - INFO - 2019-09-19 01:39:28,423 - root - INFO - Installed: 2019-09-19 01:39:28,423 - root - INFO - couchbase-server.x86_64 0:6.0.3-2879 2019-09-19 01:39:28,423 - root - INFO - 2019-09-19 01:39:28,423 - root - INFO - Complete! 2019-09-19 01:39:28,423 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2019-09-19 01:39:28,423 - root - INFO - Warming: RPMDB altered outside of yum 2019-09-19 01:39:28,486 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2019-09-19 01:39:28,636 - root - INFO - Running systemd command on this server 2019-09-19 01:39:28,636 - root - INFO - running command.raw on 172.23.98.195: systemctl start couchbase-server.service 2019-09-19 01:39:28,703 - root - INFO - command executed successfully 2019-09-19 01:39:28,738 - root - INFO - wait 5 seconds for Couchbase server to start 2019-09-19 01:39:33,651 - root - INFO - command executed successfully 2019-09-19 01:39:33,652 - root - ERROR - Warning: RPMDB altered outside of yum. 2019-09-19 01:39:33,652 - root - INFO - Loaded plugins: fastestmirror 2019-09-19 01:39:33,652 - root - INFO - Examining /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm: couchbase-server-6.0.3-2879.x86_64 2019-09-19 01:39:33,652 - root - INFO - Marking /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm to be installed 2019-09-19 01:39:33,652 - root - INFO - Resolving Dependencies 2019-09-19 01:39:33,652 - root - INFO - --> Running transaction check 2019-09-19 01:39:33,652 - root - INFO - ---> Package couchbase-server.x86_64 0:6.0.3-2879 will be installed 2019-09-19 01:39:33,652 - root - INFO - --> Finished Dependency Resolution 2019-09-19 01:39:33,652 - root - INFO - 2019-09-19 01:39:33,652 - root - INFO - Dependencies Resolved 2019-09-19 01:39:33,653 - root - INFO - 2019-09-19 01:39:33,653 - root - INFO - ================================================================================ 2019-09-19 01:39:33,653 - root - INFO - Package 2019-09-19 01:39:33,653 - root - INFO - Arch Version 2019-09-19 01:39:33,653 - root - INFO - Repository Size 2019-09-19 01:39:33,653 - root - INFO - ================================================================================ 2019-09-19 01:39:33,653 - root - INFO - Installing: 2019-09-19 01:39:33,653 - root - INFO - couchbase-server 2019-09-19 01:39:33,653 - root - INFO - x86_64 6.0.3-2879 2019-09-19 01:39:33,653 - root - INFO - /couchbase-server-enterprise-6.0.3-2879-centos7.x86_64 762 M 2019-09-19 01:39:33,653 - root - INFO - 2019-09-19 01:39:33,653 - root - INFO - Transaction Summary 2019-09-19 01:39:33,654 - root - INFO - ================================================================================ 2019-09-19 01:39:33,654 - root - INFO - Install 1 Package 2019-09-19 01:39:33,654 - root - INFO - 2019-09-19 01:39:33,654 - root - INFO - Total size: 762 M 2019-09-19 01:39:33,654 - root - INFO - Installed size: 762 M 2019-09-19 01:39:33,654 - root - INFO - Downloading packages: 2019-09-19 01:39:33,654 - root - INFO - Running transaction check 2019-09-19 01:39:33,654 - root - INFO - Running transaction test 2019-09-19 01:39:33,654 - root - INFO - Transaction test succeeded 2019-09-19 01:39:33,654 - root - INFO - Running transaction 2019-09-19 01:39:33,654 - root - INFO - Minimum RAM required : 4 GB 2019-09-19 01:39:33,654 - root - INFO - System RAM configured : 3.93 GB 2019-09-19 01:39:33,655 - root - INFO - 2019-09-19 01:39:33,655 - root - INFO - Minimum number of processors required : 4 cores 2019-09-19 01:39:33,655 - root - INFO - Number of processors on the system : 4 cores 2019-09-19 01:39:33,655 - root - INFO - 2019-09-19 01:39:33,655 - root - INFO - Installing : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:39:33,655 - root - INFO - 2019-09-19 01:39:33,655 - root - INFO - You have successfully installed Couchbase Server. 2019-09-19 01:39:33,655 - root - INFO - Please browse to http://localhost.localdomain:8091/ to configure your server. 2019-09-19 01:39:33,655 - root - INFO - Please refer to http://couchbase.com for additional resources. 2019-09-19 01:39:33,656 - root - INFO - 2019-09-19 01:39:33,656 - root - INFO - Please note that you have to update your firewall configuration to 2019-09-19 01:39:33,656 - root - INFO - allow connections to the following ports: 2019-09-19 01:39:33,656 - root - INFO - 4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211, 2019-09-19 01:39:33,656 - root - INFO - 11214, 11215, 18091 to 18093, and from 21100 to 21299. 2019-09-19 01:39:33,656 - root - INFO - 2019-09-19 01:39:33,656 - root - INFO - By using this software you agree to the End User License Agreement. 2019-09-19 01:39:33,656 - root - INFO - See /opt/couchbase/LICENSE.txt. 2019-09-19 01:39:33,656 - root - INFO - 2019-09-19 01:39:33,656 - root - INFO - Verifying : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:39:33,656 - root - INFO - 2019-09-19 01:39:33,656 - root - INFO - Installed: 2019-09-19 01:39:33,656 - root - INFO - couchbase-server.x86_64 0:6.0.3-2879 2019-09-19 01:39:33,657 - root - INFO - 2019-09-19 01:39:33,657 - root - INFO - Complete! 2019-09-19 01:39:33,657 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2019-09-19 01:39:33,657 - root - INFO - Warming: RPMDB altered outside of yum 2019-09-19 01:39:33,705 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2019-09-19 01:39:33,892 - root - INFO - Running systemd command on this server 2019-09-19 01:39:33,892 - root - INFO - running command.raw on 172.23.121.10: systemctl start couchbase-server.service 2019-09-19 01:39:33,959 - root - INFO - command executed successfully 2019-09-19 01:39:34,009 - root - INFO - wait 5 seconds for Couchbase server to start 2019-09-19 01:39:34,856 - root - INFO - command executed successfully 2019-09-19 01:39:34,856 - root - ERROR - http://mirrors.greenmountainaccess.net/centos/7.7.1908/extras/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to mirrors.greenmountainaccess.net:80; Operation now in progress" 2019-09-19 01:39:34,856 - root - ERROR - Trying other mirror. 2019-09-19 01:39:34,856 - root - ERROR - Warning: RPMDB altered outside of yum. 2019-09-19 01:39:34,856 - root - INFO - Loaded plugins: fastestmirror, langpacks 2019-09-19 01:39:34,857 - root - INFO - Examining /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm: couchbase-server-6.0.3-2879.x86_64 2019-09-19 01:39:34,857 - root - INFO - Marking /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm to be installed 2019-09-19 01:39:34,857 - root - INFO - Resolving Dependencies 2019-09-19 01:39:34,857 - root - INFO - --> Running transaction check 2019-09-19 01:39:34,857 - root - INFO - ---> Package couchbase-server.x86_64 0:6.0.3-2879 will be installed 2019-09-19 01:39:34,857 - root - INFO - --> Finished Dependency Resolution 2019-09-19 01:39:34,857 - root - INFO - 2019-09-19 01:39:34,857 - root - INFO - Dependencies Resolved 2019-09-19 01:39:34,857 - root - INFO - 2019-09-19 01:39:34,858 - root - INFO - ================================================================================ 2019-09-19 01:39:34,858 - root - INFO - Package 2019-09-19 01:39:34,858 - root - INFO - Arch Version 2019-09-19 01:39:34,858 - root - INFO - Repository Size 2019-09-19 01:39:34,858 - root - INFO - ================================================================================ 2019-09-19 01:39:34,858 - root - INFO - Installing: 2019-09-19 01:39:34,858 - root - INFO - couchbase-server 2019-09-19 01:39:34,858 - root - INFO - x86_64 6.0.3-2879 2019-09-19 01:39:34,858 - root - INFO - /couchbase-server-enterprise-6.0.3-2879-centos7.x86_64 762 M 2019-09-19 01:39:34,858 - root - INFO - 2019-09-19 01:39:34,858 - root - INFO - Transaction Summary 2019-09-19 01:39:34,858 - root - INFO - ================================================================================ 2019-09-19 01:39:34,858 - root - INFO - Install 1 Package 2019-09-19 01:39:34,859 - root - INFO - 2019-09-19 01:39:34,859 - root - INFO - Total size: 762 M 2019-09-19 01:39:34,859 - root - INFO - Installed size: 762 M 2019-09-19 01:39:34,859 - root - INFO - Downloading packages: 2019-09-19 01:39:34,859 - root - INFO - Running transaction check 2019-09-19 01:39:34,859 - root - INFO - Running transaction test 2019-09-19 01:39:34,859 - root - INFO - Transaction test succeeded 2019-09-19 01:39:34,859 - root - INFO - Running transaction 2019-09-19 01:39:34,859 - root - INFO - Minimum RAM required : 4 GB 2019-09-19 01:39:34,859 - root - INFO - System RAM configured : 3.45 GB 2019-09-19 01:39:34,859 - root - INFO - 2019-09-19 01:39:34,859 - root - INFO - Minimum number of processors required : 4 cores 2019-09-19 01:39:34,859 - root - INFO - Number of processors on the system : 4 cores 2019-09-19 01:39:34,860 - root - INFO - 2019-09-19 01:39:34,860 - root - INFO - Installing : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:39:34,860 - root - INFO - 2019-09-19 01:39:34,860 - root - INFO - You have successfully installed Couchbase Server. 2019-09-19 01:39:34,860 - root - INFO - Please browse to http://s81706:8091/ to configure your server. 2019-09-19 01:39:34,860 - root - INFO - Please refer to http://couchbase.com for additional resources. 2019-09-19 01:39:34,860 - root - INFO - 2019-09-19 01:39:34,860 - root - INFO - Please note that you have to update your firewall configuration to 2019-09-19 01:39:34,860 - root - INFO - allow connections to the following ports: 2019-09-19 01:39:34,860 - root - INFO - 4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211, 2019-09-19 01:39:34,860 - root - INFO - 11214, 11215, 18091 to 18093, and from 21100 to 21299. 2019-09-19 01:39:34,860 - root - INFO - 2019-09-19 01:39:34,861 - root - INFO - By using this software you agree to the End User License Agreement. 2019-09-19 01:39:34,861 - root - INFO - See /opt/couchbase/LICENSE.txt. 2019-09-19 01:39:34,861 - root - INFO - 2019-09-19 01:39:34,861 - root - INFO - Verifying : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:39:34,861 - root - INFO - 2019-09-19 01:39:34,861 - root - INFO - Installed: 2019-09-19 01:39:34,861 - root - INFO - couchbase-server.x86_64 0:6.0.3-2879 2019-09-19 01:39:34,861 - root - INFO - 2019-09-19 01:39:34,861 - root - INFO - Complete! 2019-09-19 01:39:34,861 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2019-09-19 01:39:34,861 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2019-09-19 01:39:34,861 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2019-09-19 01:39:34,861 - root - INFO - Warming: RPMDB altered outside of yum 2019-09-19 01:39:34,955 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2019-09-19 01:39:35,137 - root - INFO - Running systemd command on this server 2019-09-19 01:39:35,137 - root - INFO - running command.raw on 172.23.98.196: systemctl start couchbase-server.service 2019-09-19 01:39:35,204 - root - INFO - command executed successfully 2019-09-19 01:39:35,226 - root - INFO - wait 5 seconds for Couchbase server to start 2019-09-19 01:39:39,114 - root - INFO - command executed successfully 2019-09-19 01:39:39,114 - root - ERROR - Warning: RPMDB altered outside of yum. 2019-09-19 01:39:39,115 - root - INFO - Loaded plugins: fastestmirror 2019-09-19 01:39:39,115 - root - INFO - Examining /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm: couchbase-server-6.0.3-2879.x86_64 2019-09-19 01:39:39,115 - root - INFO - Marking /tmp/couchbase-server-enterprise-6.0.3-2879-centos7.x86_64.rpm to be installed 2019-09-19 01:39:39,115 - root - INFO - Resolving Dependencies 2019-09-19 01:39:39,115 - root - INFO - --> Running transaction check 2019-09-19 01:39:39,115 - root - INFO - ---> Package couchbase-server.x86_64 0:6.0.3-2879 will be installed 2019-09-19 01:39:39,115 - root - INFO - --> Finished Dependency Resolution 2019-09-19 01:39:39,116 - root - INFO - 2019-09-19 01:39:39,116 - root - INFO - Dependencies Resolved 2019-09-19 01:39:39,116 - root - INFO - 2019-09-19 01:39:39,116 - root - INFO - ================================================================================ 2019-09-19 01:39:39,116 - root - INFO - Package 2019-09-19 01:39:39,116 - root - INFO - Arch Version 2019-09-19 01:39:39,116 - root - INFO - Repository Size 2019-09-19 01:39:39,116 - root - INFO - ================================================================================ 2019-09-19 01:39:39,116 - root - INFO - Installing: 2019-09-19 01:39:39,116 - root - INFO - couchbase-server 2019-09-19 01:39:39,117 - root - INFO - x86_64 6.0.3-2879 2019-09-19 01:39:39,117 - root - INFO - /couchbase-server-enterprise-6.0.3-2879-centos7.x86_64 762 M 2019-09-19 01:39:39,117 - root - INFO - 2019-09-19 01:39:39,117 - root - INFO - Transaction Summary 2019-09-19 01:39:39,117 - root - INFO - ================================================================================ 2019-09-19 01:39:39,117 - root - INFO - Install 1 Package 2019-09-19 01:39:39,117 - root - INFO - 2019-09-19 01:39:39,117 - root - INFO - Total size: 762 M 2019-09-19 01:39:39,117 - root - INFO - Installed size: 762 M 2019-09-19 01:39:39,117 - root - INFO - Downloading packages: 2019-09-19 01:39:39,117 - root - INFO - Running transaction check 2019-09-19 01:39:39,117 - root - INFO - Running transaction test 2019-09-19 01:39:39,117 - root - INFO - Transaction test succeeded 2019-09-19 01:39:39,118 - root - INFO - Running transaction 2019-09-19 01:39:39,118 - root - INFO - Warning: Transparent hugepages looks to be active and should not be. 2019-09-19 01:39:39,118 - root - INFO - Please look at https://developer.couchbase.com/documentation/server/current/install/thp-disable.html as for how to PERMANENTLY alter this setting. 2019-09-19 01:39:39,118 - root - INFO - Minimum RAM required : 4 GB 2019-09-19 01:39:39,118 - root - INFO - System RAM configured : 5.66 GB 2019-09-19 01:39:39,118 - root - INFO - 2019-09-19 01:39:39,118 - root - INFO - Minimum number of processors required : 4 cores 2019-09-19 01:39:39,118 - root - INFO - Number of processors on the system : 4 cores 2019-09-19 01:39:39,118 - root - INFO - 2019-09-19 01:39:39,118 - root - INFO - Installing : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:39:39,118 - root - INFO - 2019-09-19 01:39:39,118 - root - INFO - You have successfully installed Couchbase Server. 2019-09-19 01:39:39,118 - root - INFO - Please browse to http://localhost.localdomain:8091/ to configure your server. 2019-09-19 01:39:39,118 - root - INFO - Please refer to http://couchbase.com for additional resources. 2019-09-19 01:39:39,119 - root - INFO - 2019-09-19 01:39:39,119 - root - INFO - Please note that you have to update your firewall configuration to 2019-09-19 01:39:39,119 - root - INFO - allow connections to the following ports: 2019-09-19 01:39:39,119 - root - INFO - 4369, 8091 to 8094, 9100 to 9105, 9998, 9999, 11207, 11209 to 11211, 2019-09-19 01:39:39,119 - root - INFO - 11214, 11215, 18091 to 18093, and from 21100 to 21299. 2019-09-19 01:39:39,119 - root - INFO - 2019-09-19 01:39:39,119 - root - INFO - By using this software you agree to the End User License Agreement. 2019-09-19 01:39:39,119 - root - INFO - See /opt/couchbase/LICENSE.txt. 2019-09-19 01:39:39,119 - root - INFO - 2019-09-19 01:39:39,119 - root - INFO - Verifying : couchbase-server-6.0.3-2879.x86_64 1/1 2019-09-19 01:39:39,119 - root - INFO - 2019-09-19 01:39:39,119 - root - INFO - Installed: 2019-09-19 01:39:39,119 - root - INFO - couchbase-server.x86_64 0:6.0.3-2879 2019-09-19 01:39:39,120 - root - INFO - 2019-09-19 01:39:39,120 - root - INFO - Complete! 2019-09-19 01:39:39,120 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2019-09-19 01:39:39,120 - root - INFO - Warming: RPMDB altered outside of yum 2019-09-19 01:39:39,120 - root - INFO - There is a warning about transparent_hugepage may be in used when install cb server. So we will disable transparent_hugepage in this vm 2019-09-19 01:39:39,120 - root - INFO - running command.raw on 172.23.104.186: echo never > /sys/kernel/mm/transparent_hugepage/enabled 2019-09-19 01:39:39,185 - root - INFO - command executed successfully 2019-09-19 01:39:39,245 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2019-09-19 01:39:39,431 - root - INFO - Running systemd command on this server 2019-09-19 01:39:39,431 - root - INFO - running command.raw on 172.23.104.186: systemctl start couchbase-server.service 2019-09-19 01:39:39,498 - root - INFO - command executed successfully 2019-09-19 01:39:39,547 - root - INFO - wait 5 seconds for Couchbase server to start thread installer-thread-172.23.98.196 finished thread installer-thread-172.23.120.206 finished thread installer-thread-172.23.98.195 finished thread installer-thread-172.23.121.10 finished thread installer-thread-172.23.104.186 finished thread installer-thread-172.23.120.201 finished 2019-09-19 01:39:44,675 - root - INFO - connecting to 172.23.120.206 with username:root 2019-09-19 01:39:44,677 - root - INFO - connecting to 172.23.98.195 with username:root 2019-09-19 01:39:44,682 - root - INFO - connecting to 172.23.98.196 with username:root 2019-09-19 01:39:44,686 - root - INFO - connecting to 172.23.104.186 with username:root 2019-09-19 01:39:44,691 - root - INFO - connecting to 172.23.120.201 with username:root 2019-09-19 01:39:44,691 - root - INFO - connecting to 172.23.121.10 with username:root 2019-09-19 01:39:44,790 - root - INFO - Connected to 172.23.120.201 2019-09-19 01:39:44,808 - root - INFO - Connected to 172.23.120.206 2019-09-19 01:39:44,847 - root - INFO - Connected to 172.23.98.195 2019-09-19 01:39:44,851 - root - INFO - Connected to 172.23.98.196 2019-09-19 01:39:44,853 - root - INFO - Connected to 172.23.104.186 2019-09-19 01:39:44,857 - root - INFO - Connected to 172.23.121.10 2019-09-19 01:39:44,910 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:39:45,002 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:39:45,287 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:39:45,374 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:39:45,403 - root - INFO - 172.23.120.206 **** The version file /opt/couchbase/ VERSION.txt exists 2019-09-19 01:39:45,433 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:39:45,437 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:39:45,474 - root - INFO - 172.23.120.201 **** The version file /opt/couchbase/ VERSION.txt exists 2019-09-19 01:39:45,814 - root - INFO - 172.23.121.10 **** The version file /opt/couchbase/ VERSION.txt exists 2019-09-19 01:39:45,815 - root - ERROR - socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2019-09-19 01:39:45,895 - root - INFO - 172.23.98.195 **** The version file /opt/couchbase/ VERSION.txt exists 2019-09-19 01:39:45,958 - root - INFO - 172.23.104.186 **** The version file /opt/couchbase/ VERSION.txt exists 2019-09-19 01:39:45,959 - root - ERROR - socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2019-09-19 01:39:45,962 - root - INFO - 172.23.98.196 **** The version file /opt/couchbase/ VERSION.txt exists 2019-09-19 01:39:45,963 - root - ERROR - socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2019-09-19 01:39:48,964 - root - ERROR - socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2019-09-19 01:39:49,430 - root - INFO - quota for kv: 1893 MB 2019-09-19 01:39:49,431 - root - INFO - pools/default params : memoryQuota=1893 2019-09-19 01:39:49,436 - root - INFO - /node/controller/setupServices params on 172.23.120.206: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2019-09-19 01:39:49,443 - root - INFO - settings/web params on 172.23.120.206:8091:username=Administrator&password=password&port=8091 2019-09-19 01:39:49,485 - root - INFO - Node version in cluster 6.0.3-2879-enterprise 2019-09-19 01:39:49,486 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2019-09-19 01:39:49,496 - root - INFO - quota for kv: 1893 MB 2019-09-19 01:39:49,497 - root - INFO - pools/default params : memoryQuota=1893 2019-09-19 01:39:49,504 - root - INFO - /node/controller/setupServices params on 172.23.120.201: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2019-09-19 01:39:49,510 - root - INFO - settings/web params on 172.23.120.201:8091:username=Administrator&password=password&port=8091 2019-09-19 01:39:49,554 - root - INFO - Node version in cluster 6.0.3-2879-enterprise 2019-09-19 01:39:49,555 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2019-09-19 01:39:49,803 - root - INFO - check if man installed on vm? 2019-09-19 01:39:49,803 - root - INFO - running command.raw on 172.23.120.206: rpm -qa | grep 'man-db' 2019-09-19 01:39:49,896 - root - INFO - check if man installed on vm? 2019-09-19 01:39:49,896 - root - INFO - running command.raw on 172.23.120.201: rpm -qa | grep 'man-db' 2019-09-19 01:39:49,932 - root - INFO - quota for kv: 1893 MB 2019-09-19 01:39:49,932 - root - INFO - pools/default params : memoryQuota=1893 2019-09-19 01:39:49,941 - root - INFO - /node/controller/setupServices params on 172.23.98.195: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2019-09-19 01:39:49,952 - root - INFO - settings/web params on 172.23.98.195:8091:username=Administrator&password=password&port=8091 2019-09-19 01:39:50,051 - root - INFO - Node version in cluster 6.0.3-2879-enterprise 2019-09-19 01:39:50,052 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2019-09-19 01:39:50,333 - root - INFO - command executed successfully 2019-09-19 01:39:50,333 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.120.206 2019-09-19 01:39:50,334 - root - INFO - running command.raw on 172.23.120.206: unzip 2019-09-19 01:39:50,348 - root - INFO - check if man installed on vm? 2019-09-19 01:39:50,348 - root - INFO - running command.raw on 172.23.98.195: rpm -qa | grep 'man-db' 2019-09-19 01:39:50,426 - root - INFO - command executed successfully 2019-09-19 01:39:50,427 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.120.201 2019-09-19 01:39:50,427 - root - INFO - running command.raw on 172.23.120.201: unzip 2019-09-19 01:39:50,431 - root - INFO - command executed successfully 2019-09-19 01:39:50,431 - root - INFO - unzip command is ready 2019-09-19 01:39:50,431 - root - INFO - Check if ntp is installed 2019-09-19 01:39:50,431 - root - INFO - This OS version centos 7 2019-09-19 01:39:50,431 - root - INFO - running command.raw on 172.23.120.206: systemctl status ntpd 2019-09-19 01:39:50,475 - root - INFO - command executed successfully 2019-09-19 01:39:50,475 - root - INFO - unzip command is ready 2019-09-19 01:39:50,476 - root - INFO - Check if ntp is installed 2019-09-19 01:39:50,476 - root - INFO - This OS version centos 7 2019-09-19 01:39:50,476 - root - INFO - running command.raw on 172.23.120.201: systemctl status ntpd 2019-09-19 01:39:50,573 - root - INFO - command executed successfully 2019-09-19 01:39:50,573 - root - INFO - running command.raw on 172.23.120.201: date 2019-09-19 01:39:50,578 - root - INFO - command executed successfully 2019-09-19 01:39:50,578 - root - INFO - running command.raw on 172.23.120.206: date 2019-09-19 01:39:50,588 - root - INFO - command executed successfully 2019-09-19 01:39:50,621 - root - INFO - command executed successfully 2019-09-19 01:39:50,793 - root - INFO - ['Thu Sep 19 01:39:50 PDT 2019'] IP: 172.23.120.206 2019-09-19 01:39:50,799 - root - INFO - ['Thu Sep 19 01:39:50 PDT 2019'] IP: 172.23.120.201 2019-09-19 01:39:51,598 - root - INFO - command executed successfully 2019-09-19 01:39:51,598 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.98.195 2019-09-19 01:39:51,598 - root - INFO - running command.raw on 172.23.98.195: unzip 2019-09-19 01:39:51,649 - root - INFO - command executed successfully 2019-09-19 01:39:51,649 - root - INFO - unzip command is ready 2019-09-19 01:39:51,649 - root - INFO - Check if ntp is installed 2019-09-19 01:39:51,650 - root - INFO - This OS version centos 7 2019-09-19 01:39:51,650 - root - INFO - running command.raw on 172.23.98.195: systemctl status ntpd 2019-09-19 01:39:51,747 - root - INFO - command executed successfully 2019-09-19 01:39:51,747 - root - INFO - running command.raw on 172.23.98.195: date 2019-09-19 01:39:51,845 - root - INFO - command executed successfully 2019-09-19 01:39:52,075 - root - INFO - ['Thu Sep 19 01:39:52 PDT 2019'] IP: 172.23.98.195 2019-09-19 01:39:52,851 - root - INFO - quota for kv: 2155 MB 2019-09-19 01:39:52,851 - root - INFO - pools/default params : memoryQuota=2155 2019-09-19 01:39:52,860 - root - INFO - /node/controller/setupServices params on 172.23.121.10: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2019-09-19 01:39:52,873 - root - INFO - settings/web params on 172.23.121.10:8091:username=Administrator&password=password&port=8091 2019-09-19 01:39:52,960 - root - INFO - Node version in cluster 6.0.3-2879-enterprise 2019-09-19 01:39:52,960 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2019-09-19 01:39:52,999 - root - INFO - quota for kv: 1893 MB 2019-09-19 01:39:52,999 - root - INFO - pools/default params : memoryQuota=1893 2019-09-19 01:39:53,010 - root - INFO - /node/controller/setupServices params on 172.23.98.196: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2019-09-19 01:39:53,022 - root - INFO - settings/web params on 172.23.98.196:8091:username=Administrator&password=password&port=8091 2019-09-19 01:39:53,114 - root - INFO - Node version in cluster 6.0.3-2879-enterprise 2019-09-19 01:39:53,114 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2019-09-19 01:39:53,290 - root - INFO - check if man installed on vm? 2019-09-19 01:39:53,290 - root - INFO - running command.raw on 172.23.121.10: rpm -qa | grep 'man-db' 2019-09-19 01:39:53,475 - root - INFO - check if man installed on vm? 2019-09-19 01:39:53,475 - root - INFO - running command.raw on 172.23.98.196: rpm -qa | grep 'man-db' 2019-09-19 01:39:54,575 - root - INFO - command executed successfully 2019-09-19 01:39:54,575 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.98.196 2019-09-19 01:39:54,575 - root - INFO - running command.raw on 172.23.98.196: unzip 2019-09-19 01:39:54,624 - root - INFO - command executed successfully 2019-09-19 01:39:54,624 - root - INFO - unzip command is ready 2019-09-19 01:39:54,624 - root - INFO - Check if ntp is installed 2019-09-19 01:39:54,624 - root - INFO - This OS version centos 7 2019-09-19 01:39:54,625 - root - INFO - running command.raw on 172.23.98.196: systemctl status ntpd 2019-09-19 01:39:54,772 - root - INFO - command executed successfully 2019-09-19 01:39:54,773 - root - INFO - running command.raw on 172.23.98.196: date 2019-09-19 01:39:54,807 - root - INFO - command executed successfully 2019-09-19 01:39:55,004 - root - INFO - ['Thu Sep 19 01:39:54 PDT 2019'] IP: 172.23.98.196 thread initializer-thread-172.23.98.196 finished thread initializer-thread-172.23.120.206 finished thread initializer-thread-172.23.98.195 finished 2019-09-19 01:39:55,191 - root - INFO - command executed successfully 2019-09-19 01:39:55,191 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.121.10 2019-09-19 01:39:55,191 - root - INFO - running command.raw on 172.23.121.10: unzip 2019-09-19 01:39:55,289 - root - INFO - command executed successfully 2019-09-19 01:39:55,289 - root - INFO - unzip command is ready 2019-09-19 01:39:55,289 - root - INFO - Check if ntp is installed 2019-09-19 01:39:55,289 - root - INFO - This OS version centos 7 2019-09-19 01:39:55,290 - root - INFO - running command.raw on 172.23.121.10: systemctl status ntpd 2019-09-19 01:39:55,387 - root - INFO - command executed successfully 2019-09-19 01:39:55,388 - root - INFO - running command.raw on 172.23.121.10: date 2019-09-19 01:39:55,453 - root - INFO - command executed successfully 2019-09-19 01:39:55,635 - root - INFO - ['Thu Sep 19 01:39:55 PDT 2019'] IP: 172.23.121.10 thread initializer-thread-172.23.121.10 finished 2019-09-19 01:39:56,001 - root - INFO - quota for kv: 3105 MB 2019-09-19 01:39:56,001 - root - INFO - pools/default params : memoryQuota=3105 2019-09-19 01:39:56,011 - root - INFO - /node/controller/setupServices params on 172.23.104.186: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2019-09-19 01:39:56,024 - root - INFO - settings/web params on 172.23.104.186:8091:username=Administrator&password=password&port=8091 2019-09-19 01:39:56,112 - root - INFO - Node version in cluster 6.0.3-2879-enterprise 2019-09-19 01:39:56,112 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2019-09-19 01:39:56,412 - root - INFO - check if man installed on vm? 2019-09-19 01:39:56,412 - root - INFO - running command.raw on 172.23.104.186: rpm -qa | grep 'man-db' 2019-09-19 01:39:58,063 - root - INFO - command executed successfully 2019-09-19 01:39:58,064 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.186 2019-09-19 01:39:58,064 - root - INFO - running command.raw on 172.23.104.186: unzip 2019-09-19 01:39:58,162 - root - INFO - command executed successfully 2019-09-19 01:39:58,162 - root - INFO - unzip command is ready 2019-09-19 01:39:58,162 - root - INFO - Check if ntp is installed 2019-09-19 01:39:58,162 - root - INFO - This OS version centos 7 2019-09-19 01:39:58,162 - root - INFO - running command.raw on 172.23.104.186: systemctl status ntpd 2019-09-19 01:39:58,260 - root - INFO - command executed successfully 2019-09-19 01:39:58,261 - root - INFO - running command.raw on 172.23.104.186: date 2019-09-19 01:39:58,326 - root - INFO - command executed successfully 2019-09-19 01:39:58,509 - root - INFO - ['Thu Sep 19 01:39:58 PDT 2019'] IP: 172.23.104.186 thread initializer-thread-172.23.104.186 finished thread initializer-thread-172.23.120.201 finished 2019-09-19 01:39:58,625 - root - INFO - connecting to 172.23.98.196 with username:root 2019-09-19 01:39:58,790 - root - INFO - Connected to 172.23.98.196 2019-09-19 01:39:59,371 - root - INFO - It could be other operating system. Go to check at other location verify installation... 2019-09-19 01:39:59,854 - root - INFO - connecting to 172.23.98.196 with username:root 2019-09-19 01:39:59,990 - root - INFO - Connected to 172.23.98.196 2019-09-19 01:40:00,530 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:40:01,037 - root - INFO - 172.23.98.196 **** The version file /opt/couchbase/ VERSION.txt exists 2019-09-19 01:40:01,040 - root - INFO - connecting to 172.23.120.206 with username:root 2019-09-19 01:40:01,121 - root - INFO - Connected to 172.23.120.206 2019-09-19 01:40:01,219 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:40:01,673 - root - INFO - 172.23.120.206 **** The version file /opt/couchbase/ VERSION.txt exists 2019-09-19 01:40:01,677 - root - INFO - connecting to 172.23.98.195 with username:root 2019-09-19 01:40:01,843 - root - INFO - Connected to 172.23.98.195 2019-09-19 01:40:02,351 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:40:02,848 - root - INFO - 172.23.98.195 **** The version file /opt/couchbase/ VERSION.txt exists 2019-09-19 01:40:02,850 - root - INFO - connecting to 172.23.121.10 with username:root 2019-09-19 01:40:03,017 - root - INFO - Connected to 172.23.121.10 2019-09-19 01:40:03,476 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:40:04,043 - root - INFO - 172.23.121.10 **** The version file /opt/couchbase/ VERSION.txt exists 2019-09-19 01:40:04,045 - root - INFO - connecting to 172.23.104.186 with username:root 2019-09-19 01:40:04,211 - root - INFO - Connected to 172.23.104.186 2019-09-19 01:40:04,738 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:40:05,227 - root - INFO - 172.23.104.186 **** The version file /opt/couchbase/ VERSION.txt exists 2019-09-19 01:40:05,229 - root - INFO - connecting to 172.23.120.201 with username:root 2019-09-19 01:40:05,326 - root - INFO - Connected to 172.23.120.201 2019-09-19 01:40:05,540 - root - INFO - It could be other operating system. Go to check at other location 2019-09-19 01:40:06,035 - root - INFO - 172.23.120.201 **** The version file /opt/couchbase/ VERSION.txt exists + python testrunner.py -i /tmp/win10-bucket-ops.ini -p get-cbcollect-info=True,flusher_batch_split_trigger=10 -t rebalance.rebalance_high_ops_pillowfight.RebalanceHighOpsWithPillowFight.test_graceful_failover_addback,node_out=3,replicas=2,nodes_init=4,items=2000000,batch_size=1000,rate_limit=100000,recovery_type=delta,instances=2,threads=5,loader=high_ops,flusher_batch_split_trigger=1 test_graceful_failover_addback (rebalance.rebalance_high_ops_pillowfight.RebalanceHighOpsWithPillowFight) ... Global Test input params: {'cluster_name': 'win10-bucket-ops', 'flusher_batch_split_trigger': '10', 'get-cbcollect-info': 'True', 'ini': '/tmp/win10-bucket-ops.ini', 'num_nodes': 6} Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-19-Sep-19_01-40-06/test_1 ./testrunner -i /tmp/win10-bucket-ops.ini -p get-cbcollect-info=True,flusher_batch_split_trigger=10 -t rebalance.rebalance_high_ops_pillowfight.RebalanceHighOpsWithPillowFight.test_graceful_failover_addback,node_out=3,replicas=2,nodes_init=4,items=2000000,batch_size=1000,rate_limit=100000,recovery_type=delta,instances=2,threads=5,loader=high_ops,flusher_batch_split_trigger=1 Test Input params: {'get-cbcollect-info': 'True', 'replicas': '2', 'items': '2000000', 'rate_limit': '100000', 'recovery_type': 'delta', 'batch_size': '1000', 'loader': 'high_ops', 'cluster_name': 'win10-bucket-ops', 'instances': '2', 'threads': '5', 'flusher_batch_split_trigger': '10', 'case_number': 1, 'node_out': '3', 'num_nodes': 6, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-19-Sep-19_01-40-06/test_1', 'nodes_init': '4', 'ini': '/tmp/win10-bucket-ops.ini'} Run before suite setup for rebalance.rebalance_high_ops_pillowfight.RebalanceHighOpsWithPillowFight.test_graceful_failover_addback 2019-09-19 01:40:06 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.196 with username:root 2019-09-19 01:40:07 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.196 2019-09-19 01:40:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 6.0.3-2879-enterprise 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'6.0.3-2879-enterprise'] 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_graceful_failover_addback============== 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.98.196:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.98.196:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.120.206:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.120.206:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.98.195:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.98.195:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.10:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.10:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.186:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.186:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.120.201:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.120.201:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.98.196:8091 => {'swap_mem_used': 0, 'cpu_utilization': 3.258145363408521, 'mem_free': 3010236416, 'swap_mem_total': 3909087232, 'mem_total': 3706343424, 'services': [u'kv']} 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_graceful_failover_addback ============== 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.98.196:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.98.196:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.120.206:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.120.206:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.98.195:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.98.195:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.10:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.10:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.186:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.186:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.120.201:8091 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.120.201:8091 is running 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_graceful_failover_addback ============== Cluster instance shutdown with force 2019-09-19 01:40:08 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2019-09-19 01:40:09 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.98.196 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.196', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'28', 'mcdMemoryReserved': 2827, 'storageTotalRam': 3534, 'hostname': u'172.23.98.196:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.3-2879-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3010236416, 'memoryTotal': 3706343424, 'memoryQuota': 1893, 'mcdMemoryAllocated': 2827, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-09-19 01:40:09 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1894 2019-09-19 01:40:09 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.98.196:8091:username=Administrator&password=password&port=8091 2019-09-19 01:40:09 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.98.196 with username:root 2019-09-19 01:40:09 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.98.196 2019-09-19 01:40:10 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.196: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.98.196:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.98.196:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.120.206 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.120.206', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'88', 'mcdMemoryReserved': 2827, 'storageTotalRam': 3534, 'hostname': u'172.23.120.206:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.3-2879-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3056164864, 'memoryTotal': 3705872384, 'memoryQuota': 1893, 'mcdMemoryAllocated': 2827, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1894 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.120.206:8091:username=Administrator&password=password&port=8091 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.120.206 with username:root 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.120.206 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.120.206: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.120.206:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.120.206:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.98.195 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.195', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'37', 'mcdMemoryReserved': 2827, 'storageTotalRam': 3534, 'hostname': u'172.23.98.195:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.3-2879-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 2962624512, 'memoryTotal': 3706343424, 'memoryQuota': 1893, 'mcdMemoryAllocated': 2827, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1894 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.98.195:8091:username=Administrator&password=password&port=8091 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.98.195 with username:root 2019-09-19 01:40:11 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.98.195 2019-09-19 01:40:12 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.195: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-09-19 01:40:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-09-19 01:40:13 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.98.195:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:13 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.98.195:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:13 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-09-19 01:40:13 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.121.10 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.121.10', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'33', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4023, 'hostname': u'172.23.121.10:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.3-2879-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3534381056, 'memoryTotal': 4219432960, 'memoryQuota': 2155, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-09-19 01:40:13 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2156 2019-09-19 01:40:13 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.121.10:8091:username=Administrator&password=password&port=8091 2019-09-19 01:40:13 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.121.10 with username:root 2019-09-19 01:40:13 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.121.10 2019-09-19 01:40:13 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:14 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.121.10: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-09-19 01:40:14 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-09-19 01:40:14 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.10:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:14 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.10:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:14 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-09-19 01:40:14 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.186 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.186', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'27', 'mcdMemoryReserved': 4637, 'storageTotalRam': 5797, 'hostname': u'172.23.104.186:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.3-2879-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 5382426624, 'memoryTotal': 6078930944, 'memoryQuota': 3105, 'mcdMemoryAllocated': 4637, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-09-19 01:40:14 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=3106 2019-09-19 01:40:14 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.186:8091:username=Administrator&password=password&port=8091 2019-09-19 01:40:14 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.104.186 with username:root 2019-09-19 01:40:14 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.104.186 2019-09-19 01:40:15 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.186: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-09-19 01:40:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-09-19 01:40:15 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.186:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:15 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.186:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:15 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-09-19 01:40:15 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.120.201 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.120.201', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'68', 'mcdMemoryReserved': 2827, 'storageTotalRam': 3534, 'hostname': u'172.23.120.201:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.3-2879-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3033993216, 'memoryTotal': 3705872384, 'memoryQuota': 1893, 'mcdMemoryAllocated': 2827, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-09-19 01:40:15 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1894 2019-09-19 01:40:15 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.120.201:8091:username=Administrator&password=password&port=8091 2019-09-19 01:40:15 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.120.201 with username:root 2019-09-19 01:40:15 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.120.201 2019-09-19 01:40:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.120.201: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-09-19 01:40:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-09-19 01:40:16 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.120.201:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:16 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.120.201:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). 2019-09-19 01:40:16 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-09-19 01:40:16 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.98.196 **** 2019-09-19 01:40:16 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.98.196:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password 2019-09-19 01:40:16 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -"User was not found." 2019-09-19 01:40:16 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2019-09-19 01:40:16 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2019-09-19 01:40:17 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.120.206:8091 to cluster 2019-09-19 01:40:17 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.120.206:8091 to this cluster @172.23.98.196:8091 2019-09-19 01:40:19 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.98.195:8091 to cluster 2019-09-19 01:40:19 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.98.195:8091 to this cluster @172.23.98.196:8091 2019-09-19 01:40:24 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.121.10:8091 to cluster 2019-09-19 01:40:24 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.121.10:8091 to this cluster @172.23.98.196:8091 2019-09-19 01:40:29 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.120.206,ns_1@172.23.98.195,ns_1@172.23.121.10,ns_1@172.23.98.196'} 2019-09-19 01:40:29 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2019-09-19 01:40:29 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-09-19 01:40:29 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-09-19 01:40:39 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2019-09-19 01:40:39 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.0408530235 sec 2019-09-19 01:40:39 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.196 with username:root 2019-09-19 01:40:39 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.196 2019-09-19 01:40:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.196: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-09-19 01:40:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-09-19 01:40:41 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.98.196 2019-09-19 01:40:41 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.98.196:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1894&authType=sasl&name=default&flushEnabled=1&replicaNumber=2&compressionMode=passive&saslPassword=None&replicaIndex=1 2019-09-19 01:40:41 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket default 2019-09-19 01:40:41 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.98.196 to accept set ops 2019-09-19 01:40:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:44 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2019-09-19 01:40:44 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.98.196 to accept set ops 2019-09-19 01:40:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:46 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2019-09-19 01:40:46 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.98.196 to accept set ops 2019-09-19 01:40:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:47 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2019-09-19 01:40:47 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.98.196 to accept set ops 2019-09-19 01:40:47 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:47 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:47 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:47 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:47 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:47 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:47 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:47 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:48 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2019-09-19 01:40:48 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.98.196 to accept set ops 2019-09-19 01:40:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:49 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2019-09-19 01:40:49 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.98.196 to accept set ops 2019-09-19 01:40:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:40:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:40:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:40:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:40:50 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_graceful_failover_addback ============== 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.120.206:8091 => {'swap_mem_used': 0, 'cpu_utilization': 3.508771929824561, 'mem_free': 2989375488, 'swap_mem_total': 0, 'mem_total': 3705872384, 'services': [u'kv']} 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.98.195:8091 => {'swap_mem_used': 974848, 'cpu_utilization': 7.556675062972293, 'mem_free': 2894180352, 'swap_mem_total': 3909087232, 'mem_total': 3706343424, 'services': [u'kv']} 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.121.10:8091 => {'swap_mem_used': 0, 'cpu_utilization': 6.818181818181818, 'mem_free': 3468890112, 'swap_mem_total': 3758092288, 'mem_total': 4219432960, 'services': [u'kv']} 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.98.196:8091 => {'swap_mem_used': 0, 'cpu_utilization': 11.55778894472362, 'mem_free': 2931617792, 'swap_mem_total': 3909087232, 'mem_total': 3706343424, 'services': [u'kv']} 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [basetestcase.set_flusher_batch_split_trigger] Changing the bucket properties by changing flusher_batch_split_trigger to 10 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.98.196:8091: True content: ok command: ns_bucket:update_bucket_props("default", [{extra_config_string, "flusher_batch_split_trigger=10"}]). 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'6.0.3-2879-enterprise', u'6.0.3-2879-enterprise', u'6.0.3-2879-enterprise', u'6.0.3-2879-enterprise'] 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.120.206 port:8091 ssh_username:root, ip:172.23.98.195 port:8091 ssh_username:root, ip:172.23.121.10 port:8091 ssh_username:root, ip:172.23.98.196 port:8091 ssh_username:root] 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.196 with username:root 2019-09-19 01:40:50 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.196 2019-09-19 01:40:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:51 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.120.206 with username:root 2019-09-19 01:40:51 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.120.206 2019-09-19 01:40:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:52 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.195 with username:root 2019-09-19 01:40:52 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.195 2019-09-19 01:40:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.121.10 with username:root 2019-09-19 01:40:54 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.121.10 2019-09-19 01:40:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] It could be other operating system. Go to check at other location 2019-09-19 01:40:55 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2019-09-19 01:41:25 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'6.0.3-2879-enterprise', u'6.0.3-2879-enterprise', u'6.0.3-2879-enterprise', u'6.0.3-2879-enterprise'] 2019-09-19 01:41:25 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.120.206 port:8091 ssh_username:root, ip:172.23.98.195 port:8091 ssh_username:root, ip:172.23.121.10 port:8091 ssh_username:root, ip:172.23.98.196 port:8091 ssh_username:root] 2019-09-19 01:41:25 | INFO | MainProcess | test_thread | [rebalance_high_ops_pillowfight.test_graceful_failover_addback] starting the load thread... 2019-09-19 01:41:25 | INFO | MainProcess | high_ops_load | [rest_client.get_nodes_version] Node version in cluster 6.0.3-2879-enterprise 2019-09-19 01:41:25 | INFO | MainProcess | high_ops_load | [rebalance_high_ops_pillowfight.load_buckets_with_high_ops] Running python scripts/thanosied.py --spec couchbase://172.23.98.196 --bucket default --user Administrator --password password --count 2000000 --batch_size 1000 --threads 5 --start_document 0 --cb_version 6.0 --workers 2 --ttl 0 --rate_limit 100000 --passes 1 2019-09-19 01:42:24 | INFO | MainProcess | high_ops_load | [rest_client.get_nodes_version] Node version in cluster 6.0.3-2879-enterprise 2019-09-19 01:42:24 | INFO | MainProcess | high_ops_load | [rebalance_high_ops_pillowfight.load_buckets_with_high_ops] Running python scripts/thanosied.py --spec couchbase://172.23.98.196 --bucket default --user Administrator --password password --count 2000000 --batch_size 1000 --threads 5 --start_document 2000000 --cb_version 6.0 --workers 2 --ttl 0 --rate_limit 100000 --passes 1 2019-09-19 01:42:24 | INFO | MainProcess | Cluster_Thread | [task._failover_nodes] Failing over 172.23.121.10:8091 with graceful=graceful 2019-09-19 01:42:24 | INFO | MainProcess | Cluster_Thread | [rest_client.fail_over] fail_over node ns_1@172.23.121.10 successful 2019-09-19 01:42:24 | INFO | MainProcess | Cluster_Thread | [task.execute] 360 seconds sleep after failover, for nodes to go pending.... 2019-09-19 01:48:24 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=delta for node :: ns_1@172.23.121.10 2019-09-19 01:48:24 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.121.10 set successful 2019-09-19 01:48:24 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2019-09-19 01:48:24 | INFO | MainProcess | high_ops_load | [rest_client.get_nodes_version] Node version in cluster 6.0.3-2879-enterprise 2019-09-19 01:48:24 | INFO | MainProcess | high_ops_load | [rebalance_high_ops_pillowfight.load_buckets_with_high_ops] Running python scripts/thanosied.py --spec couchbase://172.23.98.196 --bucket default --user Administrator --password password --count 4000000 --batch_size 1000 --threads 5 --start_document 4000000 --cb_version 6.0 --workers 2 --ttl 0 --rate_limit 100000 --passes 1 2019-09-19 01:48:34 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.121.10 successful 2019-09-19 01:48:34 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=delta for node :: ns_1@172.23.121.10 2019-09-19 01:48:34 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.121.10 set successful 2019-09-19 01:48:34 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2019-09-19 01:48:44 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.121.10 successful 2019-09-19 01:48:44 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=delta for node :: ns_1@172.23.121.10 2019-09-19 01:48:44 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.121.10 set successful 2019-09-19 01:48:44 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2019-09-19 01:48:54 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.121.10 successful 2019-09-19 01:48:54 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=delta for node :: ns_1@172.23.121.10 2019-09-19 01:48:54 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.121.10 set successful 2019-09-19 01:48:54 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2019-09-19 01:49:04 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.121.10 successful 2019-09-19 01:49:04 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=delta for node :: ns_1@172.23.121.10 2019-09-19 01:49:04 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.121.10 set successful 2019-09-19 01:49:04 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2019-09-19 01:49:14 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.121.10 successful 2019-09-19 01:49:14 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=delta for node :: ns_1@172.23.121.10 2019-09-19 01:49:14 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.121.10 set successful 2019-09-19 01:49:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2019-09-19 01:49:24 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.121.10 successful 2019-09-19 01:49:24 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=delta for node :: ns_1@172.23.121.10 2019-09-19 01:49:24 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.121.10 set successful 2019-09-19 01:49:24 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2019-09-19 01:49:34 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.121.10 successful 2019-09-19 01:49:34 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=delta for node :: ns_1@172.23.121.10 2019-09-19 01:49:34 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.121.10 set successful 2019-09-19 01:49:34 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2019-09-19 01:49:44 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.121.10 successful 2019-09-19 01:49:44 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=delta for node :: ns_1@172.23.121.10 2019-09-19 01:49:44 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.121.10 set successful 2019-09-19 01:49:44 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2019-09-19 01:49:54 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.121.10 successful 2019-09-19 01:49:54 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=delta for node :: ns_1@172.23.121.10 2019-09-19 01:49:54 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.121.10 set successful 2019-09-19 01:49:54 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2019-09-19 01:50:04 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.121.10 successful 2019-09-19 01:50:04 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=delta for node :: ns_1@172.23.121.10 2019-09-19 01:50:04 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@172.23.121.10 set successful 2019-09-19 01:50:04 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2019-09-19 01:50:35 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.120.206,ns_1@172.23.98.195,ns_1@172.23.121.10,ns_1@172.23.98.196'} 2019-09-19 01:50:35 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2019-09-19 01:50:35 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-09-19 01:50:35 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-09-19 01:50:37 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.00605392456 seconds 2019-09-19 01:50:45 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-09-19 01:50:45 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-09-19 01:50:55 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-09-19 01:50:55 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-09-19 01:51:05 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-09-19 01:51:05 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-09-19 01:51:15 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-09-19 01:51:15 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-09-19 01:51:25 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-09-19 01:51:25 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-09-19 01:51:35 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-09-19 01:51:35 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-09-19 01:51:45 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 8.81 % 2019-09-19 01:51:45 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 8.81% 2019-09-19 01:51:55 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 17.42 % 2019-09-19 01:51:55 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 17.42% 2019-09-19 01:52:06 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 18.42 % 2019-09-19 01:52:06 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 18.42% 2019-09-19 01:52:16 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 18.81 % 2019-09-19 01:52:16 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 18.81% 2019-09-19 01:52:26 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 18.81 % 2019-09-19 01:52:26 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 18.81% 2019-09-19 01:52:36 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 56.49 % 2019-09-19 01:52:36 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 56.49% 2019-09-19 01:52:46 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 91.20 % 2019-09-19 01:52:46 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 91.20% 2019-09-19 01:52:56 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2019-09-19 01:52:56 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 140.592445135 sec 2019-09-19 01:52:56 | INFO | MainProcess | test_thread | [basetestcase.check_snap_start_corruption] Servers to check bucket-seqno: [ip:172.23.98.196 port:8091 ssh_username:root, ip:172.23.120.206 port:8091 ssh_username:root, ip:172.23.98.195 port:8091 ssh_username:root, ip:172.23.121.10 port:8091 ssh_username:root] 2019-09-19 01:52:56 | INFO | MainProcess | test_thread | [basetestcase._record_vbuckets] Record vbucket for the bucket default 2019-09-19 01:52:56 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:52:56 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:52:56 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:52:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:52:57 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 6.0.3-2879-enterprise 2019-09-19 01:52:57 | INFO | MainProcess | test_thread | [rebalance_high_ops_pillowfight.check_dataloss_for_high_ops_loader] Running python scripts/thanosied.py --spec couchbase://172.23.98.196 --bucket default --user Administrator --password password --count 8000000 --batch_size 1000 --threads 5 --start_document 0 --cb_version 6.0 --validation 1 --rate_limit 100000 --passes 1 2019-09-19 01:56:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:56:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:56:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:56:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:56:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.120.206:11210 default 2019-09-19 01:56:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.121.10:11210 default 2019-09-19 01:56:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.98.195:11210 default 2019-09-19 01:56:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.98.196:11210 default 2019-09-19 01:56:58 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.98.196:8091/pools/default/buckets/default/stats?zoom=minute 2019-09-19 01:56:58 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.98.196:8091/pools/default/buckets/default/stats?zoom=minute 2019-09-19 01:56:58 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.98.196:8091/pools/default/buckets/default/stats?zoom=minute 2019-09-19 01:56:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2019-09-19 01:56:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.120.206:8091 => {'swap_mem_used': 0, 'cpu_utilization': 49.21052631578947, 'mem_free': 1651150848, 'swap_mem_total': 0, 'mem_total': 3705872384, 'services': [u'kv']} 2019-09-19 01:56:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.98.195:8091 => {'swap_mem_used': 974848, 'cpu_utilization': 37.91348600508906, 'mem_free': 1508827136, 'swap_mem_total': 3909087232, 'mem_total': 3706343424, 'services': [u'kv']} 2019-09-19 01:56:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.121.10:8091 => {'swap_mem_used': 0, 'cpu_utilization': 97.25, 'mem_free': 2077003776, 'swap_mem_total': 3758092288, 'mem_total': 4219432960, 'services': [u'kv']} 2019-09-19 01:56:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.98.196:8091 => {'swap_mem_used': 0, 'cpu_utilization': 34.77157360406091, 'mem_free': 1572765696, 'swap_mem_total': 3909087232, 'mem_total': 3706343424, 'services': [u'kv']} 2019-09-19 01:56:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2019-09-19 01:56:59 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_graceful_failover_addback ============== 2019-09-19 01:56:59 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 172.23.98.196 2019-09-19 01:56:59 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ... 2019-09-19 01:57:15 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.98.196 2019-09-19 01:57:15 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2019-09-19 01:57:15 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.98.196 existing buckets : [] 2019-09-19 01:57:15 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] sleep 2 seconds to make sure all buckets ([u'default']) were deleted completely. 2019-09-19 01:57:17 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2019-09-19 01:57:17 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.120.206,ns_1@172.23.98.195,ns_1@172.23.121.10', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.120.206,ns_1@172.23.98.195,ns_1@172.23.121.10,ns_1@172.23.98.196'} 2019-09-19 01:57:17 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2019-09-19 01:57:17 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-09-19 01:57:37 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 20.03 seconds 2019-09-19 01:57:37 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2019-09-19 01:57:47 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.98.196 port:8091 ssh_username:root ? [(u'ns_1@172.23.120.206', 8091), (u'ns_1@172.23.98.195', 8091), (u'ns_1@172.23.121.10', 8091)] 2019-09-19 01:57:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.98.196:8091 2019-09-19 01:57:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.98.196:8091 is running 2019-09-19 01:57:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.120.206:8091 2019-09-19 01:57:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.120.206:8091 is running 2019-09-19 01:57:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.98.195:8091 2019-09-19 01:57:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.98.195:8091 is running 2019-09-19 01:57:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.10:8091 2019-09-19 01:57:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.10:8091 is running 2019-09-19 01:57:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.186:8091 2019-09-19 01:57:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.186:8091 is running 2019-09-19 01:57:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.120.201:8091 2019-09-19 01:57:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.120.201:8091 is running 2019-09-19 01:57:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_graceful_failover_addback ============== ok ---------------------------------------------------------------------- Ran 1 test in 1061.157s OK Cluster instance shutdown with force summary so far suite rebalance.rebalance_high_ops_pillowfight.RebalanceHighOpsWithPillowFight , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-19-Sep-19_01-40-06/test_1 Run after suite setup for rebalance.rebalance_high_ops_pillowfight.RebalanceHighOpsWithPillowFight.test_graceful_failover_addback Thread Thread-23 was not properly terminated, will be terminated now. Thread Thread-28 was not properly terminated, will be terminated now. Thread Thread-38 was not properly terminated, will be terminated now. Thread Thread-18 was not properly terminated, will be terminated now. Thread Thread-33 was not properly terminated, will be terminated now. Thread Cluster_Thread was not properly terminated, will be terminated now. Thread Thread-8 was not properly terminated, will be terminated now. Thread Thread-13 was not properly terminated, will be terminated now. [temp_rebalance_even] $ /bin/sh -xe /tmp/jenkins6210632170088745701.sh Archiving artifacts Recording test results [description-setter] Description set: 6.0.3-2879,product=cb,parallel=True Notifying upstream projects of job completion temp_durability is disabled. Triggering skipped Finished: SUCCESS