Started by user Balakumaran G [EnvInject] - Loading node environment variables. Building remotely on slv-s71801 (P0) in workspace /data/workspace/test_bala_upgrade_new1 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/bkumaran/testrunner2.git > /usr/bin/git init /data/workspace/test_bala_upgrade_new1 # timeout=10 Fetching upstream changes from git://github.com/bkumaran/testrunner2.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/bkumaran/testrunner2.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/bkumaran/testrunner2.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/bkumaran/testrunner2.git # timeout=10 Fetching upstream changes from git://github.com/bkumaran/testrunner2.git > /usr/bin/git fetch --tags --progress git://github.com/bkumaran/testrunner2.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 9cf026e1a20e70cf7c8be44e4baa973b92bee5d0 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 9cf026e1a20e70cf7c8be44e4baa973b92bee5d0 > /usr/bin/git rev-list 877bfd6e43089873e735fb949ce1761a3bf72c06 # timeout=10 [test_bala_upgrade_new1] $ /bin/sh -xe /tmp/jenkins6315249558295036982.sh + echo '[global] username:root password:couchbase port:8091 index_port:9102 n1ql_port:8093 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 [cluster1] 1:_1 2:_2 [cluster2] 1:_3 2:_4 [_1] ip:172.23.104.103 [_2] ip:172.23.104.104 [_3] ip:172.23.104.105 [_4] ip:172.23.104.106 [_5] ip:172.23.104.107 [_6] ip:172.23.104.108 [_7] ip:172.23.104.109 [_8] ip:172.23.104.90 ' [test_bala_upgrade_new1] $ /bin/sh -xe /tmp/jenkins3853305252620329004.sh + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From git://github.com/bkumaran/testrunner2 * branch master -> FETCH_HEAD Already up-to-date. + python scripts/install.py -i /tmp/upgrade3.ini -p version=6.0.0-35007,product=cb,parallel=True,url=http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm 2018-09-04 03:40:55,687 - root - INFO - *****Starting the complete install process **** 2018-09-04 03:40:55,701 - root - INFO - Doing parallel install**** 2018-09-04 03:40:55,723 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:40:55,738 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:40:55,749 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:40:55,762 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:40:55,809 - root - INFO - connecting to 172.23.104.90 with username:root password:couchbase ssh_key: 2018-09-04 03:40:55,809 - root - INFO - connecting to 172.23.104.107 with username:root password:couchbase ssh_key: 2018-09-04 03:40:55,823 - root - INFO - connecting to 172.23.104.108 with username:root password:couchbase ssh_key: 2018-09-04 03:40:55,828 - root - INFO - connecting to 172.23.104.109 with username:root password:couchbase ssh_key: 2018-09-04 03:40:56,092 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:40:56,106 - root - INFO - Connected to 172.23.104.109 2018-09-04 03:40:56,111 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:40:56,115 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:40:56,124 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:40:56,126 - root - INFO - Connected to 172.23.104.90 2018-09-04 03:40:56,152 - root - INFO - Connected to 172.23.104.107 2018-09-04 03:40:56,165 - root - INFO - Connected to 172.23.104.108 2018-09-04 03:40:56,470 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:40:56,473 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:40:56,563 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:40:56,615 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:40:56,747 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:40:57,024 - root - INFO - 172.23.104.103 *****In couchbase uninstall**** 2018-09-04 03:40:57,029 - root - INFO - CentOS 2018-09-04 03:40:57,052 - root - INFO - 172.23.104.104 *****In couchbase uninstall**** 2018-09-04 03:40:57,057 - root - INFO - CentOS 2018-09-04 03:40:57,066 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:40:57,092 - root - INFO - 172.23.104.105 *****In couchbase uninstall**** 2018-09-04 03:40:57,096 - root - INFO - CentOS 2018-09-04 03:40:57,124 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:40:57,142 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:40:57,185 - root - INFO - 172.23.104.106 *****In couchbase uninstall**** 2018-09-04 03:40:57,187 - root - INFO - CentOS 2018-09-04 03:40:57,279 - root - INFO - 172.23.104.90 *****In couchbase uninstall**** 2018-09-04 03:40:57,280 - root - INFO - CentOS 2018-09-04 03:40:57,317 - root - INFO - running command.raw on 172.23.104.104: killall -9 rpm 2018-09-04 03:40:57,355 - root - INFO - command executed successfully 2018-09-04 03:40:57,357 - root - ERROR - rpm: no process found 2018-09-04 03:40:57,364 - root - INFO - running command.raw on 172.23.104.104: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:40:57,366 - root - INFO - running command.raw on 172.23.104.105: killall -9 rpm 2018-09-04 03:40:57,403 - root - INFO - running command.raw on 172.23.104.103: killall -9 rpm 2018-09-04 03:40:57,422 - root - INFO - command executed successfully 2018-09-04 03:40:57,424 - root - INFO - command executed successfully 2018-09-04 03:40:57,426 - root - ERROR - rpm: no process found 2018-09-04 03:40:57,429 - root - INFO - running command.raw on 172.23.104.105: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:40:57,446 - root - INFO - command executed successfully 2018-09-04 03:40:57,447 - root - ERROR - rpm: no process found 2018-09-04 03:40:57,448 - root - INFO - running command.raw on 172.23.104.103: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:40:57,478 - root - INFO - command executed successfully 2018-09-04 03:40:57,506 - root - INFO - command executed successfully 2018-09-04 03:40:57,527 - root - INFO - running command.raw on 172.23.104.106: killall -9 rpm 2018-09-04 03:40:57,555 - root - INFO - command executed successfully 2018-09-04 03:40:57,557 - root - ERROR - bash: killall: command not found 2018-09-04 03:40:57,558 - root - INFO - running command.raw on 172.23.104.106: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:40:57,581 - root - INFO - running command.raw on 172.23.104.90: killall -9 rpm 2018-09-04 03:40:57,643 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:40:57,644 - root - INFO - running command.raw on 172.23.104.104: rpm -e couchbase-server 2018-09-04 03:40:57,657 - root - INFO - command executed successfully 2018-09-04 03:40:57,660 - root - INFO - command executed successfully 2018-09-04 03:40:57,661 - root - ERROR - rpm: no process found 2018-09-04 03:40:57,662 - root - INFO - running command.raw on 172.23.104.90: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:40:57,690 - root - INFO - command executed successfully 2018-09-04 03:40:57,701 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:40:57,702 - root - INFO - running command.raw on 172.23.104.105: rpm -e couchbase-server 2018-09-04 03:40:57,807 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:40:57,807 - root - INFO - running command.raw on 172.23.104.103: rpm -e couchbase-server 2018-09-04 03:40:57,858 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:40:57,858 - root - INFO - running command.raw on 172.23.104.90: rpm -e couchbase-server 2018-09-04 03:40:57,906 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:40:57,907 - root - INFO - running command.raw on 172.23.104.106: rpm -e couchbase-server 2018-09-04 03:40:58,035 - root - INFO - 172.23.104.108 *****In couchbase uninstall**** 2018-09-04 03:40:58,036 - root - INFO - CentOS 2018-09-04 03:40:58,073 - root - INFO - 172.23.104.109 *****In couchbase uninstall**** 2018-09-04 03:40:58,074 - root - INFO - CentOS 2018-09-04 03:40:58,089 - root - INFO - 172.23.104.107 *****In couchbase uninstall**** 2018-09-04 03:40:58,089 - root - INFO - CentOS 2018-09-04 03:40:58,362 - root - INFO - running command.raw on 172.23.104.108: killall -9 rpm 2018-09-04 03:40:58,413 - root - INFO - running command.raw on 172.23.104.107: killall -9 rpm 2018-09-04 03:40:58,479 - root - INFO - running command.raw on 172.23.104.109: killall -9 rpm 2018-09-04 03:40:58,492 - root - INFO - command executed successfully 2018-09-04 03:40:58,493 - root - ERROR - rpm: no process found 2018-09-04 03:40:58,494 - root - INFO - running command.raw on 172.23.104.108: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:40:58,537 - root - INFO - command executed successfully 2018-09-04 03:40:58,538 - root - ERROR - rpm: no process found 2018-09-04 03:40:58,539 - root - INFO - running command.raw on 172.23.104.107: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:40:58,573 - root - INFO - command executed successfully 2018-09-04 03:40:58,598 - root - INFO - command executed successfully 2018-09-04 03:40:58,599 - root - ERROR - rpm: no process found 2018-09-04 03:40:58,600 - root - INFO - running command.raw on 172.23.104.109: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:40:58,618 - root - INFO - command executed successfully 2018-09-04 03:40:58,672 - root - INFO - command executed successfully 2018-09-04 03:40:58,922 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:40:58,923 - root - INFO - running command.raw on 172.23.104.107: rpm -e couchbase-server 2018-09-04 03:40:58,943 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:40:58,944 - root - INFO - running command.raw on 172.23.104.108: rpm -e couchbase-server 2018-09-04 03:40:59,124 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:40:59,124 - root - INFO - running command.raw on 172.23.104.109: rpm -e couchbase-server 2018-09-04 03:41:01,096 - root - INFO - command executed successfully 2018-09-04 03:41:01,098 - root - ERROR - **** Uninstalling on server: 172.23.104.104 **** 2018-09-04 03:41:01,099 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:41:01,100 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:41:01,260 - root - INFO - command executed successfully 2018-09-04 03:41:01,261 - root - ERROR - **** Uninstalling on server: 172.23.104.105 **** 2018-09-04 03:41:01,261 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:41:01,262 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:41:01,703 - root - INFO - command executed successfully 2018-09-04 03:41:01,704 - root - ERROR - **** Uninstalling on server: 172.23.104.103 **** 2018-09-04 03:41:01,705 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:41:01,705 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:41:02,253 - root - INFO - running command.raw on 172.23.104.104: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:41:02,315 - root - INFO - command executed successfully 2018-09-04 03:41:02,316 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:41:02,317 - root - INFO - running command.raw on 172.23.104.104: ipcrm 2018-09-04 03:41:02,395 - root - INFO - running command.raw on 172.23.104.105: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:41:02,398 - root - INFO - command executed successfully 2018-09-04 03:41:02,482 - root - INFO - command executed successfully 2018-09-04 03:41:02,483 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:41:02,483 - root - INFO - running command.raw on 172.23.104.105: ipcrm 2018-09-04 03:41:02,562 - root - INFO - command executed successfully 2018-09-04 03:41:02,814 - root - INFO - running command.raw on 172.23.104.103: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:41:02,899 - root - INFO - command executed successfully 2018-09-04 03:41:02,901 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:41:02,901 - root - INFO - running command.raw on 172.23.104.103: ipcrm 2018-09-04 03:41:02,978 - root - INFO - command executed successfully thread uninstaller-thread-172.23.104.103 finished thread uninstaller-thread-172.23.104.104 finished thread uninstaller-thread-172.23.104.105 finished 2018-09-04 03:41:03,614 - root - INFO - command executed successfully 2018-09-04 03:41:03,615 - root - ERROR - **** Uninstalling on server: 172.23.104.90 **** 2018-09-04 03:41:03,616 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory 2018-09-04 03:41:03,617 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2018-09-04 03:41:03,617 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:41:03,617 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:41:03,984 - root - INFO - command executed successfully 2018-09-04 03:41:03,986 - root - ERROR - **** Uninstalling on server: 172.23.104.106 **** 2018-09-04 03:41:03,986 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory 2018-09-04 03:41:03,986 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2018-09-04 03:41:03,987 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:41:03,987 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:41:04,423 - root - INFO - command executed successfully 2018-09-04 03:41:04,424 - root - ERROR - **** Uninstalling on server: 172.23.104.108 **** 2018-09-04 03:41:04,424 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:41:04,425 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:41:04,561 - root - INFO - command executed successfully 2018-09-04 03:41:04,562 - root - ERROR - **** Uninstalling on server: 172.23.104.107 **** 2018-09-04 03:41:04,563 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:41:04,563 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:41:04,662 - root - INFO - command executed successfully 2018-09-04 03:41:04,664 - root - ERROR - **** Uninstalling on server: 172.23.104.109 **** 2018-09-04 03:41:04,665 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:41:04,665 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:41:05,109 - root - INFO - running command.raw on 172.23.104.90: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:41:05,213 - root - INFO - command executed successfully 2018-09-04 03:41:05,215 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:41:05,215 - root - INFO - running command.raw on 172.23.104.90: ipcrm 2018-09-04 03:41:05,306 - root - INFO - command executed successfully 2018-09-04 03:41:05,605 - root - INFO - running command.raw on 172.23.104.106: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:41:05,707 - root - INFO - command executed successfully 2018-09-04 03:41:05,707 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:41:05,708 - root - INFO - running command.raw on 172.23.104.106: ipcrm 2018-09-04 03:41:05,794 - root - INFO - command executed successfully thread uninstaller-thread-172.23.104.106 finished 2018-09-04 03:41:06,747 - root - INFO - running command.raw on 172.23.104.108: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:41:06,901 - root - INFO - command executed successfully 2018-09-04 03:41:06,902 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:41:06,903 - root - INFO - running command.raw on 172.23.104.108: ipcrm 2018-09-04 03:41:06,947 - root - INFO - running command.raw on 172.23.104.107: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:41:06,977 - root - INFO - running command.raw on 172.23.104.109: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:41:07,037 - root - INFO - command executed successfully 2018-09-04 03:41:07,106 - root - INFO - command executed successfully 2018-09-04 03:41:07,107 - root - INFO - command executed successfully 2018-09-04 03:41:07,108 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:41:07,110 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:41:07,111 - root - INFO - running command.raw on 172.23.104.109: ipcrm 2018-09-04 03:41:07,112 - root - INFO - running command.raw on 172.23.104.107: ipcrm 2018-09-04 03:41:07,242 - root - INFO - command executed successfully 2018-09-04 03:41:07,248 - root - INFO - command executed successfully thread uninstaller-thread-172.23.104.107 finished thread uninstaller-thread-172.23.104.108 finished thread uninstaller-thread-172.23.104.109 finished thread uninstaller-thread-172.23.104.90 finished 2018-09-04 03:41:07,256 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:41:07,389 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:41:07,739 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:08,151 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:41:08,318 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:41:08,660 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:09,171 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:41:09,338 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:41:09,636 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:10,119 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:41:10,318 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:41:10,687 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:11,201 - root - INFO - connecting to 172.23.104.107 with username:root password:couchbase ssh_key: 2018-09-04 03:41:11,398 - root - INFO - Connected to 172.23.104.107 2018-09-04 03:41:12,172 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:13,074 - root - INFO - connecting to 172.23.104.108 with username:root password:couchbase ssh_key: 2018-09-04 03:41:13,271 - root - INFO - Connected to 172.23.104.108 2018-09-04 03:41:13,955 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:14,860 - root - INFO - connecting to 172.23.104.109 with username:root password:couchbase ssh_key: 2018-09-04 03:41:15,093 - root - INFO - Connected to 172.23.104.109 2018-09-04 03:41:15,819 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:16,799 - root - INFO - connecting to 172.23.104.90 with username:root password:couchbase ssh_key: 2018-09-04 03:41:16,998 - root - INFO - Connected to 172.23.104.90 2018-09-04 03:41:17,560 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:18,053 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:41:18,054 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:41:18,054 - root - INFO - inside try 2018-09-04 03:41:18,056 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:41:18,056 - root - INFO - inside try 2018-09-04 03:41:18,058 - root - INFO - inside try 2018-09-04 03:41:18,059 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:41:18,064 - root - INFO - inside try 2018-09-04 03:41:18,066 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:41:18,069 - root - INFO - inside try 2018-09-04 03:41:18,070 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:41:18,074 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:41:18,076 - root - INFO - inside try 2018-09-04 03:41:18,078 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:41:18,084 - root - INFO - inside try 2018-09-04 03:41:18,088 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:41:18,089 - root - INFO - inside try 2018-09-04 03:41:18,120 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:41:18,124 - root - INFO - connecting to 172.23.104.107 with username:root password:couchbase ssh_key: 2018-09-04 03:41:18,128 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:41:18,135 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:41:18,163 - root - INFO - connecting to 172.23.104.109 with username:root password:couchbase ssh_key: 2018-09-04 03:41:18,166 - root - INFO - connecting to 172.23.104.108 with username:root password:couchbase ssh_key: 2018-09-04 03:41:18,173 - root - INFO - connecting to 172.23.104.90 with username:root password:couchbase ssh_key: 2018-09-04 03:41:18,321 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:41:18,326 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:41:18,328 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:41:18,358 - root - INFO - Connected to 172.23.104.107 2018-09-04 03:41:18,395 - root - INFO - Connected to 172.23.104.90 2018-09-04 03:41:18,396 - root - INFO - Connected to 172.23.104.109 2018-09-04 03:41:18,398 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:41:18,426 - root - INFO - Connected to 172.23.104.108 2018-09-04 03:41:18,634 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:18,733 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:18,778 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:18,825 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:19,024 - root - INFO - It could be other operating system. Go to check at other location *** OS version of this server 172.23.104.105 is CentOS 7 *** 2018-09-04 03:41:19,271 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: *** OS version of this server 172.23.104.104 is CentOS 7 *** 2018-09-04 03:41:19,301 - root - INFO - It could be other operating system. Go to check at other location *** OS version of this server 172.23.104.103 is CentOS 7 *** 2018-09-04 03:41:19,350 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:41:19,361 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:19,395 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: *** OS version of this server 172.23.104.106 is CentOS 7 *** 2018-09-04 03:41:19,425 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:19,483 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:41:19,527 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:41:19,552 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:41:19,605 - root - INFO - Connected to 172.23.104.103 *** OS version of this server 172.23.104.90 is CentOS 7 *** 2018-09-04 03:41:19,648 - root - INFO - connecting to 172.23.104.90 with username:root password:couchbase ssh_key: 2018-09-04 03:41:19,691 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:41:19,834 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:19,854 - root - INFO - Connected to 172.23.104.90 2018-09-04 03:41:19,960 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:20,005 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:20,102 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:20,278 - root - INFO - Check if url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is ok *** OS version of this server 172.23.104.109 is CentOS 7 *** *** OS version of this server 172.23.104.107 is CentOS 7 *** 2018-09-04 03:41:20,343 - root - INFO - connecting to 172.23.104.107 with username:root password:couchbase ssh_key: 2018-09-04 03:41:20,353 - root - INFO - This url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is live *** OS version of this server 172.23.104.108 is CentOS 7 *** 2018-09-04 03:41:20,401 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:41:20,420 - root - INFO - connecting to 172.23.104.109 with username:root password:couchbase ssh_key: 2018-09-04 03:41:20,437 - root - INFO - connecting to 172.23.104.108 with username:root password:couchbase ssh_key: 2018-09-04 03:41:20,462 - root - INFO - Check if url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is ok 2018-09-04 03:41:20,490 - root - INFO - Check if url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is ok 2018-09-04 03:41:20,576 - root - INFO - This url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is live 2018-09-04 03:41:20,583 - root - INFO - This url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is live 2018-09-04 03:41:20,590 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:20,617 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:41:20,623 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:41:20,634 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:41:20,638 - root - INFO - Connected to 172.23.104.107 2018-09-04 03:41:20,669 - root - INFO - Connected to 172.23.104.109 2018-09-04 03:41:20,679 - root - INFO - Check if url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is ok 2018-09-04 03:41:20,699 - root - INFO - This url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is live 2018-09-04 03:41:20,715 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:41:20,756 - root - INFO - Connected to 172.23.104.108 2018-09-04 03:41:20,767 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:41:20,805 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:41:20,916 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:41:20,940 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:21,159 - root - INFO - Check if url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is ok 2018-09-04 03:41:21,171 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:21,249 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:21,264 - root - INFO - This url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is live 2018-09-04 03:41:21,291 - root - INFO - connecting to 172.23.104.90 with username:root password:couchbase ssh_key: 2018-09-04 03:41:21,385 - root - INFO - inside else 2018-09-04 03:41:21,389 - root - INFO - running command.raw on 172.23.104.105: /sbin/iptables -F 2018-09-04 03:41:21,446 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:21,484 - root - INFO - command executed successfully 2018-09-04 03:41:21,485 - root - INFO - running command.raw on 172.23.104.105: /sbin/iptables -t nat -F 2018-09-04 03:41:21,531 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:21,549 - root - INFO - Connected to 172.23.104.90 2018-09-04 03:41:21,555 - root - INFO - command executed successfully 2018-09-04 03:41:21,559 - root - INFO - Connect to node: 172.23.104.105 as user: root 2018-09-04 03:41:21,620 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:21,632 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:21,711 - root - INFO - inside else 2018-09-04 03:41:21,712 - root - INFO - running command.raw on 172.23.104.103: /sbin/iptables -F 2018-09-04 03:41:21,752 - root - INFO - Connected to 172.23.104.105 as root 2018-09-04 03:41:21,756 - root - INFO - running command.raw on 172.23.104.105: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:41:21,794 - root - INFO - command executed successfully 2018-09-04 03:41:21,796 - root - INFO - running command.raw on 172.23.104.103: /sbin/iptables -t nat -F 2018-09-04 03:41:21,825 - root - INFO - inside else 2018-09-04 03:41:21,826 - root - INFO - running command.raw on 172.23.104.104: /sbin/iptables -F 2018-09-04 03:41:21,851 - root - INFO - command executed successfully 2018-09-04 03:41:21,852 - root - INFO - Connect to node: 172.23.104.103 as user: root 2018-09-04 03:41:21,910 - root - INFO - command executed successfully 2018-09-04 03:41:21,912 - root - INFO - running command.raw on 172.23.104.104: /sbin/iptables -t nat -F 2018-09-04 03:41:21,988 - root - INFO - Connected to 172.23.104.103 as root 2018-09-04 03:41:21,988 - root - INFO - running command.raw on 172.23.104.103: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:41:21,997 - root - INFO - command executed successfully 2018-09-04 03:41:21,998 - root - INFO - Connect to node: 172.23.104.104 as user: root 2018-09-04 03:41:22,007 - root - INFO - command executed successfully 2018-09-04 03:41:22,010 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm’: No such file or directory 2018-09-04 03:41:22,010 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:41:22,013 - root - ERROR - rm: cannot remove ‘cb_xvmb’: Is a directory 2018-09-04 03:41:22,014 - root - ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory 2018-09-04 03:41:22,017 - root - ERROR - rm: cannot remove ‘systemd-private-4ac815aee3424d938d2263668d460ba5-ntpd.service-U2ye4B’: Is a directory 2018-09-04 03:41:22,019 - root - ERROR - mv: cannot stat ‘cb_xvmb/*’: No such file or directory 2018-09-04 03:41:22,020 - root - INFO - running command.raw on 172.23.104.105: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:41:22,053 - root - INFO - inside else 2018-09-04 03:41:22,055 - root - INFO - running command.raw on 172.23.104.106: /sbin/iptables -F 2018-09-04 03:41:22,156 - root - INFO - command executed successfully 2018-09-04 03:41:22,158 - root - INFO - running command.raw on 172.23.104.106: /sbin/iptables -t nat -F 2018-09-04 03:41:22,176 - root - INFO - Connected to 172.23.104.104 as root 2018-09-04 03:41:22,178 - root - INFO - running command.raw on 172.23.104.104: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:41:22,183 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:22,250 - root - INFO - command executed successfully 2018-09-04 03:41:22,252 - root - INFO - Connect to node: 172.23.104.106 as user: root 2018-09-04 03:41:22,269 - root - INFO - command executed successfully 2018-09-04 03:41:22,270 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm’: No such file or directory 2018-09-04 03:41:22,271 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:41:22,271 - root - ERROR - rm: cannot remove ‘backups’: Is a directory 2018-09-04 03:41:22,272 - root - ERROR - rm: cannot remove ‘cb_RroF’: Is a directory 2018-09-04 03:41:22,272 - root - ERROR - rm: cannot remove ‘systemd-private-83a29c879dd548ddb57f9a0683aa9254-ntpd.service-G5roHa’: Is a directory 2018-09-04 03:41:22,272 - root - ERROR - rm: cannot remove ‘tmp.3wHdyvLuME’: Is a directory 2018-09-04 03:41:22,273 - root - ERROR - mv: cannot stat ‘cb_RroF/*’: No such file or directory 2018-09-04 03:41:22,273 - root - INFO - running command.raw on 172.23.104.103: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:41:22,451 - root - INFO - command executed successfully 2018-09-04 03:41:22,453 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm’: No such file or directory 2018-09-04 03:41:22,454 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:41:22,454 - root - ERROR - rm: cannot remove ‘cb_O3bv’: Is a directory 2018-09-04 03:41:22,455 - root - ERROR - rm: cannot remove ‘systemd-private-dbeed978d97747f6860a8195194b6ada-ntpd.service-A2MgZO’: Is a directory 2018-09-04 03:41:22,455 - root - ERROR - mv: cannot stat ‘cb_O3bv/*’: No such file or directory 2018-09-04 03:41:22,455 - root - INFO - running command.raw on 172.23.104.104: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:41:22,458 - root - INFO - Check if url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is ok 2018-09-04 03:41:22,459 - root - INFO - Connected to 172.23.104.106 as root 2018-09-04 03:41:22,462 - root - INFO - running command.raw on 172.23.104.106: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:41:22,475 - root - INFO - Check if url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is ok 2018-09-04 03:41:22,523 - root - INFO - This url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is live 2018-09-04 03:41:22,529 - root - INFO - connecting to 172.23.104.109 with username:root password:couchbase ssh_key: 2018-09-04 03:41:22,543 - root - INFO - Check if url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is ok 2018-09-04 03:41:22,546 - root - INFO - This url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is live 2018-09-04 03:41:22,551 - root - INFO - connecting to 172.23.104.107 with username:root password:couchbase ssh_key: 2018-09-04 03:41:22,611 - root - INFO - This url http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm is live 2018-09-04 03:41:22,615 - root - INFO - connecting to 172.23.104.108 with username:root password:couchbase ssh_key: 2018-09-04 03:41:22,691 - root - INFO - inside else 2018-09-04 03:41:22,693 - root - INFO - running command.raw on 172.23.104.90: /sbin/iptables -F 2018-09-04 03:41:22,728 - root - INFO - Connected to 172.23.104.109 2018-09-04 03:41:22,751 - root - INFO - Connected to 172.23.104.107 2018-09-04 03:41:22,783 - root - INFO - command executed successfully 2018-09-04 03:41:22,784 - root - INFO - running command.raw on 172.23.104.90: /sbin/iptables -t nat -F 2018-09-04 03:41:22,814 - root - INFO - command executed successfully 2018-09-04 03:41:22,815 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm’: No such file or directory 2018-09-04 03:41:22,815 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:41:22,817 - root - ERROR - rm: cannot remove ‘cb_GEIc’: Is a directory 2018-09-04 03:41:22,817 - root - INFO - Connected to 172.23.104.108 2018-09-04 03:41:22,818 - root - ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory 2018-09-04 03:41:22,820 - root - ERROR - rm: cannot remove ‘systemd-private-03a85ff6ef3c4dd3b00e67c9fb47abbf-ntpd.service-JQIjQj’: Is a directory 2018-09-04 03:41:22,820 - root - ERROR - rm: cannot remove ‘tmpOWeU1l’: Is a directory 2018-09-04 03:41:22,821 - root - ERROR - mv: cannot stat ‘cb_GEIc/*’: No such file or directory 2018-09-04 03:41:22,821 - root - INFO - running command.raw on 172.23.104.106: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:41:22,873 - root - INFO - command executed successfully 2018-09-04 03:41:22,875 - root - INFO - Connect to node: 172.23.104.90 as user: root 2018-09-04 03:41:23,073 - root - INFO - Connected to 172.23.104.90 as root 2018-09-04 03:41:23,073 - root - INFO - running command.raw on 172.23.104.90: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:41:23,451 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:23,512 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:23,545 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:41:23,628 - root - INFO - command executed successfully 2018-09-04 03:41:23,631 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm’: No such file or directory 2018-09-04 03:41:23,632 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:41:23,633 - root - ERROR - rm: cannot remove ‘cb_i17i’: Is a directory 2018-09-04 03:41:23,633 - root - ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory 2018-09-04 03:41:23,633 - root - ERROR - rm: cannot remove ‘systemd-private-7bce9ed22f6242989afe9a0baa900bfe-ntpd.service-IaU83M’: Is a directory 2018-09-04 03:41:23,634 - root - ERROR - rm: cannot remove ‘tmpCwTbRu’: Is a directory 2018-09-04 03:41:23,634 - root - ERROR - rm: cannot remove ‘tmpQogqe0’: Is a directory 2018-09-04 03:41:23,634 - root - ERROR - mv: cannot stat ‘cb_i17i/*’: No such file or directory 2018-09-04 03:41:23,634 - root - INFO - running command.raw on 172.23.104.90: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:41:24,249 - root - INFO - inside else 2018-09-04 03:41:24,249 - root - INFO - running command.raw on 172.23.104.109: /sbin/iptables -F 2018-09-04 03:41:24,345 - root - INFO - inside else 2018-09-04 03:41:24,345 - root - INFO - running command.raw on 172.23.104.108: /sbin/iptables -F 2018-09-04 03:41:24,372 - root - INFO - inside else 2018-09-04 03:41:24,372 - root - INFO - running command.raw on 172.23.104.107: /sbin/iptables -F 2018-09-04 03:41:24,385 - root - INFO - command executed successfully 2018-09-04 03:41:24,386 - root - INFO - running command.raw on 172.23.104.109: /sbin/iptables -t nat -F 2018-09-04 03:41:24,479 - root - INFO - command executed successfully 2018-09-04 03:41:24,481 - root - INFO - running command.raw on 172.23.104.108: /sbin/iptables -t nat -F 2018-09-04 03:41:24,511 - root - INFO - command executed successfully 2018-09-04 03:41:24,512 - root - INFO - running command.raw on 172.23.104.107: /sbin/iptables -t nat -F 2018-09-04 03:41:24,526 - root - INFO - command executed successfully 2018-09-04 03:41:24,527 - root - INFO - Connect to node: 172.23.104.109 as user: root 2018-09-04 03:41:24,617 - root - INFO - command executed successfully 2018-09-04 03:41:24,618 - root - INFO - Connect to node: 172.23.104.108 as user: root 2018-09-04 03:41:24,651 - root - INFO - command executed successfully 2018-09-04 03:41:24,652 - root - INFO - Connect to node: 172.23.104.107 as user: root 2018-09-04 03:41:24,761 - root - INFO - Connected to 172.23.104.109 as root 2018-09-04 03:41:24,761 - root - INFO - running command.raw on 172.23.104.109: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:41:24,856 - root - INFO - Connected to 172.23.104.108 as root 2018-09-04 03:41:24,857 - root - INFO - running command.raw on 172.23.104.108: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:41:24,887 - root - INFO - Connected to 172.23.104.107 as root 2018-09-04 03:41:24,888 - root - INFO - running command.raw on 172.23.104.107: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:41:25,428 - root - INFO - command executed successfully 2018-09-04 03:41:25,429 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm’: No such file or directory 2018-09-04 03:41:25,430 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:41:25,430 - root - ERROR - rm: cannot remove ‘cb_ejKE’: Is a directory 2018-09-04 03:41:25,431 - root - ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory 2018-09-04 03:41:25,431 - root - ERROR - rm: cannot remove ‘systemd-private-5bed7a9972d44759b1a0e2b86dbb69d5-ntpd.service-tVnqjF’: Is a directory 2018-09-04 03:41:25,431 - root - ERROR - mv: cannot stat ‘cb_ejKE/*’: No such file or directory 2018-09-04 03:41:25,431 - root - INFO - running command.raw on 172.23.104.109: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:41:25,557 - root - INFO - command executed successfully 2018-09-04 03:41:25,559 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm’: No such file or directory 2018-09-04 03:41:25,559 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:41:25,559 - root - ERROR - rm: cannot remove ‘cb_tbec’: Is a directory 2018-09-04 03:41:25,559 - root - ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory 2018-09-04 03:41:25,560 - root - ERROR - rm: cannot remove ‘systemd-private-7ae56a5410ee4dc19682ceb9314c3682-ntpd.service-vuFzP4’: Is a directory 2018-09-04 03:41:25,560 - root - ERROR - mv: cannot stat ‘cb_tbec/*’: No such file or directory 2018-09-04 03:41:25,560 - root - INFO - running command.raw on 172.23.104.107: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:41:25,571 - root - INFO - command executed successfully 2018-09-04 03:41:25,572 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm’: No such file or directory 2018-09-04 03:41:25,573 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:41:25,573 - root - ERROR - rm: cannot remove ‘atop.d’: Is a directory 2018-09-04 03:41:25,573 - root - ERROR - rm: cannot remove ‘cb_2PIX’: Is a directory 2018-09-04 03:41:25,573 - root - ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory 2018-09-04 03:41:25,573 - root - ERROR - rm: cannot remove ‘systemd-private-a24af647e7994a019890df6f2a98ecf0-ntpd.service-wP721k’: Is a directory 2018-09-04 03:41:25,573 - root - ERROR - mv: cannot stat ‘cb_2PIX/*’: No such file or directory 2018-09-04 03:41:25,574 - root - INFO - running command.raw on 172.23.104.108: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:41:41,069 - root - INFO - command executed successfully 2018-09-04 03:41:41,071 - root - INFO - total 243M 2018-09-04 03:41:41,071 - root - INFO - -rw-r--r--. 1 root root 243M Sep 3 04:37 couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm 2018-09-04 03:41:41,071 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Sep 3 05:51 hsperfdata_couchbase 2018-09-04 03:41:41,072 - root - INFO - drwx------. 3 root root 16 Jul 2 13:28 systemd-private-03a85ff6ef3c4dd3b00e67c9fb47abbf-ntpd.service-JQIjQj 2018-09-04 03:41:41,072 - root - INFO - drwx------. 2 root root 4.0K Aug 27 09:40 tmpOWeU1l 2018-09-04 03:41:41,168 - root - INFO - *****install server *** 2018-09-04 03:41:41,169 - root - INFO - deliverable_type : rpm 2018-09-04 03:41:41,169 - root - INFO - /tmp/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:41:41,170 - root - INFO - running command.raw on 172.23.104.106: /sbin/sysctl vm.swappiness=0 2018-09-04 03:41:41,257 - root - INFO - command executed successfully 2018-09-04 03:41:41,258 - root - INFO - running command.raw on 172.23.104.106: cat /etc/redhat-release 2018-09-04 03:41:41,349 - root - INFO - command executed successfully 2018-09-04 03:41:41,349 - root - INFO - CentOS Linux release 7.2.1511 (Core) 2018-09-04 03:41:41,350 - root - INFO - no need to install pkgconfig 2018-09-04 03:41:44,292 - root - INFO - command executed successfully 2018-09-04 03:41:44,294 - root - INFO - total 243M 2018-09-04 03:41:44,295 - root - INFO - -rw-r--r--. 1 root root 243M Sep 3 04:37 couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm 2018-09-04 03:41:44,295 - root - INFO - drwx------. 3 root root 16 Aug 26 04:30 systemd-private-dbeed978d97747f6860a8195194b6ada-ntpd.service-A2MgZO 2018-09-04 03:41:44,381 - root - INFO - *****install server *** 2018-09-04 03:41:44,381 - root - INFO - deliverable_type : rpm 2018-09-04 03:41:44,382 - root - INFO - /tmp/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:41:44,383 - root - INFO - running command.raw on 172.23.104.104: /sbin/sysctl vm.swappiness=0 2018-09-04 03:41:44,464 - root - INFO - command executed successfully 2018-09-04 03:41:44,465 - root - INFO - running command.raw on 172.23.104.104: cat /etc/redhat-release 2018-09-04 03:41:44,541 - root - INFO - command executed successfully 2018-09-04 03:41:44,543 - root - INFO - CentOS Linux release 7.4.1708 (Core) 2018-09-04 03:41:44,543 - root - INFO - no need to install pkgconfig 2018-09-04 03:41:46,581 - root - INFO - command executed successfully 2018-09-04 03:41:46,582 - root - INFO - total 243M 2018-09-04 03:41:46,583 - root - INFO - drwx------. 2 root root 22 Aug 30 03:52 atop.d 2018-09-04 03:41:46,584 - root - INFO - -rw-r--r--. 1 root root 243M Sep 3 04:37 couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm 2018-09-04 03:41:46,584 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Aug 26 19:33 hsperfdata_couchbase 2018-09-04 03:41:46,585 - root - INFO - drwx------. 3 root root 16 Aug 31 14:25 systemd-private-a24af647e7994a019890df6f2a98ecf0-ntpd.service-wP721k 2018-09-04 03:41:46,729 - root - INFO - *****install server *** 2018-09-04 03:41:46,730 - root - INFO - deliverable_type : rpm 2018-09-04 03:41:46,730 - root - INFO - /tmp/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:41:46,731 - root - INFO - running command.raw on 172.23.104.108: /sbin/sysctl vm.swappiness=0 2018-09-04 03:41:46,877 - root - INFO - command executed successfully 2018-09-04 03:41:46,878 - root - INFO - running command.raw on 172.23.104.108: cat /etc/redhat-release 2018-09-04 03:41:47,009 - root - INFO - command executed successfully 2018-09-04 03:41:47,011 - root - INFO - CentOS Linux release 7.4.1708 (Core) 2018-09-04 03:41:47,011 - root - INFO - no need to install pkgconfig 2018-09-04 03:41:52,725 - root - INFO - command executed successfully 2018-09-04 03:41:52,726 - root - INFO - total 243M 2018-09-04 03:41:52,727 - root - INFO - drwxrwx---. 4 root root 45 Sep 3 04:29 backups 2018-09-04 03:41:52,727 - root - INFO - -rw-r--r--. 1 root root 243M Sep 3 04:37 couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm 2018-09-04 03:41:52,727 - root - INFO - drwx------. 3 root root 16 Aug 28 12:33 systemd-private-83a29c879dd548ddb57f9a0683aa9254-ntpd.service-G5roHa 2018-09-04 03:41:52,727 - root - INFO - drwx------. 2 root root 6 Aug 24 18:03 tmp.3wHdyvLuME 2018-09-04 03:41:52,814 - root - INFO - *****install server *** 2018-09-04 03:41:52,815 - root - INFO - deliverable_type : rpm 2018-09-04 03:41:52,815 - root - INFO - /tmp/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:41:52,815 - root - INFO - running command.raw on 172.23.104.103: /sbin/sysctl vm.swappiness=0 2018-09-04 03:41:52,897 - root - INFO - command executed successfully 2018-09-04 03:41:52,899 - root - INFO - running command.raw on 172.23.104.103: cat /etc/redhat-release 2018-09-04 03:41:52,954 - root - INFO - command executed successfully 2018-09-04 03:41:52,955 - root - INFO - CentOS Linux release 7.5.1804 (Core) 2018-09-04 03:41:52,956 - root - INFO - no need to install pkgconfig 2018-09-04 03:41:56,451 - root - INFO - command executed successfully 2018-09-04 03:41:56,452 - root - INFO - total 243M 2018-09-04 03:41:56,452 - root - INFO - -rw-r--r--. 1 root root 243M Sep 3 04:37 couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm 2018-09-04 03:41:56,455 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Sep 3 03:25 hsperfdata_couchbase 2018-09-04 03:41:56,455 - root - INFO - drwx------. 3 root root 16 Sep 1 10:33 systemd-private-5bed7a9972d44759b1a0e2b86dbb69d5-ntpd.service-tVnqjF 2018-09-04 03:41:56,602 - root - INFO - *****install server *** 2018-09-04 03:41:56,602 - root - INFO - deliverable_type : rpm 2018-09-04 03:41:56,602 - root - INFO - /tmp/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:41:56,603 - root - INFO - running command.raw on 172.23.104.109: /sbin/sysctl vm.swappiness=0 2018-09-04 03:41:56,743 - root - INFO - command executed successfully 2018-09-04 03:41:56,744 - root - INFO - running command.raw on 172.23.104.109: cat /etc/redhat-release 2018-09-04 03:41:56,882 - root - INFO - command executed successfully 2018-09-04 03:41:56,883 - root - INFO - CentOS Linux release 7.4.1708 (Core) 2018-09-04 03:41:56,883 - root - INFO - no need to install pkgconfig 2018-09-04 03:42:01,990 - root - INFO - command executed successfully 2018-09-04 03:42:01,992 - root - INFO - total 243M 2018-09-04 03:42:01,992 - root - INFO - -rw-r--r--. 1 root root 243M Sep 3 04:37 couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm 2018-09-04 03:42:01,993 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Aug 26 19:33 hsperfdata_couchbase 2018-09-04 03:42:01,994 - root - INFO - drwx------. 3 root root 16 Aug 31 22:36 systemd-private-4ac815aee3424d938d2263668d460ba5-ntpd.service-U2ye4B 2018-09-04 03:42:02,073 - root - INFO - command executed successfully 2018-09-04 03:42:02,074 - root - INFO - total 243M 2018-09-04 03:42:02,076 - root - INFO - -rw-r--r--. 1 root root 243M Sep 3 04:37 couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm 2018-09-04 03:42:02,076 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Aug 26 19:33 hsperfdata_couchbase 2018-09-04 03:42:02,078 - root - INFO - drwx------. 3 root root 16 Aug 28 05:20 systemd-private-7ae56a5410ee4dc19682ceb9314c3682-ntpd.service-vuFzP4 2018-09-04 03:42:02,087 - root - INFO - *****install server *** 2018-09-04 03:42:02,088 - root - INFO - deliverable_type : rpm 2018-09-04 03:42:02,088 - root - INFO - /tmp/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:42:02,089 - root - INFO - running command.raw on 172.23.104.105: /sbin/sysctl vm.swappiness=0 2018-09-04 03:42:02,173 - root - INFO - command executed successfully 2018-09-04 03:42:02,173 - root - INFO - running command.raw on 172.23.104.105: cat /etc/redhat-release 2018-09-04 03:42:02,227 - root - INFO - *****install server *** 2018-09-04 03:42:02,228 - root - INFO - deliverable_type : rpm 2018-09-04 03:42:02,228 - root - INFO - /tmp/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:42:02,228 - root - INFO - running command.raw on 172.23.104.107: /sbin/sysctl vm.swappiness=0 2018-09-04 03:42:02,251 - root - INFO - command executed successfully 2018-09-04 03:42:02,251 - root - INFO - CentOS Linux release 7.4.1708 (Core) 2018-09-04 03:42:02,252 - root - INFO - no need to install pkgconfig 2018-09-04 03:42:02,369 - root - INFO - command executed successfully 2018-09-04 03:42:02,369 - root - INFO - running command.raw on 172.23.104.107: cat /etc/redhat-release 2018-09-04 03:42:02,512 - root - INFO - command executed successfully 2018-09-04 03:42:02,513 - root - INFO - CentOS Linux release 7.4.1708 (Core) 2018-09-04 03:42:02,514 - root - INFO - no need to install pkgconfig 2018-09-04 03:42:08,582 - root - INFO - command executed successfully 2018-09-04 03:42:08,583 - root - INFO - total 243M 2018-09-04 03:42:08,583 - root - INFO - -rw-r--r--. 1 root root 243M Sep 3 04:37 couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm 2018-09-04 03:42:08,583 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Sep 3 05:51 hsperfdata_couchbase 2018-09-04 03:42:08,584 - root - INFO - drwx------. 3 root root 16 Jul 2 13:30 systemd-private-7bce9ed22f6242989afe9a0baa900bfe-ntpd.service-IaU83M 2018-09-04 03:42:08,584 - root - INFO - drwx------. 2 root root 26 Aug 27 10:10 tmpCwTbRu 2018-09-04 03:42:08,584 - root - INFO - drwx------. 2 root root 4.0K Aug 27 06:54 tmpQogqe0 2018-09-04 03:42:08,688 - root - INFO - *****install server *** 2018-09-04 03:42:08,689 - root - INFO - deliverable_type : rpm 2018-09-04 03:42:08,689 - root - INFO - /tmp/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:42:08,689 - root - INFO - running command.raw on 172.23.104.90: /sbin/sysctl vm.swappiness=0 2018-09-04 03:42:08,782 - root - INFO - command executed successfully 2018-09-04 03:42:08,783 - root - INFO - running command.raw on 172.23.104.90: cat /etc/redhat-release 2018-09-04 03:42:08,849 - root - INFO - command executed successfully 2018-09-04 03:42:08,851 - root - INFO - CentOS Linux release 7.4.1708 (Core) 2018-09-04 03:42:08,852 - root - INFO - no need to install pkgconfig 2018-09-04 03:42:10,598 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:42:10,747 - root - INFO - Running systemd command on this server 2018-09-04 03:42:10,748 - root - INFO - running command.raw on 172.23.104.104: systemctl start couchbase-server.service 2018-09-04 03:42:10,771 - root - INFO - command executed successfully 2018-09-04 03:42:10,827 - root - INFO - wait 5 seconds for Couchbase server to start 2018-09-04 03:42:19,441 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:42:19,582 - root - INFO - Running systemd command on this server 2018-09-04 03:42:19,583 - root - INFO - running command.raw on 172.23.104.103: systemctl start couchbase-server.service 2018-09-04 03:42:19,606 - root - INFO - command executed successfully 2018-09-04 03:42:19,685 - root - INFO - wait 5 seconds for Couchbase server to start thread installer-thread-172.23.104.103 finished thread installer-thread-172.23.104.104 finished 2018-09-04 03:42:28,388 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:42:28,608 - root - INFO - Running systemd command on this server 2018-09-04 03:42:28,609 - root - INFO - running command.raw on 172.23.104.105: systemctl start couchbase-server.service 2018-09-04 03:42:28,628 - root - INFO - command executed successfully 2018-09-04 03:42:28,706 - root - INFO - wait 5 seconds for Couchbase server to start thread installer-thread-172.23.104.105 finished 2018-09-04 03:42:38,405 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:42:38,658 - root - INFO - Running systemd command on this server 2018-09-04 03:42:38,659 - root - INFO - running command.raw on 172.23.104.106: systemctl start couchbase-server.service 2018-09-04 03:42:38,701 - root - INFO - command executed successfully 2018-09-04 03:42:38,790 - root - INFO - wait 5 seconds for Couchbase server to start thread installer-thread-172.23.104.106 finished 2018-09-04 03:42:57,662 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:42:58,084 - root - INFO - Running systemd command on this server 2018-09-04 03:42:58,085 - root - INFO - running command.raw on 172.23.104.108: systemctl start couchbase-server.service 2018-09-04 03:42:58,174 - root - INFO - command executed successfully 2018-09-04 03:42:58,244 - root - INFO - wait 5 seconds for Couchbase server to start 2018-09-04 03:43:05,100 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:43:05,290 - root - INFO - Running systemd command on this server 2018-09-04 03:43:05,291 - root - INFO - running command.raw on 172.23.104.90: systemctl start couchbase-server.service 2018-09-04 03:43:05,327 - root - INFO - command executed successfully 2018-09-04 03:43:05,417 - root - INFO - wait 5 seconds for Couchbase server to start 2018-09-04 03:43:07,681 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:43:08,135 - root - INFO - Running systemd command on this server 2018-09-04 03:43:08,136 - root - INFO - running command.raw on 172.23.104.109: systemctl start couchbase-server.service 2018-09-04 03:43:08,223 - root - INFO - command executed successfully 2018-09-04 03:43:08,299 - root - INFO - wait 5 seconds for Couchbase server to start 2018-09-04 03:43:13,109 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:43:13,516 - root - INFO - Running systemd command on this server 2018-09-04 03:43:13,517 - root - INFO - running command.raw on 172.23.104.107: systemctl start couchbase-server.service 2018-09-04 03:43:13,601 - root - INFO - command executed successfully 2018-09-04 03:43:13,670 - root - INFO - wait 5 seconds for Couchbase server to start thread installer-thread-172.23.104.107 finished thread installer-thread-172.23.104.108 finished thread installer-thread-172.23.104.109 finished thread installer-thread-172.23.104.90 finished 2018-09-04 03:43:18,692 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:43:18,706 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:43:18,736 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:43:18,755 - root - INFO - connecting to 172.23.104.108 with username:root password:couchbase ssh_key: 2018-09-04 03:43:18,764 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:43:18,775 - root - INFO - connecting to 172.23.104.107 with username:root password:couchbase ssh_key: 2018-09-04 03:43:18,776 - root - INFO - connecting to 172.23.104.109 with username:root password:couchbase ssh_key: 2018-09-04 03:43:18,780 - root - INFO - connecting to 172.23.104.90 with username:root password:couchbase ssh_key: 2018-09-04 03:43:18,936 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:43:18,937 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:43:18,944 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:43:19,006 - root - INFO - Connected to 172.23.104.90 2018-09-04 03:43:19,010 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:43:19,014 - root - INFO - Connected to 172.23.104.108 2018-09-04 03:43:19,032 - root - INFO - Connected to 172.23.104.109 2018-09-04 03:43:19,066 - root - INFO - Connected to 172.23.104.107 2018-09-04 03:43:19,347 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:19,394 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:19,396 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:19,500 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:19,668 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:19,968 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:19,991 - root - INFO - 172.23.104.104 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:20,039 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:20,084 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:20,127 - root - INFO - 172.23.104.105 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:20,162 - root - INFO - 172.23.104.103 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:20,200 - root - INFO - 172.23.104.106 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:20,323 - root - INFO - 172.23.104.90 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:21,065 - root - INFO - 172.23.104.109 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:21,075 - root - INFO - 172.23.104.108 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:21,190 - root - INFO - 172.23.104.107 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:21,192 - root - ERROR - socket error while connecting to http://172.23.104.107:8091/nodes/self error [Errno 111] Connection refused 2018-09-04 03:43:24,197 - root - ERROR - socket error while connecting to http://172.23.104.107:8091/nodes/self error [Errno 111] Connection refused 2018-09-04 03:43:24,220 - root - INFO - quota for kv: 2145 MB 2018-09-04 03:43:24,221 - root - INFO - pools/default params : memoryQuota=2145 2018-09-04 03:43:24,224 - root - INFO - quota for kv: 2145 MB 2018-09-04 03:43:24,225 - root - INFO - pools/default params : memoryQuota=2145 2018-09-04 03:43:24,228 - root - INFO - quota for kv: 2145 MB 2018-09-04 03:43:24,229 - root - INFO - pools/default params : memoryQuota=2145 2018-09-04 03:43:24,236 - root - INFO - /node/controller/setupServices params on 172.23.104.103: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:43:24,240 - root - INFO - /node/controller/setupServices params on 172.23.104.104: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:43:24,244 - root - INFO - /node/controller/setupServices params on 172.23.104.105: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:43:24,267 - root - INFO - settings/web params on 172.23.104.103:8091:username=Administrator&password=password&port=8091 2018-09-04 03:43:24,270 - root - INFO - settings/web params on 172.23.104.104:8091:username=Administrator&password=password&port=8091 2018-09-04 03:43:24,272 - root - INFO - settings/web params on 172.23.104.105:8091:username=Administrator&password=password&port=8091 2018-09-04 03:43:24,278 - root - INFO - quota for kv: 1883 MB 2018-09-04 03:43:24,279 - root - INFO - pools/default params : memoryQuota=1883 2018-09-04 03:43:24,293 - root - INFO - /node/controller/setupServices params on 172.23.104.106: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:43:24,328 - root - INFO - settings/web params on 172.23.104.106:8091:username=Administrator&password=password&port=8091 2018-09-04 03:43:24,381 - root - INFO - Node version in cluster 6.0.0-35007-enterprise 2018-09-04 03:43:24,382 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:43:24,419 - root - INFO - quota for kv: 2145 MB 2018-09-04 03:43:24,420 - root - INFO - pools/default params : memoryQuota=2145 2018-09-04 03:43:24,439 - root - INFO - Node version in cluster 6.0.0-35007-enterprise 2018-09-04 03:43:24,440 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:43:24,441 - root - INFO - Node version in cluster 6.0.0-35007-enterprise 2018-09-04 03:43:24,445 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:43:24,458 - root - INFO - /node/controller/setupServices params on 172.23.104.90: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:43:24,472 - root - INFO - settings/web params on 172.23.104.90:8091:username=Administrator&password=password&port=8091 2018-09-04 03:43:24,496 - root - INFO - Node version in cluster 6.0.0-35007-enterprise 2018-09-04 03:43:24,496 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:43:24,606 - root - INFO - Node version in cluster 6.0.0-35007-enterprise 2018-09-04 03:43:24,607 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:43:24,653 - root - INFO - check if man installed on vm? 2018-09-04 03:43:24,654 - root - INFO - running command.raw on 172.23.104.103: rpm -qa | grep 'man-db' 2018-09-04 03:43:24,692 - root - INFO - check if man installed on vm? 2018-09-04 03:43:24,694 - root - INFO - running command.raw on 172.23.104.104: rpm -qa | grep 'man-db' 2018-09-04 03:43:24,732 - root - INFO - check if man installed on vm? 2018-09-04 03:43:24,733 - root - INFO - running command.raw on 172.23.104.105: rpm -qa | grep 'man-db' 2018-09-04 03:43:24,779 - root - INFO - check if man installed on vm? 2018-09-04 03:43:24,780 - root - INFO - running command.raw on 172.23.104.106: rpm -qa | grep 'man-db' 2018-09-04 03:43:24,938 - root - INFO - check if man installed on vm? 2018-09-04 03:43:24,939 - root - INFO - running command.raw on 172.23.104.90: rpm -qa | grep 'man-db' 2018-09-04 03:43:25,120 - root - INFO - quota for kv: 2145 MB 2018-09-04 03:43:25,120 - root - INFO - pools/default params : memoryQuota=2145 2018-09-04 03:43:25,133 - root - INFO - /node/controller/setupServices params on 172.23.104.108: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:43:25,148 - root - INFO - settings/web params on 172.23.104.108:8091:username=Administrator&password=password&port=8091 2018-09-04 03:43:25,259 - root - INFO - Node version in cluster 6.0.0-35007-enterprise 2018-09-04 03:43:25,259 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:43:25,402 - root - INFO - command executed successfully 2018-09-04 03:43:25,403 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.104 2018-09-04 03:43:25,404 - root - INFO - running command.raw on 172.23.104.104: unzip 2018-09-04 03:43:25,424 - root - INFO - command executed successfully 2018-09-04 03:43:25,425 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.105 2018-09-04 03:43:25,425 - root - INFO - running command.raw on 172.23.104.105: unzip 2018-09-04 03:43:25,483 - root - INFO - command executed successfully 2018-09-04 03:43:25,485 - root - INFO - unzip command is ready 2018-09-04 03:43:25,485 - root - INFO - Check if ntp is installed 2018-09-04 03:43:25,486 - root - INFO - This OS version centos 7 2018-09-04 03:43:25,486 - root - INFO - running command.raw on 172.23.104.104: systemctl status ntpd 2018-09-04 03:43:25,506 - root - INFO - command executed successfully 2018-09-04 03:43:25,507 - root - INFO - unzip command is ready 2018-09-04 03:43:25,507 - root - INFO - Check if ntp is installed 2018-09-04 03:43:25,509 - root - INFO - This OS version centos 7 2018-09-04 03:43:25,510 - root - INFO - running command.raw on 172.23.104.105: systemctl status ntpd 2018-09-04 03:43:25,524 - root - INFO - command executed successfully 2018-09-04 03:43:25,525 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.103 2018-09-04 03:43:25,525 - root - INFO - running command.raw on 172.23.104.103: unzip 2018-09-04 03:43:25,581 - root - INFO - command executed successfully 2018-09-04 03:43:25,582 - root - INFO - running command.raw on 172.23.104.104: date 2018-09-04 03:43:25,601 - root - INFO - command executed successfully 2018-09-04 03:43:25,602 - root - INFO - running command.raw on 172.23.104.105: date 2018-09-04 03:43:25,610 - root - INFO - command executed successfully 2018-09-04 03:43:25,611 - root - INFO - unzip command is ready 2018-09-04 03:43:25,611 - root - INFO - Check if ntp is installed 2018-09-04 03:43:25,612 - root - INFO - This OS version centos 7 2018-09-04 03:43:25,613 - root - INFO - running command.raw on 172.23.104.103: systemctl status ntpd 2018-09-04 03:43:25,662 - root - INFO - command executed successfully 2018-09-04 03:43:25,683 - root - INFO - command executed successfully 2018-09-04 03:43:25,706 - root - INFO - command executed successfully 2018-09-04 03:43:25,707 - root - INFO - running command.raw on 172.23.104.103: date 2018-09-04 03:43:25,763 - root - INFO - command executed successfully 2018-09-04 03:43:25,851 - root - INFO - ['Tue Sep 4 03:43:25 PDT 2018'] IP: 172.23.104.104 2018-09-04 03:43:25,872 - root - INFO - ['Tue Sep 4 03:43:25 PDT 2018'] IP: 172.23.104.105 2018-09-04 03:43:25,880 - root - INFO - check if man installed on vm? 2018-09-04 03:43:25,881 - root - INFO - running command.raw on 172.23.104.108: rpm -qa | grep 'man-db' 2018-09-04 03:43:25,927 - root - INFO - ['Tue Sep 4 03:43:25 PDT 2018'] IP: 172.23.104.103 2018-09-04 03:43:25,970 - root - INFO - command executed successfully 2018-09-04 03:43:25,972 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.106 2018-09-04 03:43:25,972 - root - INFO - running command.raw on 172.23.104.106: unzip thread initializer-thread-172.23.104.103 finished thread initializer-thread-172.23.104.104 finished thread initializer-thread-172.23.104.105 finished 2018-09-04 03:43:26,059 - root - INFO - command executed successfully 2018-09-04 03:43:26,061 - root - INFO - unzip command is ready 2018-09-04 03:43:26,061 - root - INFO - Check if ntp is installed 2018-09-04 03:43:26,062 - root - INFO - This OS version centos 7 2018-09-04 03:43:26,062 - root - INFO - running command.raw on 172.23.104.106: systemctl status ntpd 2018-09-04 03:43:26,180 - root - INFO - command executed successfully 2018-09-04 03:43:26,182 - root - INFO - running command.raw on 172.23.104.106: date 2018-09-04 03:43:26,268 - root - INFO - command executed successfully 2018-09-04 03:43:26,483 - root - INFO - ['Tue Sep 4 03:43:26 PDT 2018'] IP: 172.23.104.106 thread initializer-thread-172.23.104.106 finished 2018-09-04 03:43:26,532 - root - INFO - quota for kv: 2145 MB 2018-09-04 03:43:26,532 - root - INFO - pools/default params : memoryQuota=2145 2018-09-04 03:43:26,544 - root - INFO - /node/controller/setupServices params on 172.23.104.109: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:43:26,559 - root - INFO - settings/web params on 172.23.104.109:8091:username=Administrator&password=password&port=8091 2018-09-04 03:43:26,636 - root - INFO - command executed successfully 2018-09-04 03:43:26,637 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.90 2018-09-04 03:43:26,638 - root - INFO - running command.raw on 172.23.104.90: unzip 2018-09-04 03:43:26,660 - root - INFO - Node version in cluster 6.0.0-35007-enterprise 2018-09-04 03:43:26,660 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:43:26,730 - root - INFO - command executed successfully 2018-09-04 03:43:26,731 - root - INFO - unzip command is ready 2018-09-04 03:43:26,732 - root - INFO - Check if ntp is installed 2018-09-04 03:43:26,732 - root - INFO - This OS version centos 7 2018-09-04 03:43:26,733 - root - INFO - running command.raw on 172.23.104.90: systemctl status ntpd 2018-09-04 03:43:26,842 - root - INFO - command executed successfully 2018-09-04 03:43:26,843 - root - INFO - running command.raw on 172.23.104.90: date 2018-09-04 03:43:26,929 - root - INFO - command executed successfully 2018-09-04 03:43:27,129 - root - INFO - ['Tue Sep 4 03:43:27 PDT 2018'] IP: 172.23.104.90 2018-09-04 03:43:27,242 - root - INFO - check if man installed on vm? 2018-09-04 03:43:27,244 - root - INFO - running command.raw on 172.23.104.109: rpm -qa | grep 'man-db' 2018-09-04 03:43:28,002 - root - INFO - command executed successfully 2018-09-04 03:43:28,003 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.108 2018-09-04 03:43:28,003 - root - INFO - running command.raw on 172.23.104.108: unzip 2018-09-04 03:43:28,137 - root - INFO - command executed successfully 2018-09-04 03:43:28,137 - root - INFO - unzip command is ready 2018-09-04 03:43:28,138 - root - INFO - Check if ntp is installed 2018-09-04 03:43:28,138 - root - INFO - This OS version centos 7 2018-09-04 03:43:28,138 - root - INFO - running command.raw on 172.23.104.108: systemctl status ntpd 2018-09-04 03:43:28,299 - root - INFO - command executed successfully 2018-09-04 03:43:28,300 - root - INFO - running command.raw on 172.23.104.108: date 2018-09-04 03:43:28,433 - root - INFO - command executed successfully 2018-09-04 03:43:28,709 - root - INFO - ['Tue Sep 4 03:43:28 PDT 2018'] IP: 172.23.104.108 2018-09-04 03:43:29,260 - root - INFO - command executed successfully 2018-09-04 03:43:29,262 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.109 2018-09-04 03:43:29,341 - root - INFO - running command.raw on 172.23.104.109: unzip 2018-09-04 03:43:29,418 - root - INFO - command executed successfully 2018-09-04 03:43:29,419 - root - INFO - unzip command is ready 2018-09-04 03:43:29,420 - root - INFO - Check if ntp is installed 2018-09-04 03:43:29,420 - root - INFO - This OS version centos 7 2018-09-04 03:43:29,421 - root - INFO - running command.raw on 172.23.104.109: systemctl status ntpd 2018-09-04 03:43:29,580 - root - INFO - command executed successfully 2018-09-04 03:43:29,581 - root - INFO - running command.raw on 172.23.104.109: date 2018-09-04 03:43:29,725 - root - INFO - command executed successfully 2018-09-04 03:43:30,062 - root - INFO - ['Tue Sep 4 03:43:30 PDT 2018'] IP: 172.23.104.109 2018-09-04 03:43:32,263 - root - INFO - quota for kv: 2145 MB 2018-09-04 03:43:32,264 - root - INFO - pools/default params : memoryQuota=2145 2018-09-04 03:43:32,278 - root - INFO - /node/controller/setupServices params on 172.23.104.107: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:43:32,293 - root - INFO - settings/web params on 172.23.104.107:8091:username=Administrator&password=password&port=8091 2018-09-04 03:43:32,396 - root - INFO - Node version in cluster 6.0.0-35007-enterprise 2018-09-04 03:43:32,397 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:43:32,934 - root - INFO - check if man installed on vm? 2018-09-04 03:43:32,935 - root - INFO - running command.raw on 172.23.104.107: rpm -qa | grep 'man-db' 2018-09-04 03:43:34,908 - root - INFO - command executed successfully 2018-09-04 03:43:34,910 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.107 2018-09-04 03:43:34,911 - root - INFO - running command.raw on 172.23.104.107: unzip 2018-09-04 03:43:35,049 - root - INFO - command executed successfully 2018-09-04 03:43:35,050 - root - INFO - unzip command is ready 2018-09-04 03:43:35,050 - root - INFO - Check if ntp is installed 2018-09-04 03:43:35,051 - root - INFO - This OS version centos 7 2018-09-04 03:43:35,051 - root - INFO - running command.raw on 172.23.104.107: systemctl status ntpd 2018-09-04 03:43:35,212 - root - INFO - command executed successfully 2018-09-04 03:43:35,213 - root - INFO - running command.raw on 172.23.104.107: date 2018-09-04 03:43:35,347 - root - INFO - command executed successfully 2018-09-04 03:43:35,649 - root - INFO - ['Tue Sep 4 03:43:35 PDT 2018'] IP: 172.23.104.107 thread initializer-thread-172.23.104.107 finished thread initializer-thread-172.23.104.108 finished thread initializer-thread-172.23.104.109 finished thread initializer-thread-172.23.104.90 finished 2018-09-04 03:43:35,657 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:43:35,791 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:43:36,129 - root - INFO - It could be other operating system. Go to check at other location verify installation... 2018-09-04 03:43:36,459 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:43:36,625 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:43:36,999 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:37,423 - root - INFO - 172.23.104.103 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:37,429 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:43:37,563 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:43:37,876 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:38,367 - root - INFO - 172.23.104.104 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:38,373 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:43:38,507 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:43:38,808 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:39,256 - root - INFO - 172.23.104.105 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:39,263 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:43:39,462 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:43:39,809 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:40,330 - root - INFO - 172.23.104.106 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:40,338 - root - INFO - connecting to 172.23.104.107 with username:root password:couchbase ssh_key: 2018-09-04 03:43:40,540 - root - INFO - Connected to 172.23.104.107 2018-09-04 03:43:41,286 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:42,175 - root - INFO - 172.23.104.107 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:42,182 - root - INFO - connecting to 172.23.104.108 with username:root password:couchbase ssh_key: 2018-09-04 03:43:42,382 - root - INFO - Connected to 172.23.104.108 2018-09-04 03:43:43,134 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:44,030 - root - INFO - 172.23.104.108 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:44,038 - root - INFO - connecting to 172.23.104.109 with username:root password:couchbase ssh_key: 2018-09-04 03:43:44,236 - root - INFO - Connected to 172.23.104.109 2018-09-04 03:43:44,951 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:45,812 - root - INFO - 172.23.104.109 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:43:45,820 - root - INFO - connecting to 172.23.104.90 with username:root password:couchbase ssh_key: 2018-09-04 03:43:46,019 - root - INFO - Connected to 172.23.104.90 2018-09-04 03:43:46,598 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:47,215 - root - INFO - 172.23.104.90 **** The version file /opt/couchbase/ VERSION.txt exists + python testrunner.py -i /tmp/upgrade3.ini -p get-cbcollect-info=True -c conf/eventing/eventing_upgrade.conf test_online_upgrade_with_failover_rebalance_with_eventing (eventing.eventing_upgrade.EventingUpgrade) ... filename: conf/eventing/eventing_upgrade.conf prefix: eventing.eventing_upgrade.EventingUpgrade Global Test input params: {'cluster_name': 'upgrade3', 'conf_file': 'conf/eventing/eventing_upgrade.conf', 'get-cbcollect-info': 'True', 'ini': '/tmp/upgrade3.ini', 'num_nodes': 8, 'spec': 'eventing_upgrade'} Logs will be stored at /data/workspace/test_bala_upgrade_new1/logs/testrunner-18-Sep-04_03-43-48/test_1 ./testrunner -i /tmp/upgrade3.ini -p get-cbcollect-info=True -t eventing.eventing_upgrade.EventingUpgrade.test_online_upgrade_with_failover_rebalance_with_eventing,nodes_init=4,dataset=default,groups=simple,skip_cleanup=True,initial_version=5.5.0-2958,doc-per-day=2,upgrade_version=6.0.0-1567 Test Input params: {'cluster_name': 'upgrade3', 'doc-per-day': '2', 'logs_folder': '/data/workspace/test_bala_upgrade_new1/logs/testrunner-18-Sep-04_03-43-48/test_1', 'initial_version': '5.5.0-2958', 'case_number': 1, 'upgrade_version': '6.0.0-1567', 'dataset': 'default', 'skip_cleanup': 'True', 'num_nodes': 8, 'ini': '/tmp/upgrade3.ini', 'groups': 'simple', 'conf_file': 'conf/eventing/eventing_upgrade.conf', 'get-cbcollect-info': 'True', 'spec': 'eventing_upgrade', 'nodes_init': '4'} Run before suite setup for eventing.eventing_upgrade.EventingUpgrade.test_online_upgrade_with_failover_rebalance_with_eventing 2018-09-04 03:43:48,966 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:43:49,177 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:43:49,496 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:43:49,929 - root - INFO - Node version in cluster 6.0.0-35007-enterprise 2018-09-04 03:43:49,947 - root - INFO - Node versions in cluster [u'6.0.0-35007-enterprise'] 2018-09-04 03:43:49,947 - root - INFO - ============== basetestcase setup was started for test #1 test_online_upgrade_with_failover_rebalance_with_eventing============== 2018-09-04 03:43:49,978 - root - INFO - deleting existing buckets [] on 172.23.104.103 2018-09-04 03:43:49,978 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely. 2018-09-04 03:43:51,996 - root - INFO - deleting existing buckets [] on 172.23.104.104 2018-09-04 03:43:51,997 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely. 2018-09-04 03:43:54,015 - root - INFO - deleting existing buckets [] on 172.23.104.105 2018-09-04 03:43:54,015 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely. 2018-09-04 03:43:56,034 - root - INFO - deleting existing buckets [] on 172.23.104.106 2018-09-04 03:43:56,034 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely. 2018-09-04 03:43:58,050 - root - INFO - deleting existing buckets [] on 172.23.104.107 2018-09-04 03:43:58,050 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely. 2018-09-04 03:44:00,067 - root - INFO - deleting existing buckets [] on 172.23.104.108 2018-09-04 03:44:00,068 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely. 2018-09-04 03:44:02,083 - root - INFO - deleting existing buckets [] on 172.23.104.109 2018-09-04 03:44:02,084 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely. 2018-09-04 03:44:04,102 - root - INFO - deleting existing buckets [] on 172.23.104.90 2018-09-04 03:44:04,102 - root - INFO - sleep 2 seconds to make sure all buckets were deleted completely. 2018-09-04 03:44:06,139 - root - INFO - sleep for 10 secs. ... 2018-09-04 03:44:16,159 - root - INFO - waiting for ns_server @ 172.23.104.103:8091 2018-09-04 03:44:16,167 - root - INFO - ns_server @ 172.23.104.103:8091 is running 2018-09-04 03:44:16,173 - root - INFO - waiting for ns_server @ 172.23.104.104:8091 2018-09-04 03:44:16,181 - root - INFO - ns_server @ 172.23.104.104:8091 is running 2018-09-04 03:44:16,188 - root - INFO - waiting for ns_server @ 172.23.104.105:8091 2018-09-04 03:44:16,195 - root - INFO - ns_server @ 172.23.104.105:8091 is running 2018-09-04 03:44:16,205 - root - INFO - waiting for ns_server @ 172.23.104.106:8091 2018-09-04 03:44:16,212 - root - INFO - ns_server @ 172.23.104.106:8091 is running 2018-09-04 03:44:16,223 - root - INFO - waiting for ns_server @ 172.23.104.107:8091 2018-09-04 03:44:16,230 - root - INFO - ns_server @ 172.23.104.107:8091 is running 2018-09-04 03:44:16,238 - root - INFO - waiting for ns_server @ 172.23.104.108:8091 2018-09-04 03:44:16,246 - root - INFO - ns_server @ 172.23.104.108:8091 is running 2018-09-04 03:44:16,255 - root - INFO - waiting for ns_server @ 172.23.104.109:8091 2018-09-04 03:44:16,263 - root - INFO - ns_server @ 172.23.104.109:8091 is running 2018-09-04 03:44:16,271 - root - INFO - waiting for ns_server @ 172.23.104.90:8091 2018-09-04 03:44:16,280 - root - INFO - ns_server @ 172.23.104.90:8091 is running 2018-09-04 03:44:16,281 - root - INFO - initializing cluster 2018-09-04 03:44:17,299 - root - INFO - server: ip:172.23.104.103 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'113', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4024, 'hostname': u'172.23.104.103:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-35007-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3494420480, 'memoryTotal': 4220149760, 'memoryQuota': 2145, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:44:17,300 - root - INFO - pools/default params : memoryQuota=2146 2018-09-04 03:44:17,307 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:44:17,317 - root - INFO - settings/web params on 172.23.104.103:8091:username=Administrator&password=password&port=8091 2018-09-04 03:44:17,370 - root - INFO - server: ip:172.23.104.104 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'123', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.104:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-35007-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3605692416, 'memoryTotal': 4220854272, 'memoryQuota': 2145, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:44:17,371 - root - INFO - pools/default params : memoryQuota=2146 2018-09-04 03:44:17,379 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:44:17,391 - root - INFO - settings/web params on 172.23.104.104:8091:username=Administrator&password=password&port=8091 2018-09-04 03:44:17,441 - root - INFO - server: ip:172.23.104.105 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'103', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.105:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-35007-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3546103808, 'memoryTotal': 4220854272, 'memoryQuota': 2145, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:44:17,442 - root - INFO - pools/default params : memoryQuota=2146 2018-09-04 03:44:17,449 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:44:17,461 - root - INFO - settings/web params on 172.23.104.105:8091:username=Administrator&password=password&port=8091 2018-09-04 03:44:17,519 - root - INFO - server: ip:172.23.104.106 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'92', 'mcdMemoryReserved': 2827, 'storageTotalRam': 3534, 'hostname': u'172.23.104.106:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-35007-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3044212736, 'memoryTotal': 3705864192, 'memoryQuota': 1883, 'mcdMemoryAllocated': 2827, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:44:17,520 - root - INFO - pools/default params : memoryQuota=1884 2018-09-04 03:44:17,528 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:44:17,546 - root - INFO - settings/web params on 172.23.104.106:8091:username=Administrator&password=password&port=8091 2018-09-04 03:44:17,637 - root - INFO - server: ip:172.23.104.107 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'59', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.107:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-35007-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3536740352, 'memoryTotal': 4220854272, 'memoryQuota': 2145, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:44:17,637 - root - INFO - pools/default params : memoryQuota=2146 2018-09-04 03:44:17,645 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:44:17,660 - root - INFO - settings/web params on 172.23.104.107:8091:username=Administrator&password=password&port=8091 2018-09-04 03:44:17,738 - root - INFO - server: ip:172.23.104.108 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'74', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.108:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-35007-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3540488192, 'memoryTotal': 4220854272, 'memoryQuota': 2145, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:44:17,738 - root - INFO - pools/default params : memoryQuota=2146 2018-09-04 03:44:17,746 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:44:17,759 - root - INFO - settings/web params on 172.23.104.108:8091:username=Administrator&password=password&port=8091 2018-09-04 03:44:17,843 - root - INFO - server: ip:172.23.104.109 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'63', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.109:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-35007-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3555192832, 'memoryTotal': 4220854272, 'memoryQuota': 2145, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:44:17,844 - root - INFO - pools/default params : memoryQuota=2146 2018-09-04 03:44:17,855 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:44:17,870 - root - INFO - settings/web params on 172.23.104.109:8091:username=Administrator&password=password&port=8091 2018-09-04 03:44:17,961 - root - INFO - server: ip:172.23.104.90 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'67', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.90:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-35007-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3609407488, 'memoryTotal': 4220846080, 'memoryQuota': 2145, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:44:17,961 - root - INFO - pools/default params : memoryQuota=2146 2018-09-04 03:44:17,971 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:44:17,987 - root - INFO - settings/web params on 172.23.104.90:8091:username=Administrator&password=password&port=8091 2018-09-04 03:44:18,081 - root - INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.103 **** 2018-09-04 03:44:18,106 - root - ERROR - DELETE http://172.23.104.103:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password 2018-09-04 03:44:18,113 - root - INFO - Exception while deleting user. Exception is -"User was not found." 2018-09-04 03:44:18,146 - root - INFO - sleep for 10 secs. ... 2018-09-04 03:44:28,157 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2018-09-04 03:44:28,185 - root - INFO - sleep for 10 secs. ... 2018-09-04 03:44:38,196 - root - INFO - done initializing cluster 2018-09-04 03:44:39,090 - root - INFO - adding node 172.23.104.104:8091 to cluster 2018-09-04 03:44:39,090 - root - INFO - adding remote node @172.23.104.104:8091 to this cluster @172.23.104.103:8091 2018-09-04 03:44:41,026 - root - INFO - adding node 172.23.104.105:8091 to cluster 2018-09-04 03:44:41,027 - root - INFO - adding remote node @172.23.104.105:8091 to this cluster @172.23.104.103:8091 2018-09-04 03:44:43,188 - root - INFO - adding node 172.23.104.106:8091 to cluster 2018-09-04 03:44:43,189 - root - INFO - adding remote node @172.23.104.106:8091 to this cluster @172.23.104.103:8091 2018-09-04 03:44:47,146 - root - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.104,ns_1@172.23.104.105,ns_1@172.23.104.103,ns_1@172.23.104.106'} 2018-09-04 03:44:47,155 - root - INFO - rebalance operation started 2018-09-04 03:44:47,165 - root - INFO - rebalance percentage : 0.00 % 2018-09-04 03:44:47,165 - root - INFO - Rebalance - status: running, progress: 0.00% 2018-09-04 03:44:57,195 - root - INFO - Rebalance - status: none, progress: 100.00% 2018-09-04 03:44:57,206 - root - INFO - rebalancing was completed with progress: 100% in 10.0502929688 sec 2018-09-04 03:44:58,247 - root - INFO - Node versions in cluster [u'6.0.0-35007-enterprise', u'6.0.0-35007-enterprise', u'6.0.0-35007-enterprise', u'6.0.0-35007-enterprise'] 2018-09-04 03:44:58,248 - root - INFO - http://172.23.104.103:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=2146&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 2018-09-04 03:44:58,256 - root - INFO - 0.01 seconds to create bucket default 2018-09-04 03:44:58,257 - root - INFO - waiting for memcached bucket : default in 172.23.104.103 to accept set ops 2018-09-04 03:44:59,955 - root - INFO - creating direct client 172.23.104.103:11210 default 2018-09-04 03:45:00,091 - root - INFO - creating direct client 172.23.104.104:11210 default 2018-09-04 03:45:00,210 - root - INFO - creating direct client 172.23.104.105:11210 default 2018-09-04 03:45:00,335 - root - INFO - creating direct client 172.23.104.106:11210 default 2018-09-04 03:45:00,533 - root - INFO - creating direct client 172.23.104.103:11210 default 2018-09-04 03:45:00,628 - root - INFO - creating direct client 172.23.104.104:11210 default 2018-09-04 03:45:00,728 - root - INFO - creating direct client 172.23.104.105:11210 default 2018-09-04 03:45:00,850 - root - INFO - creating direct client 172.23.104.106:11210 default 2018-09-04 03:45:01,914 - root - INFO - bucket 'default' was created with per node RAM quota: 2146 2018-09-04 03:45:01,930 - root - INFO - ============== basetestcase setup was finished for test #1 test_online_upgrade_with_failover_rebalance_with_eventing ============== 2018-09-04 03:45:01,943 - root - INFO - sleep for 10 secs. ... 2018-09-04 03:45:11,958 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:45:12,093 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:45:12,435 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:14,753 - root - INFO - cannot find service node n1ql in cluster 2018-09-04 03:45:14,759 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:45:14,929 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:45:15,253 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:15,883 - root - INFO - will install 5.5.0-2958 on ['172.23.104.103', '172.23.104.104', '172.23.104.105', '172.23.104.106'] 2018-09-04 03:45:15,897 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:45:15,922 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:45:15,923 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:45:15,925 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:45:16,101 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:45:16,110 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:45:16,128 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:45:16,131 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:45:16,525 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:16,549 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:16,569 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:16,593 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:16,934 - root - INFO - 172.23.104.103 *****In couchbase uninstall**** 2018-09-04 03:45:16,934 - root - INFO - CentOS 2018-09-04 03:45:16,948 - root - INFO - 172.23.104.104 *****In couchbase uninstall**** 2018-09-04 03:45:16,949 - root - INFO - CentOS 2018-09-04 03:45:17,040 - root - INFO - 172.23.104.105 *****In couchbase uninstall**** 2018-09-04 03:45:17,041 - root - INFO - CentOS 2018-09-04 03:45:17,069 - root - INFO - 172.23.104.106 *****In couchbase uninstall**** 2018-09-04 03:45:17,088 - root - INFO - CentOS 2018-09-04 03:45:17,100 - root - INFO - running command.raw on 172.23.104.103: killall -9 rpm 2018-09-04 03:45:17,119 - root - INFO - running command.raw on 172.23.104.104: killall -9 rpm 2018-09-04 03:45:17,151 - root - INFO - command executed successfully 2018-09-04 03:45:17,153 - root - ERROR - rpm: no process found 2018-09-04 03:45:17,154 - root - INFO - running command.raw on 172.23.104.103: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:45:17,158 - root - INFO - command executed successfully 2018-09-04 03:45:17,159 - root - ERROR - rpm: no process found 2018-09-04 03:45:17,160 - root - INFO - running command.raw on 172.23.104.104: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:45:17,213 - root - INFO - running command.raw on 172.23.104.105: killall -9 rpm 2018-09-04 03:45:17,237 - root - INFO - command executed successfully 2018-09-04 03:45:17,243 - root - INFO - command executed successfully 2018-09-04 03:45:17,243 - root - ERROR - rpm: no process found 2018-09-04 03:45:17,244 - root - INFO - running command.raw on 172.23.104.105: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:45:17,245 - root - INFO - command executed successfully 2018-09-04 03:45:17,315 - root - INFO - command executed successfully 2018-09-04 03:45:17,323 - root - INFO - running command.raw on 172.23.104.106: killall -9 rpm 2018-09-04 03:45:17,348 - root - INFO - command executed successfully 2018-09-04 03:45:17,349 - root - ERROR - bash: killall: command not found 2018-09-04 03:45:17,349 - root - INFO - running command.raw on 172.23.104.106: rm -f /var/lib/rpm/.rpm.lock 2018-09-04 03:45:17,397 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:45:17,398 - root - INFO - running command.raw on 172.23.104.103: rpm -e couchbase-server 2018-09-04 03:45:17,436 - root - INFO - command executed successfully 2018-09-04 03:45:17,491 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:45:17,491 - root - INFO - running command.raw on 172.23.104.105: rpm -e couchbase-server 2018-09-04 03:45:17,530 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:45:17,532 - root - INFO - running command.raw on 172.23.104.104: rpm -e couchbase-server 2018-09-04 03:45:17,655 - root - INFO - running rpm -e to remove couchbase-server 2018-09-04 03:45:17,655 - root - INFO - running command.raw on 172.23.104.106: rpm -e couchbase-server 2018-09-04 03:45:20,982 - root - INFO - command executed successfully 2018-09-04 03:45:20,983 - root - ERROR - **** Uninstalling on server: 172.23.104.105 **** 2018-09-04 03:45:20,983 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave 2018-09-04 03:45:20,984 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory 2018-09-04 03:45:20,984 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:45:20,984 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:45:21,037 - root - INFO - command executed successfully 2018-09-04 03:45:21,039 - root - ERROR - **** Uninstalling on server: 172.23.104.104 **** 2018-09-04 03:45:21,040 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave 2018-09-04 03:45:21,040 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory 2018-09-04 03:45:21,041 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:45:21,041 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:45:22,071 - root - INFO - command executed successfully 2018-09-04 03:45:22,072 - root - ERROR - **** Uninstalling on server: 172.23.104.106 **** 2018-09-04 03:45:22,104 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave 2018-09-04 03:45:22,105 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory 2018-09-04 03:45:22,105 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:45:22,105 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:45:22,108 - root - INFO - running command.raw on 172.23.104.105: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:45:22,160 - root - INFO - running command.raw on 172.23.104.104: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:45:22,196 - root - INFO - command executed successfully 2018-09-04 03:45:22,197 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:45:22,198 - root - INFO - running command.raw on 172.23.104.105: ipcrm 2018-09-04 03:45:22,247 - root - INFO - command executed successfully 2018-09-04 03:45:22,247 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:45:22,248 - root - INFO - running command.raw on 172.23.104.104: ipcrm 2018-09-04 03:45:22,274 - root - INFO - command executed successfully 2018-09-04 03:45:22,324 - root - INFO - command executed successfully 2018-09-04 03:45:23,495 - root - INFO - running command.raw on 172.23.104.106: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:45:23,572 - root - INFO - command executed successfully 2018-09-04 03:45:23,574 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:45:23,574 - root - INFO - running command.raw on 172.23.104.106: ipcrm 2018-09-04 03:45:23,660 - root - INFO - command executed successfully 2018-09-04 03:45:25,823 - root - INFO - command executed successfully 2018-09-04 03:45:25,824 - root - ERROR - **** Uninstalling on server: 172.23.104.103 **** 2018-09-04 03:45:25,825 - root - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory 2018-09-04 03:45:25,825 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2018-09-04 03:45:25,825 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2018-09-04 03:45:25,826 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2018-09-04 03:45:27,002 - root - INFO - running command.raw on 172.23.104.103: kill -9 $(ps aux | pgrep 'memcached') 2018-09-04 03:45:27,088 - root - INFO - command executed successfully 2018-09-04 03:45:27,090 - root - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] 2018-09-04 03:45:27,090 - root - INFO - running command.raw on 172.23.104.103: ipcrm 2018-09-04 03:45:27,166 - root - INFO - command executed successfully thread uninstaller-thread-172.23.104.103 finished thread uninstaller-thread-172.23.104.104 finished thread uninstaller-thread-172.23.104.105 finished thread uninstaller-thread-172.23.104.106 finished 2018-09-04 03:45:27,285 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:45:27,419 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:45:27,762 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:28,285 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:45:28,421 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:45:28,716 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:29,100 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:45:29,235 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:45:29,532 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:29,874 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:45:30,075 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:45:30,445 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:30,959 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:45:30,960 - root - INFO - inside try 2018-09-04 03:45:30,961 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:45:30,962 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:45:30,963 - root - INFO - inside try 2018-09-04 03:45:30,964 - root - INFO - inside try 2018-09-04 03:45:30,971 - root - INFO - ********CouchbaseServerInstaller:install 2018-09-04 03:45:30,973 - root - INFO - inside try 2018-09-04 03:45:30,973 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:45:30,987 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:45:31,007 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:45:31,009 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:45:31,145 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:45:31,160 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:45:31,164 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:45:31,211 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:45:31,456 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:31,503 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:31,514 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:31,590 - root - INFO - It could be other operating system. Go to check at other location *** OS version of this server 172.23.104.104 is CentOS 7 *** 2018-09-04 03:45:31,918 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: *** OS version of this server 172.23.104.105 is CentOS 7 *** 2018-09-04 03:45:31,947 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: *** OS version of this server 172.23.104.103 is CentOS 7 *** 2018-09-04 03:45:31,973 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: *** OS version of this server 172.23.104.106 is CentOS 7 *** 2018-09-04 03:45:32,030 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:45:32,085 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:45:32,119 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:45:32,142 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:45:32,198 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:45:32,461 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:32,479 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:32,556 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:32,620 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:32,868 - root - INFO - Check if url http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm is ok 2018-09-04 03:45:32,901 - root - INFO - This url http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm is live 2018-09-04 03:45:32,911 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:45:32,976 - root - INFO - Check if url http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm is ok 2018-09-04 03:45:32,981 - root - INFO - Check if url http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm is ok 2018-09-04 03:45:32,994 - root - INFO - This url http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm is live 2018-09-04 03:45:33,003 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:45:33,006 - root - INFO - This url http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm is live 2018-09-04 03:45:33,015 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:45:33,041 - root - INFO - Check if url http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm is ok 2018-09-04 03:45:33,051 - root - INFO - This url http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm is live 2018-09-04 03:45:33,055 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:45:33,055 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:45:33,159 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:45:33,177 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:45:33,260 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:45:33,337 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:33,468 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:33,528 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:33,624 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:45:33,707 - root - INFO - inside else 2018-09-04 03:45:33,708 - root - INFO - running command.raw on 172.23.104.105: /sbin/iptables -F 2018-09-04 03:45:33,790 - root - INFO - command executed successfully 2018-09-04 03:45:33,791 - root - INFO - running command.raw on 172.23.104.105: /sbin/iptables -t nat -F 2018-09-04 03:45:33,869 - root - INFO - inside else 2018-09-04 03:45:33,870 - root - INFO - running command.raw on 172.23.104.103: /sbin/iptables -F 2018-09-04 03:45:33,873 - root - INFO - command executed successfully 2018-09-04 03:45:33,875 - root - INFO - Connect to node: 172.23.104.105 as user: root 2018-09-04 03:45:33,906 - root - INFO - inside else 2018-09-04 03:45:33,907 - root - INFO - running command.raw on 172.23.104.104: /sbin/iptables -F 2018-09-04 03:45:33,954 - root - INFO - command executed successfully 2018-09-04 03:45:33,954 - root - INFO - running command.raw on 172.23.104.103: /sbin/iptables -t nat -F 2018-09-04 03:45:33,988 - root - INFO - command executed successfully 2018-09-04 03:45:33,989 - root - INFO - running command.raw on 172.23.104.104: /sbin/iptables -t nat -F 2018-09-04 03:45:34,013 - root - INFO - Connected to 172.23.104.105 as root 2018-09-04 03:45:34,013 - root - INFO - running command.raw on 172.23.104.105: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_5.5.0-2958.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:45:34,035 - root - INFO - command executed successfully 2018-09-04 03:45:34,036 - root - INFO - Connect to node: 172.23.104.103 as user: root 2018-09-04 03:45:34,047 - root - INFO - command executed successfully 2018-09-04 03:45:34,048 - root - INFO - Connect to node: 172.23.104.104 as user: root 2018-09-04 03:45:34,074 - root - INFO - inside else 2018-09-04 03:45:34,076 - root - INFO - running command.raw on 172.23.104.106: /sbin/iptables -F 2018-09-04 03:45:34,166 - root - INFO - command executed successfully 2018-09-04 03:45:34,168 - root - INFO - running command.raw on 172.23.104.106: /sbin/iptables -t nat -F 2018-09-04 03:45:34,206 - root - INFO - Connected to 172.23.104.103 as root 2018-09-04 03:45:34,206 - root - INFO - running command.raw on 172.23.104.103: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_5.5.0-2958.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:45:34,214 - root - INFO - Connected to 172.23.104.104 as root 2018-09-04 03:45:34,215 - root - INFO - running command.raw on 172.23.104.104: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_5.5.0-2958.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:45:34,251 - root - INFO - command executed successfully 2018-09-04 03:45:34,253 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise_x86_64_5.5.0-2958.rpm’: No such file or directory 2018-09-04 03:45:34,254 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:45:34,255 - root - ERROR - rm: cannot remove ‘cb_yIYW’: Is a directory 2018-09-04 03:45:34,256 - root - ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory 2018-09-04 03:45:34,256 - root - INFO - command executed successfully 2018-09-04 03:45:34,257 - root - ERROR - rm: cannot remove ‘systemd-private-4ac815aee3424d938d2263668d460ba5-ntpd.service-U2ye4B’: Is a directory 2018-09-04 03:45:34,258 - root - INFO - Connect to node: 172.23.104.106 as user: root 2018-09-04 03:45:34,259 - root - ERROR - mv: cannot stat ‘cb_yIYW/*’: No such file or directory 2018-09-04 03:45:34,260 - root - INFO - running command.raw on 172.23.104.105: cd /tmp;wget -q -O couchbase-server-enterprise_x86_64_5.5.0-2958.rpm http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:45:34,429 - root - INFO - Connected to 172.23.104.106 as root 2018-09-04 03:45:34,429 - root - INFO - running command.raw on 172.23.104.106: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_5.5.0-2958.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2018-09-04 03:45:34,492 - root - INFO - command executed successfully 2018-09-04 03:45:34,494 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise_x86_64_5.5.0-2958.rpm’: No such file or directory 2018-09-04 03:45:34,495 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:45:34,495 - root - ERROR - rm: cannot remove ‘cb_yHhm’: Is a directory 2018-09-04 03:45:34,495 - root - ERROR - rm: cannot remove ‘systemd-private-dbeed978d97747f6860a8195194b6ada-ntpd.service-A2MgZO’: Is a directory 2018-09-04 03:45:34,495 - root - ERROR - mv: cannot stat ‘cb_yHhm/*’: No such file or directory 2018-09-04 03:45:34,495 - root - INFO - running command.raw on 172.23.104.104: cd /tmp;wget -q -O couchbase-server-enterprise_x86_64_5.5.0-2958.rpm http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:45:34,498 - root - INFO - command executed successfully 2018-09-04 03:45:34,498 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise_x86_64_5.5.0-2958.rpm’: No such file or directory 2018-09-04 03:45:34,499 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:45:34,499 - root - ERROR - rm: cannot remove ‘backups’: Is a directory 2018-09-04 03:45:34,500 - root - ERROR - rm: cannot remove ‘cb_llMX’: Is a directory 2018-09-04 03:45:34,500 - root - ERROR - rm: cannot remove ‘systemd-private-83a29c879dd548ddb57f9a0683aa9254-ntpd.service-G5roHa’: Is a directory 2018-09-04 03:45:34,500 - root - ERROR - rm: cannot remove ‘tmp.3wHdyvLuME’: Is a directory 2018-09-04 03:45:34,501 - root - ERROR - mv: cannot stat ‘cb_llMX/*’: No such file or directory 2018-09-04 03:45:34,502 - root - INFO - running command.raw on 172.23.104.103: cd /tmp;wget -q -O couchbase-server-enterprise_x86_64_5.5.0-2958.rpm http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:45:34,780 - root - INFO - command executed successfully 2018-09-04 03:45:34,781 - root - ERROR - mv: cannot stat ‘couchbase-server-enterprise_x86_64_5.5.0-2958.rpm’: No such file or directory 2018-09-04 03:45:34,782 - root - ERROR - mv: cannot stat ‘core.*’: No such file or directory 2018-09-04 03:45:34,782 - root - ERROR - rm: cannot remove ‘cb_xYS5’: Is a directory 2018-09-04 03:45:34,782 - root - ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory 2018-09-04 03:45:34,782 - root - ERROR - rm: cannot remove ‘systemd-private-03a85ff6ef3c4dd3b00e67c9fb47abbf-ntpd.service-JQIjQj’: Is a directory 2018-09-04 03:45:34,783 - root - ERROR - rm: cannot remove ‘tmpOWeU1l’: Is a directory 2018-09-04 03:45:34,783 - root - ERROR - mv: cannot stat ‘cb_xYS5/*’: No such file or directory 2018-09-04 03:45:34,783 - root - INFO - running command.raw on 172.23.104.106: cd /tmp;wget -q -O couchbase-server-enterprise_x86_64_5.5.0-2958.rpm http://172.23.120.24/builds/releases/5.5.0/couchbase-server-enterprise-5.5.0-centos7.x86_64.rpm;cd /tmp;ls -lh 2018-09-04 03:45:44,304 - root - INFO - command executed successfully 2018-09-04 03:45:44,305 - root - INFO - total 222M 2018-09-04 03:45:44,306 - root - INFO - -rw-r--r--. 1 root root 222M Jul 9 09:57 couchbase-server-enterprise_x86_64_5.5.0-2958.rpm 2018-09-04 03:45:44,307 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Sep 3 05:51 hsperfdata_couchbase 2018-09-04 03:45:44,308 - root - INFO - drwx------. 3 root root 16 Jul 2 13:28 systemd-private-03a85ff6ef3c4dd3b00e67c9fb47abbf-ntpd.service-JQIjQj 2018-09-04 03:45:44,308 - root - INFO - drwx------. 2 root root 4.0K Aug 27 09:40 tmpOWeU1l 2018-09-04 03:45:44,408 - root - INFO - *****install server *** 2018-09-04 03:45:44,408 - root - INFO - deliverable_type : rpm 2018-09-04 03:45:44,408 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_5.5.0-2958.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:45:44,409 - root - INFO - running command.raw on 172.23.104.106: /sbin/sysctl vm.swappiness=0 2018-09-04 03:45:44,497 - root - INFO - command executed successfully 2018-09-04 03:45:44,499 - root - INFO - running command.raw on 172.23.104.106: cat /etc/redhat-release 2018-09-04 03:45:44,586 - root - INFO - command executed successfully 2018-09-04 03:45:44,587 - root - INFO - CentOS Linux release 7.2.1511 (Core) 2018-09-04 03:45:44,588 - root - INFO - no need to install pkgconfig 2018-09-04 03:45:44,823 - root - INFO - command executed successfully 2018-09-04 03:45:44,824 - root - INFO - total 222M 2018-09-04 03:45:44,824 - root - INFO - -rw-r--r--. 1 root root 222M Jul 9 09:57 couchbase-server-enterprise_x86_64_5.5.0-2958.rpm 2018-09-04 03:45:44,825 - root - INFO - drwx------. 3 root root 16 Aug 26 04:30 systemd-private-dbeed978d97747f6860a8195194b6ada-ntpd.service-A2MgZO 2018-09-04 03:45:44,882 - root - INFO - command executed successfully 2018-09-04 03:45:44,883 - root - INFO - total 222M 2018-09-04 03:45:44,884 - root - INFO - -rw-r--r--. 1 root root 222M Jul 9 09:57 couchbase-server-enterprise_x86_64_5.5.0-2958.rpm 2018-09-04 03:45:44,884 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Aug 26 19:33 hsperfdata_couchbase 2018-09-04 03:45:44,885 - root - INFO - drwx------. 3 root root 16 Aug 31 22:36 systemd-private-4ac815aee3424d938d2263668d460ba5-ntpd.service-U2ye4B 2018-09-04 03:45:44,910 - root - INFO - *****install server *** 2018-09-04 03:45:44,910 - root - INFO - deliverable_type : rpm 2018-09-04 03:45:44,911 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_5.5.0-2958.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:45:44,911 - root - INFO - running command.raw on 172.23.104.104: /sbin/sysctl vm.swappiness=0 2018-09-04 03:45:44,971 - root - INFO - *****install server *** 2018-09-04 03:45:44,972 - root - INFO - deliverable_type : rpm 2018-09-04 03:45:44,973 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_5.5.0-2958.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:45:44,973 - root - INFO - running command.raw on 172.23.104.105: /sbin/sysctl vm.swappiness=0 2018-09-04 03:45:44,991 - root - INFO - command executed successfully 2018-09-04 03:45:44,993 - root - INFO - running command.raw on 172.23.104.104: cat /etc/redhat-release 2018-09-04 03:45:45,052 - root - INFO - command executed successfully 2018-09-04 03:45:45,054 - root - INFO - running command.raw on 172.23.104.105: cat /etc/redhat-release 2018-09-04 03:45:45,072 - root - INFO - command executed successfully 2018-09-04 03:45:45,073 - root - INFO - CentOS Linux release 7.4.1708 (Core) 2018-09-04 03:45:45,073 - root - INFO - no need to install pkgconfig 2018-09-04 03:45:45,132 - root - INFO - command executed successfully 2018-09-04 03:45:45,133 - root - INFO - CentOS Linux release 7.4.1708 (Core) 2018-09-04 03:45:45,134 - root - INFO - no need to install pkgconfig 2018-09-04 03:45:46,444 - root - INFO - command executed successfully 2018-09-04 03:45:46,445 - root - INFO - total 222M 2018-09-04 03:45:46,446 - root - INFO - drwxrwx---. 4 root root 45 Sep 3 04:29 backups 2018-09-04 03:45:46,446 - root - INFO - -rw-r--r--. 1 root root 222M Jul 9 09:57 couchbase-server-enterprise_x86_64_5.5.0-2958.rpm 2018-09-04 03:45:46,446 - root - INFO - drwx------. 3 root root 16 Aug 28 12:33 systemd-private-83a29c879dd548ddb57f9a0683aa9254-ntpd.service-G5roHa 2018-09-04 03:45:46,447 - root - INFO - drwx------. 2 root root 6 Aug 24 18:03 tmp.3wHdyvLuME 2018-09-04 03:45:46,539 - root - INFO - *****install server *** 2018-09-04 03:45:46,539 - root - INFO - deliverable_type : rpm 2018-09-04 03:45:46,540 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_5.5.0-2958.rpm or /tmp/couchbase-server-enterprise 2018-09-04 03:45:46,541 - root - INFO - running command.raw on 172.23.104.103: /sbin/sysctl vm.swappiness=0 2018-09-04 03:45:46,622 - root - INFO - command executed successfully 2018-09-04 03:45:46,623 - root - INFO - running command.raw on 172.23.104.103: cat /etc/redhat-release 2018-09-04 03:45:46,702 - root - INFO - command executed successfully 2018-09-04 03:45:46,703 - root - INFO - CentOS Linux release 7.5.1804 (Core) 2018-09-04 03:45:46,704 - root - INFO - no need to install pkgconfig 2018-09-04 03:46:09,251 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:46:09,442 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:46:09,519 - root - INFO - Running systemd command on this server 2018-09-04 03:46:09,520 - root - INFO - running command.raw on 172.23.104.104: systemctl start couchbase-server.service 2018-09-04 03:46:09,572 - root - INFO - command executed successfully 2018-09-04 03:46:09,620 - root - INFO - Running systemd command on this server 2018-09-04 03:46:09,621 - root - INFO - running command.raw on 172.23.104.105: systemctl start couchbase-server.service 2018-09-04 03:46:09,628 - root - INFO - wait 5 seconds for Couchbase server to start 2018-09-04 03:46:09,669 - root - INFO - command executed successfully 2018-09-04 03:46:09,686 - root - INFO - wait 5 seconds for Couchbase server to start 2018-09-04 03:46:11,141 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:46:11,339 - root - INFO - Running systemd command on this server 2018-09-04 03:46:11,340 - root - INFO - running command.raw on 172.23.104.103: systemctl start couchbase-server.service 2018-09-04 03:46:11,366 - root - INFO - command executed successfully 2018-09-04 03:46:11,445 - root - INFO - wait 5 seconds for Couchbase server to start thread installer-thread-172.23.104.103 finished thread installer-thread-172.23.104.104 finished thread installer-thread-172.23.104.105 finished 2018-09-04 03:46:39,246 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2018-09-04 03:46:39,538 - root - INFO - Running systemd command on this server 2018-09-04 03:46:39,538 - root - INFO - running command.raw on 172.23.104.106: systemctl start couchbase-server.service 2018-09-04 03:46:39,626 - root - INFO - command executed successfully 2018-09-04 03:46:39,655 - root - INFO - wait 5 seconds for Couchbase server to start thread installer-thread-172.23.104.106 finished 2018-09-04 03:46:44,693 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:46:44,694 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:46:44,704 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:46:44,707 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:46:44,835 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:46:44,850 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:46:44,851 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:46:44,883 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:46:45,298 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:46:45,308 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:46:45,325 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:46:45,372 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:46:45,806 - root - INFO - 172.23.104.104 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:46:45,835 - root - INFO - 172.23.104.103 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:46:45,843 - root - INFO - 172.23.104.105 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:46:45,904 - root - INFO - 172.23.104.106 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:46:45,907 - root - ERROR - socket error while connecting to http://172.23.104.106:8091/nodes/self error [Errno 111] Connection refused 2018-09-04 03:46:48,912 - root - ERROR - socket error while connecting to http://172.23.104.106:8091/nodes/self error [Errno 111] Connection refused 2018-09-04 03:46:49,846 - root - INFO - quota for kv: 2145 MB 2018-09-04 03:46:49,846 - root - INFO - pools/default params : memoryQuota=2145 2018-09-04 03:46:49,856 - root - INFO - /node/controller/setupServices params on 172.23.104.104: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:46:49,870 - root - INFO - settings/web params on 172.23.104.104:8091:username=Administrator&password=password&port=8091 2018-09-04 03:46:49,882 - root - INFO - quota for kv: 2145 MB 2018-09-04 03:46:49,883 - root - INFO - pools/default params : memoryQuota=2145 2018-09-04 03:46:49,891 - root - INFO - quota for kv: 2145 MB 2018-09-04 03:46:49,892 - root - INFO - pools/default params : memoryQuota=2145 2018-09-04 03:46:49,896 - root - INFO - /node/controller/setupServices params on 172.23.104.103: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:46:49,913 - root - INFO - settings/web params on 172.23.104.103:8091:username=Administrator&password=password&port=8091 2018-09-04 03:46:49,916 - root - INFO - /node/controller/setupServices params on 172.23.104.105: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:46:49,930 - root - INFO - settings/web params on 172.23.104.105:8091:username=Administrator&password=password&port=8091 2018-09-04 03:46:49,949 - root - INFO - Node version in cluster 5.5.0-2958-enterprise 2018-09-04 03:46:49,950 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:46:50,004 - root - INFO - Node version in cluster 5.5.0-2958-enterprise 2018-09-04 03:46:50,004 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:46:50,017 - root - INFO - Node version in cluster 5.5.0-2958-enterprise 2018-09-04 03:46:50,018 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:46:50,216 - root - INFO - check if man installed on vm? 2018-09-04 03:46:50,217 - root - INFO - running command.raw on 172.23.104.104: rpm -qa | grep 'man-db' 2018-09-04 03:46:50,276 - root - INFO - check if man installed on vm? 2018-09-04 03:46:50,276 - root - INFO - running command.raw on 172.23.104.105: rpm -qa | grep 'man-db' 2018-09-04 03:46:50,291 - root - INFO - check if man installed on vm? 2018-09-04 03:46:50,293 - root - INFO - running command.raw on 172.23.104.103: rpm -qa | grep 'man-db' 2018-09-04 03:46:50,926 - root - INFO - command executed successfully 2018-09-04 03:46:50,927 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.104 2018-09-04 03:46:50,928 - root - INFO - running command.raw on 172.23.104.104: unzip 2018-09-04 03:46:50,939 - root - INFO - command executed successfully 2018-09-04 03:46:50,941 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.105 2018-09-04 03:46:50,942 - root - INFO - running command.raw on 172.23.104.105: unzip 2018-09-04 03:46:51,010 - root - INFO - command executed successfully 2018-09-04 03:46:51,012 - root - INFO - unzip command is ready 2018-09-04 03:46:51,013 - root - INFO - Check if ntp is installed 2018-09-04 03:46:51,013 - root - INFO - This OS version centos 7 2018-09-04 03:46:51,014 - root - INFO - running command.raw on 172.23.104.104: systemctl status ntpd 2018-09-04 03:46:51,023 - root - INFO - command executed successfully 2018-09-04 03:46:51,024 - root - INFO - unzip command is ready 2018-09-04 03:46:51,024 - root - INFO - Check if ntp is installed 2018-09-04 03:46:51,025 - root - INFO - This OS version centos 7 2018-09-04 03:46:51,025 - root - INFO - running command.raw on 172.23.104.105: systemctl status ntpd 2018-09-04 03:46:51,105 - root - INFO - command executed successfully 2018-09-04 03:46:51,106 - root - INFO - running command.raw on 172.23.104.104: date 2018-09-04 03:46:51,115 - root - INFO - command executed successfully 2018-09-04 03:46:51,116 - root - INFO - running command.raw on 172.23.104.105: date 2018-09-04 03:46:51,172 - root - INFO - command executed successfully 2018-09-04 03:46:51,173 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.103 2018-09-04 03:46:51,174 - root - INFO - running command.raw on 172.23.104.103: unzip 2018-09-04 03:46:51,187 - root - INFO - command executed successfully 2018-09-04 03:46:51,194 - root - INFO - command executed successfully 2018-09-04 03:46:51,255 - root - INFO - command executed successfully 2018-09-04 03:46:51,256 - root - INFO - unzip command is ready 2018-09-04 03:46:51,257 - root - INFO - Check if ntp is installed 2018-09-04 03:46:51,257 - root - INFO - This OS version centos 7 2018-09-04 03:46:51,257 - root - INFO - running command.raw on 172.23.104.103: systemctl status ntpd 2018-09-04 03:46:51,346 - root - INFO - command executed successfully 2018-09-04 03:46:51,347 - root - INFO - running command.raw on 172.23.104.103: date 2018-09-04 03:46:51,353 - root - INFO - ['Tue Sep 4 03:46:51 PDT 2018'] IP: 172.23.104.105 2018-09-04 03:46:51,368 - root - INFO - ['Tue Sep 4 03:46:51 PDT 2018'] IP: 172.23.104.104 2018-09-04 03:46:51,424 - root - INFO - command executed successfully 2018-09-04 03:46:51,609 - root - INFO - ['Tue Sep 4 03:46:51 PDT 2018'] IP: 172.23.104.103 thread initializer-thread-172.23.104.103 finished thread initializer-thread-172.23.104.104 finished thread initializer-thread-172.23.104.105 finished 2018-09-04 03:46:55,960 - root - INFO - quota for kv: 1883 MB 2018-09-04 03:46:55,960 - root - INFO - pools/default params : memoryQuota=1883 2018-09-04 03:46:55,975 - root - INFO - /node/controller/setupServices params on 172.23.104.106: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator 2018-09-04 03:46:55,989 - root - INFO - settings/web params on 172.23.104.106:8091:username=Administrator&password=password&port=8091 2018-09-04 03:46:56,099 - root - INFO - Node version in cluster 5.5.0-2958-enterprise 2018-09-04 03:46:56,099 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2018-09-04 03:46:56,474 - root - INFO - check if man installed on vm? 2018-09-04 03:46:56,475 - root - INFO - running command.raw on 172.23.104.106: rpm -qa | grep 'man-db' 2018-09-04 03:46:57,622 - root - INFO - command executed successfully 2018-09-04 03:46:57,623 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.106 2018-09-04 03:46:57,623 - root - INFO - running command.raw on 172.23.104.106: unzip 2018-09-04 03:46:57,711 - root - INFO - command executed successfully 2018-09-04 03:46:57,712 - root - INFO - unzip command is ready 2018-09-04 03:46:57,713 - root - INFO - Check if ntp is installed 2018-09-04 03:46:57,713 - root - INFO - This OS version centos 7 2018-09-04 03:46:57,713 - root - INFO - running command.raw on 172.23.104.106: systemctl status ntpd 2018-09-04 03:46:57,835 - root - INFO - command executed successfully 2018-09-04 03:46:57,836 - root - INFO - running command.raw on 172.23.104.106: date 2018-09-04 03:46:57,922 - root - INFO - command executed successfully 2018-09-04 03:46:58,142 - root - INFO - ['Tue Sep 4 03:46:58 PDT 2018'] IP: 172.23.104.106 thread initializer-thread-172.23.104.106 finished 2018-09-04 03:46:58,150 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:46:58,285 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:46:58,605 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:46:59,010 - root - INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.103 **** 2018-09-04 03:46:59,038 - root - ERROR - DELETE http://172.23.104.103:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password 2018-09-04 03:46:59,039 - root - INFO - Exception while deleting user. Exception is -"User was not found." 2018-09-04 03:46:59,075 - root - INFO - sleep for 10 secs. ... 2018-09-04 03:47:09,085 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2018-09-04 03:47:09,110 - root - INFO - sleep for 10 secs. ... 2018-09-04 03:47:19,127 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 03:47:19,262 - root - INFO - Connected to 172.23.104.103 2018-09-04 03:47:19,661 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:47:20,082 - root - INFO - 172.23.104.103 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:47:20,082 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2018-09-04 03:47:25,224 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 03:47:25,358 - root - INFO - Connected to 172.23.104.104 2018-09-04 03:47:25,694 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:47:26,099 - root - INFO - 172.23.104.104 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:47:26,100 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2018-09-04 03:47:31,238 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 03:47:31,373 - root - INFO - Connected to 172.23.104.105 2018-09-04 03:47:31,678 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:47:32,072 - root - INFO - 172.23.104.105 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:47:32,073 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2018-09-04 03:47:37,213 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 03:47:37,412 - root - INFO - Connected to 172.23.104.106 2018-09-04 03:47:37,757 - root - INFO - It could be other operating system. Go to check at other location 2018-09-04 03:47:38,335 - root - INFO - 172.23.104.106 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-04 03:47:38,335 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2018-09-04 03:47:44,128 - root - INFO - server: ip:172.23.104.103 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'89', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4024, 'hostname': u'172.23.104.103:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-2958-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3491164160, 'memoryTotal': 4220149760, 'memoryQuota': 2145, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:47:44,129 - root - INFO - pools/default params : memoryQuota=2146 2018-09-04 03:47:44,134 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:47:44,143 - root - INFO - settings/web params on 172.23.104.103:8091:username=Administrator&password=password&port=8091 2018-09-04 03:47:44,194 - root - INFO - server: ip:172.23.104.104 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'88', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.104:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-2958-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3603247104, 'memoryTotal': 4220854272, 'memoryQuota': 2145, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:47:44,195 - root - INFO - pools/default params : memoryQuota=2146 2018-09-04 03:47:44,201 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:47:44,212 - root - INFO - settings/web params on 172.23.104.104:8091:username=Administrator&password=password&port=8091 2018-09-04 03:47:44,266 - root - INFO - server: ip:172.23.104.105 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'89', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.105:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-2958-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3539140608, 'memoryTotal': 4220854272, 'memoryQuota': 2145, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:47:44,266 - root - INFO - pools/default params : memoryQuota=2146 2018-09-04 03:47:44,275 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:47:44,283 - root - INFO - settings/web params on 172.23.104.105:8091:username=Administrator&password=password&port=8091 2018-09-04 03:47:44,334 - root - INFO - server: ip:172.23.104.106 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'57', 'mcdMemoryReserved': 2827, 'storageTotalRam': 3534, 'hostname': u'172.23.104.106:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-2958-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3024216064, 'memoryTotal': 3705864192, 'memoryQuota': 1883, 'mcdMemoryAllocated': 2827, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-04 03:47:44,334 - root - INFO - pools/default params : memoryQuota=1884 2018-09-04 03:47:44,343 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-04 03:47:44,363 - root - INFO - settings/web params on 172.23.104.106:8091:username=Administrator&password=password&port=8091 2018-09-04 03:47:44,445 - root - INFO - sleep for 20 secs. wait to make sure node is ready ... 2018-09-04 03:48:05,470 - root - INFO - adding node 172.23.104.104:8091 to cluster 2018-09-04 03:48:05,470 - root - INFO - adding remote node @172.23.104.104:8091 to this cluster @172.23.104.103:8091 2018-09-04 03:48:07,694 - root - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.104,ns_1@172.23.104.103'} 2018-09-04 03:48:07,705 - root - INFO - rebalance operation started 2018-09-04 03:48:07,714 - root - INFO - rebalance percentage : 0.00 % 2018-09-04 03:48:07,714 - root - INFO - Rebalance - status: running, progress: 0.00% 2018-09-04 03:48:17,741 - root - INFO - rebalance percentage : 50.00 % 2018-09-04 03:48:17,742 - root - INFO - Rebalance - status: running, progress: 50.00% 2018-09-04 03:48:27,771 - root - INFO - Rebalance - status: none, progress: 100.00% 2018-09-04 03:48:27,778 - root - INFO - rebalancing was completed with progress: 100% in 20.0724279881 sec 2018-09-04 03:48:27,779 - root - INFO - sleep for 10 secs. ... 2018-09-04 03:48:38,792 - root - INFO - adding node 172.23.104.105:8091 to cluster 2018-09-04 03:48:38,792 - root - INFO - adding remote node @172.23.104.105:8091 to this cluster @172.23.104.103:8091 2018-09-04 03:48:41,097 - root - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.104,ns_1@172.23.104.105,ns_1@172.23.104.103'} 2018-09-04 03:48:41,104 - root - INFO - rebalance operation started 2018-09-04 03:48:41,186 - root - INFO - rebalance percentage : 33.00 % 2018-09-04 03:48:41,187 - root - INFO - Rebalance - status: running, progress: 33.00% 2018-09-04 03:48:51,222 - root - INFO - rebalance percentage : 66.00 % 2018-09-04 03:48:51,222 - root - INFO - Rebalance - status: running, progress: 66.00% 2018-09-04 03:49:01,250 - root - INFO - Rebalance - status: none, progress: 100.00% 2018-09-04 03:49:01,261 - root - INFO - rebalancing was completed with progress: 100% in 20.0811851025 sec 2018-09-04 03:49:01,262 - root - INFO - sleep for 10 secs. ... 2018-09-04 03:49:12,276 - root - INFO - adding node 172.23.104.106:8091 to cluster 2018-09-04 03:49:12,276 - root - INFO - adding remote node @172.23.104.106:8091 to this cluster @172.23.104.103:8091 2018-09-04 03:49:16,475 - root - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.104,ns_1@172.23.104.105,ns_1@172.23.104.103,ns_1@172.23.104.106'} 2018-09-04 03:49:16,484 - root - INFO - rebalance operation started 2018-09-04 03:49:16,491 - root - INFO - rebalance percentage : 0.00 % 2018-09-04 03:49:16,491 - root - INFO - Rebalance - status: running, progress: 0.00% 2018-09-04 03:49:26,521 - root - INFO - rebalance percentage : 66.00 % 2018-09-04 03:49:26,522 - root - INFO - Rebalance - status: running, progress: 66.00% 2018-09-04 03:49:36,551 - root - INFO - Rebalance - status: none, progress: 100.00% 2018-09-04 03:49:36,558 - root - INFO - rebalancing was completed with progress: 100% in 20.0743010044 sec 2018-09-04 03:49:36,559 - root - INFO - sleep for 10 secs. ... 2018-09-04 03:49:47,664 - root - INFO - Node versions in cluster [u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise'] 2018-09-04 03:49:47,664 - root - INFO - http://172.23.104.103:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1884&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 2018-09-04 03:49:47,673 - root - INFO - 0.01 seconds to create bucket default 2018-09-04 03:49:47,674 - root - INFO - waiting for memcached bucket : default in 172.23.104.103 to accept set ops 2018-09-04 03:49:48,843 - root - INFO - creating direct client 172.23.104.103:11210 default 2018-09-04 03:49:48,965 - root - INFO - creating direct client 172.23.104.103:11210 default 2018-09-04 03:49:50,726 - root - INFO - bucket 'default' was created with per node RAM quota: 1884 2018-09-04 03:49:50,808 - root - INFO - creating direct client 172.23.104.103:11210 default 2018-09-04 03:49:51,823 - root - INFO - creating direct client 172.23.104.103:11210 default 2018-09-04 03:49:51,835 - root - INFO - creating direct client 172.23.104.103:11210 default 2018-09-04 03:49:51,845 - root - INFO - creating direct client 172.23.104.103:11210 default 2018-09-04 03:49:51,852 - root - INFO - creating direct client 172.23.104.103:11210 default 2018-09-04 03:49:52,088 - root - INFO - Node versions in cluster [u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise'] 2018-09-04 03:49:52,102 - root - INFO - list of kv nodes in cluster: [ip:172.23.104.103 port:8091 ssh_username:root] 2018-09-04 03:49:52,748 - root - INFO - creating direct client 172.23.104.103:11210 default 2018-09-04 03:49:52,837 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.103:8091',default bucket 2018-09-04 03:49:52,871 - root - INFO - pools/default params : memoryQuota=700 2018-09-04 03:49:52,879 - root - ERROR - POST http://172.23.104.103:8091/pools/default body: memoryQuota=700 headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"memoryQuota":"The data service quota (700MB) cannot be less than 1884MB (current total buckets quota, or at least 256MB)."}} auth: Administrator:password 2018-09-04 03:49:53,316 - root - INFO - Create the required buckets in the initial version 2018-09-04 03:49:53,877 - root - INFO - Node versions in cluster [u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise'] 2018-09-04 03:49:53,878 - root - INFO - http://172.23.104.103:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=src_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 2018-09-04 03:49:53,886 - root - INFO - 0.01 seconds to create bucket src_bucket 2018-09-04 03:49:53,887 - root - INFO - waiting for memcached bucket : src_bucket in 172.23.104.103 to accept set ops 2018-09-04 03:49:55,119 - root - INFO - creating direct client 172.23.104.103:11210 src_bucket 2018-09-04 03:49:55,244 - root - INFO - creating direct client 172.23.104.103:11210 src_bucket 2018-09-04 03:49:56,536 - root - INFO - bucket 'src_bucket' was created with per node RAM quota: 100 2018-09-04 03:49:56,563 - root - INFO - sleep for 60 secs. ... 2018-09-04 03:50:57,647 - root - INFO - Node versions in cluster [u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise'] 2018-09-04 03:50:57,647 - root - INFO - http://172.23.104.103:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=dst_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 2018-09-04 03:50:57,663 - root - INFO - 0.02 seconds to create bucket dst_bucket 2018-09-04 03:50:57,664 - root - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.103 to accept set ops 2018-09-04 03:50:58,828 - root - INFO - creating direct client 172.23.104.103:11210 dst_bucket 2018-09-04 03:50:58,958 - root - INFO - creating direct client 172.23.104.103:11210 dst_bucket 2018-09-04 03:51:00,806 - root - INFO - bucket 'dst_bucket' was created with per node RAM quota: 100 2018-09-04 03:51:00,807 - root - INFO - sleep for 60 secs. ... 2018-09-04 03:52:00,921 - root - INFO - Node versions in cluster [u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise'] 2018-09-04 03:52:00,922 - root - INFO - http://172.23.104.103:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=metadata&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 2018-09-04 03:52:00,932 - root - INFO - 0.01 seconds to create bucket metadata 2018-09-04 03:52:00,932 - root - INFO - waiting for memcached bucket : metadata in 172.23.104.103 to accept set ops 2018-09-04 03:52:02,129 - root - INFO - creating direct client 172.23.104.103:11210 metadata 2018-09-04 03:52:02,259 - root - INFO - creating direct client 172.23.104.103:11210 metadata 2018-09-04 03:52:04,086 - root - INFO - bucket 'metadata' was created with per node RAM quota: 100 2018-09-04 03:52:04,087 - root - INFO - sleep for 60 secs. ... 2018-09-04 03:53:04,199 - root - INFO - Node versions in cluster [u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise', u'5.5.0-2958-enterprise'] 2018-09-04 03:53:04,200 - root - INFO - http://172.23.104.103:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=dst_bucket1&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 2018-09-04 03:53:04,210 - root - INFO - 0.01 seconds to create bucket dst_bucket1 2018-09-04 03:53:04,210 - root - INFO - waiting for memcached bucket : dst_bucket1 in 172.23.104.103 to accept set ops 2018-09-04 03:53:05,397 - root - INFO - creating direct client 172.23.104.103:11210 dst_bucket1 2018-09-04 03:53:05,627 - root - INFO - creating direct client 172.23.104.103:11210 dst_bucket1 2018-09-04 03:53:07,397 - root - INFO - bucket 'dst_bucket1' was created with per node RAM quota: 100 2018-09-04 03:53:07,464 - root - INFO - Load the data in older version in the initial version 2018-09-04 03:53:08,578 - root - INFO - create 4032 to src_bucket documents... 2018-09-04 03:53:08,651 - root - INFO - creating direct client 172.23.104.103:11210 src_bucket 2018-09-04 03:53:17,576 - root - INFO - LOAD IS FINISHED 2018-09-04 03:53:17,612 - root - INFO - list of eventing nodes in cluster: [ip:172.23.104.104 port:8091 ssh_username:root] 2018-09-04 03:53:17,622 - root - INFO - Deploy the function in the initial version 2018-09-04 03:53:19,012 - root - INFO - sleep for 180 secs. ... 2018-09-04 03:56:19,685 - root - INFO - adding node 172.23.104.107:8091 to cluster 2018-09-04 03:56:19,686 - root - INFO - adding remote node @172.23.104.107:8091 to this cluster @172.23.104.103:8091 2018-09-04 03:56:24,634 - root - INFO - adding node 172.23.104.108:8091 to cluster 2018-09-04 03:56:24,635 - root - INFO - adding remote node @172.23.104.108:8091 to this cluster @172.23.104.103:8091 2018-09-04 03:56:29,966 - root - INFO - adding node 172.23.104.109:8091 to cluster 2018-09-04 03:56:29,966 - root - INFO - adding remote node @172.23.104.109:8091 to this cluster @172.23.104.103:8091 2018-09-04 03:56:35,273 - root - INFO - adding node 172.23.104.90:8091 to cluster 2018-09-04 03:56:35,273 - root - INFO - adding remote node @172.23.104.90:8091 to this cluster @172.23.104.103:8091 2018-09-04 03:56:40,156 - root - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.107,ns_1@172.23.104.108,ns_1@172.23.104.106,ns_1@172.23.104.104,ns_1@172.23.104.90,ns_1@172.23.104.109,ns_1@172.23.104.105,ns_1@172.23.104.103'} 2018-09-04 03:56:40,192 - root - INFO - rebalance operation started 2018-09-04 03:56:40,201 - root - INFO - rebalance percentage : 0.00 % 2018-09-04 03:56:40,202 - root - INFO - Rebalance - status: running, progress: 0.00% 2018-09-04 03:56:50,236 - root - INFO - rebalance percentage : 1.52 % 2018-09-04 03:56:50,236 - root - INFO - Rebalance - status: running, progress: 1.52% 2018-09-04 03:57:00,274 - root - INFO - rebalance percentage : 3.39 % 2018-09-04 03:57:00,275 - root - INFO - Rebalance - status: running, progress: 3.39% 2018-09-04 03:57:10,315 - root - INFO - rebalance percentage : 5.21 % 2018-09-04 03:57:10,315 - root - INFO - Rebalance - status: running, progress: 5.21% 2018-09-04 03:57:20,353 - root - INFO - rebalance percentage : 6.45 % 2018-09-04 03:57:20,353 - root - INFO - Rebalance - status: running, progress: 6.45% 2018-09-04 03:57:30,395 - root - INFO - rebalance percentage : 7.54 % 2018-09-04 03:57:30,396 - root - INFO - Rebalance - status: running, progress: 7.54% 2018-09-04 03:57:40,434 - root - INFO - rebalance percentage : 8.58 % 2018-09-04 03:57:40,435 - root - INFO - Rebalance - status: running, progress: 8.58% 2018-09-04 03:57:50,474 - root - INFO - rebalance percentage : 10.36 % 2018-09-04 03:57:50,475 - root - INFO - Rebalance - status: running, progress: 10.36% 2018-09-04 03:58:00,517 - root - INFO - rebalance percentage : 12.15 % 2018-09-04 03:58:00,517 - root - INFO - Rebalance - status: running, progress: 12.15% 2018-09-04 03:58:10,559 - root - INFO - rebalance percentage : 13.97 % 2018-09-04 03:58:10,559 - root - INFO - Rebalance - status: running, progress: 13.97% 2018-09-04 03:58:20,600 - root - INFO - rebalance percentage : 15.03 % 2018-09-04 03:58:20,600 - root - INFO - Rebalance - status: running, progress: 15.03% 2018-09-04 03:58:30,647 - root - INFO - rebalance percentage : 16.10 % 2018-09-04 03:58:30,647 - root - INFO - Rebalance - status: running, progress: 16.10% 2018-09-04 03:58:40,687 - root - INFO - rebalance percentage : 17.15 % 2018-09-04 03:58:40,687 - root - INFO - Rebalance - status: running, progress: 17.15% 2018-09-04 03:58:50,726 - root - INFO - rebalance percentage : 18.90 % 2018-09-04 03:58:50,727 - root - INFO - Rebalance - status: running, progress: 18.90% 2018-09-04 03:59:00,773 - root - INFO - rebalance percentage : 20.58 % 2018-09-04 03:59:00,773 - root - INFO - Rebalance - status: running, progress: 20.58% 2018-09-04 03:59:10,808 - root - INFO - rebalance percentage : 22.33 % 2018-09-04 03:59:10,809 - root - INFO - Rebalance - status: running, progress: 22.33% 2018-09-04 03:59:20,850 - root - INFO - rebalance percentage : 23.32 % 2018-09-04 03:59:20,851 - root - INFO - Rebalance - status: running, progress: 23.32% 2018-09-04 03:59:30,889 - root - INFO - rebalance percentage : 24.35 % 2018-09-04 03:59:30,890 - root - INFO - Rebalance - status: running, progress: 24.35% 2018-09-04 03:59:40,932 - root - INFO - rebalance percentage : 25.29 % 2018-09-04 03:59:40,933 - root - INFO - Rebalance - status: running, progress: 25.29% 2018-09-04 03:59:50,973 - root - INFO - rebalance percentage : 26.88 % 2018-09-04 03:59:50,973 - root - INFO - Rebalance - status: running, progress: 26.88% 2018-09-04 04:00:01,014 - root - INFO - rebalance percentage : 28.49 % 2018-09-04 04:00:01,015 - root - INFO - Rebalance - status: running, progress: 28.49% 2018-09-04 04:00:11,054 - root - INFO - rebalance percentage : 30.12 % 2018-09-04 04:00:11,055 - root - INFO - Rebalance - status: running, progress: 30.12% 2018-09-04 04:00:21,099 - root - INFO - rebalance percentage : 31.26 % 2018-09-04 04:00:21,099 - root - INFO - Rebalance - status: running, progress: 31.26% 2018-09-04 04:00:31,149 - root - INFO - rebalance percentage : 32.24 % 2018-09-04 04:00:31,149 - root - INFO - Rebalance - status: running, progress: 32.24% 2018-09-04 04:00:41,188 - root - INFO - rebalance percentage : 33.21 % 2018-09-04 04:00:41,188 - root - INFO - Rebalance - status: running, progress: 33.21% 2018-09-04 04:00:51,227 - root - INFO - rebalance percentage : 66.00 % 2018-09-04 04:00:51,227 - root - INFO - Rebalance - status: running, progress: 66.00% 2018-09-04 04:01:01,266 - root - INFO - rebalance percentage : 66.93 % 2018-09-04 04:01:01,266 - root - INFO - Rebalance - status: running, progress: 66.93% 2018-09-04 04:01:11,308 - root - INFO - rebalance percentage : 68.40 % 2018-09-04 04:01:11,309 - root - INFO - Rebalance - status: running, progress: 68.40% 2018-09-04 04:01:21,355 - root - INFO - rebalance percentage : 69.94 % 2018-09-04 04:01:21,355 - root - INFO - Rebalance - status: running, progress: 69.94% 2018-09-04 04:01:31,396 - root - INFO - rebalance percentage : 71.62 % 2018-09-04 04:01:31,397 - root - INFO - Rebalance - status: running, progress: 71.62% 2018-09-04 04:01:41,446 - root - INFO - rebalance percentage : 73.87 % 2018-09-04 04:01:41,446 - root - INFO - Rebalance - status: running, progress: 73.87% 2018-09-04 04:01:51,488 - root - INFO - rebalance percentage : 75.45 % 2018-09-04 04:01:51,489 - root - INFO - Rebalance - status: running, progress: 75.45% 2018-09-04 04:02:01,531 - root - INFO - rebalance percentage : 77.64 % 2018-09-04 04:02:01,532 - root - INFO - Rebalance - status: running, progress: 77.64% 2018-09-04 04:02:11,572 - root - INFO - rebalance percentage : 79.56 % 2018-09-04 04:02:11,572 - root - INFO - Rebalance - status: running, progress: 79.56% 2018-09-04 04:02:21,613 - root - INFO - rebalance percentage : 81.16 % 2018-09-04 04:02:21,613 - root - INFO - Rebalance - status: running, progress: 81.16% 2018-09-04 04:02:31,657 - root - INFO - rebalance percentage : 83.39 % 2018-09-04 04:02:31,657 - root - INFO - Rebalance - status: running, progress: 83.39% 2018-09-04 04:02:41,696 - root - INFO - rebalance percentage : 85.25 % 2018-09-04 04:02:41,697 - root - INFO - Rebalance - status: running, progress: 85.25% 2018-09-04 04:02:51,737 - root - INFO - rebalance percentage : 86.83 % 2018-09-04 04:02:51,737 - root - INFO - Rebalance - status: running, progress: 86.83% 2018-09-04 04:03:01,779 - root - INFO - rebalance percentage : 88.26 % 2018-09-04 04:03:01,779 - root - INFO - Rebalance - status: running, progress: 88.26% 2018-09-04 04:03:11,822 - root - INFO - rebalance percentage : 89.42 % 2018-09-04 04:03:11,822 - root - INFO - Rebalance - status: running, progress: 89.42% 2018-09-04 04:03:21,861 - root - INFO - rebalance percentage : 90.46 % 2018-09-04 04:03:21,862 - root - INFO - Rebalance - status: running, progress: 90.46% 2018-09-04 04:03:31,905 - root - INFO - rebalance percentage : 91.88 % 2018-09-04 04:03:31,905 - root - INFO - Rebalance - status: running, progress: 91.88% 2018-09-04 04:03:41,941 - root - INFO - rebalance percentage : 92.93 % 2018-09-04 04:03:41,942 - root - INFO - Rebalance - status: running, progress: 92.93% 2018-09-04 04:03:51,982 - root - INFO - rebalance percentage : 93.97 % 2018-09-04 04:03:51,982 - root - INFO - Rebalance - status: running, progress: 93.97% 2018-09-04 04:04:02,020 - root - INFO - rebalance percentage : 95.45 % 2018-09-04 04:04:02,021 - root - INFO - Rebalance - status: running, progress: 95.45% 2018-09-04 04:04:12,061 - root - INFO - rebalance percentage : 96.51 % 2018-09-04 04:04:12,062 - root - INFO - Rebalance - status: running, progress: 96.51% 2018-09-04 04:04:22,099 - root - INFO - rebalance percentage : 97.54 % 2018-09-04 04:04:22,100 - root - INFO - Rebalance - status: running, progress: 97.54% 2018-09-04 04:04:32,141 - root - INFO - rebalance percentage : 98.98 % 2018-09-04 04:04:32,142 - root - INFO - Rebalance - status: running, progress: 98.98% 2018-09-04 04:04:42,181 - root - INFO - rebalance percentage : 99.68 % 2018-09-04 04:04:42,182 - root - INFO - Rebalance - status: running, progress: 99.68% 2018-09-04 04:04:52,219 - root - INFO - rebalance percentage : 99.92 % 2018-09-04 04:04:52,220 - root - INFO - Rebalance - status: running, progress: 99.92% 2018-09-04 04:05:02,259 - root - INFO - Rebalance - status: none, progress: 100.00% 2018-09-04 04:05:02,276 - root - INFO - rebalancing was completed with progress: 100% in 502.084038973 sec 2018-09-04 04:05:02,277 - root - INFO - Rebalance in all 6.0.0-1567 nodes 2018-09-04 04:05:02,280 - root - INFO - sleep for 15 secs. ... 2018-09-04 04:05:17,316 - root - INFO - /diag/eval status on 172.23.104.103:8091: True content: 'ns_1@172.23.104.107' command: mb_master:master_node(). 2018-09-04 04:05:17,333 - root - INFO - 6.0.0-1567 node 172.23.104.107 becomes the master 2018-09-04 04:05:17,345 - root - INFO - failover and rebalance nodes 2018-09-04 04:05:18,343 - root - INFO - Failing over 172.23.104.103:8091 with graceful=False 2018-09-04 04:05:21,907 - root - INFO - fail_over node ns_1@172.23.104.103 successful 2018-09-04 04:05:21,928 - root - INFO - Failing over 172.23.104.104:8091 with graceful=False 2018-09-04 04:05:22,364 - root - INFO - fail_over node ns_1@172.23.104.104 successful 2018-09-04 04:05:22,390 - root - INFO - Failing over 172.23.104.105:8091 with graceful=False 2018-09-04 04:05:22,849 - root - INFO - fail_over node ns_1@172.23.104.105 successful 2018-09-04 04:05:22,871 - root - INFO - Failing over 172.23.104.106:8091 with graceful=False 2018-09-04 04:05:23,241 - root - INFO - fail_over node ns_1@172.23.104.106 successful 2018-09-04 04:05:23,242 - root - INFO - 0 seconds sleep after failover, for nodes to go pending.... 2018-09-04 04:05:24,274 - root - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.104.103,ns_1@172.23.104.104,ns_1@172.23.104.105,ns_1@172.23.104.106', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.107,ns_1@172.23.104.108,ns_1@172.23.104.106,ns_1@172.23.104.104,ns_1@172.23.104.90,ns_1@172.23.104.109,ns_1@172.23.104.105,ns_1@172.23.104.103'} 2018-09-04 04:05:24,289 - root - INFO - rebalance operation started 2018-09-04 04:05:24,302 - root - INFO - rebalance percentage : 0.00 % 2018-09-04 04:05:24,303 - root - INFO - Rebalance - status: running, progress: 0.00% 2018-09-04 04:05:34,339 - root - INFO - Rebalance - status: none, progress: 100.00% 2018-09-04 04:05:34,345 - root - ERROR - GET http://172.23.104.103:8091/pools/default body: headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password http://172.23.104.103:8091/pools/default with status False: unknown pool 2018-09-04 04:05:34,445 - root - INFO - rebalancing was completed with progress: 100% in 10.154709816 sec 2018-09-04 04:05:34,490 - root - INFO - Output of pools/default from master node after upgrade is { "alerts": [], "alertsSilenceURL": "/controller/resetAlerts?uuid=e98c9768a5e01a07af80b3b1a1234a2e&token=0", "autoCompactionSettings": { "databaseFragmentationThreshold": { "percentage": 30, "size": "undefined" }, "indexCircularCompaction": { "daysOfWeek": "Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday", "interval": { "abortOutside": false, "fromHour": 0, "fromMinute": 0, "toHour": 0, "toMinute": 0 } }, "indexCompactionMode": "circular", "indexFragmentationThreshold": { "percentage": 30 }, "parallelDBAndViewCompaction": false, "viewFragmentationThreshold": { "percentage": 30, "size": "undefined" } }, "balanced": true, "buckets": { "terseBucketsBase": "/pools/default/b/", "terseStreamingBucketsBase": "/pools/default/bs/", "uri": "/pools/default/buckets?v=81698835&uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "cbasMemoryQuota": 1117, "checkPermissionsURI": "/pools/default/checkPermissions?v=yKoOoxPWyzw5SBu%2B1BxMUE1YdN4%3D", "clusterName": "", "controllers": { "addNode": { "uri": "/controller/addNodeV2?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "clusterLogsCollection": { "cancelURI": "/controller/cancelLogsCollection?uuid=e98c9768a5e01a07af80b3b1a1234a2e", "startURI": "/controller/startLogsCollection?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "ejectNode": { "uri": "/controller/ejectNode?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "failOver": { "uri": "/controller/failOver?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "reAddNode": { "uri": "/controller/reAddNode?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "reFailOver": { "uri": "/controller/reFailOver?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "rebalance": { "uri": "/controller/rebalance?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "replication": { "createURI": "/controller/createReplication?uuid=e98c9768a5e01a07af80b3b1a1234a2e", "validateURI": "/controller/createReplication?just_validate=1" }, "setAutoCompaction": { "uri": "/controller/setAutoCompaction?uuid=e98c9768a5e01a07af80b3b1a1234a2e", "validateURI": "/controller/setAutoCompaction?just_validate=1" }, "setRecoveryType": { "uri": "/controller/setRecoveryType?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "startGracefulFailover": { "uri": "/controller/startGracefulFailover?uuid=e98c9768a5e01a07af80b3b1a1234a2e" } }, "counters": { "failover": 4, "failover_complete": 4, "rebalance_start": 5, "rebalance_success": 4 }, "eventingMemoryQuota": 256, "ftsMemoryQuota": 256, "indexMemoryQuota": 512, "indexStatusURI": "/indexStatus?v=21137658", "maxBucketCount": 10, "memoryQuota": 2146, "name": "default", "nodeStatusesUri": "/nodeStatuses", "nodes": [ { "clusterCompatibility": 327685, "clusterMembership": "active", "couchApiBase": "http://172.23.104.107:8092/", "couchApiBaseHTTPS": "https://172.23.104.107:18092/", "cpuCount": 4, "hostname": "172.23.104.107:8091", "interestingStats": { "cmd_get": 193, "couch_docs_actual_disk_size": 149818718, "couch_docs_data_size": 121022323, "couch_spatial_data_size": 0, "couch_spatial_disk_size": 0, "couch_views_actual_disk_size": 0, "couch_views_data_size": 0, "curr_items": 9090, "curr_items_tot": 9090, "ep_bg_fetched": 0, "get_hits": 193, "mem_used": 96243008, "ops": 194, "vb_active_num_non_resident": 0, "vb_replica_curr_items": 0 }, "mcdMemoryAllocated": 3220, "mcdMemoryReserved": 3220, "memoryFree": 3040751616, "memoryTotal": 4220854272, "os": "x86_64-unknown-linux-gnu", "otpNode": "ns_1@172.23.104.107", "ports": { "direct": 11210, "httpsCAPI": 18092, "httpsMgmt": 18091, "proxy": 11211 }, "recoveryType": "none", "services": [ "kv" ], "status": "healthy", "systemStats": { "cpu_utilization_rate": 7.088607594936708, "mem_free": 3040751616, "mem_total": 4220854272, "swap_total": 3758092288, "swap_used": 0 }, "thisNode": true, "uptime": "1332", "version": "6.0.0-35007-enterprise" }, { "clusterCompatibility": 327685, "clusterMembership": "active", "couchApiBase": "http://172.23.104.108:8092/", "couchApiBaseHTTPS": "https://172.23.104.108:18092/", "cpuCount": 4, "hostname": "172.23.104.108:8091", "interestingStats": {}, "mcdMemoryAllocated": 3220, "mcdMemoryReserved": 3220, "memoryFree": 3178827776, "memoryTotal": 4220854272, "os": "x86_64-unknown-linux-gnu", "otpNode": "ns_1@172.23.104.108", "ports": { "direct": 11210, "httpsCAPI": 18092, "httpsMgmt": 18091, "proxy": 11211 }, "recoveryType": "none", "services": [ "eventing" ], "status": "healthy", "systemStats": { "cpu_utilization_rate": 51.86170212765958, "mem_free": 3178827776, "mem_total": 4220854272, "swap_total": 3758092288, "swap_used": 0 }, "uptime": "1348", "version": "6.0.0-35007-enterprise" }, { "clusterCompatibility": 327685, "clusterMembership": "active", "couchApiBase": "http://172.23.104.109:8092/", "couchApiBaseHTTPS": "https://172.23.104.109:18092/", "cpuCount": 4, "hostname": "172.23.104.109:8091", "interestingStats": {}, "mcdMemoryAllocated": 3220, "mcdMemoryReserved": 3220, "memoryFree": 3252645888, "memoryTotal": 4220854272, "os": "x86_64-unknown-linux-gnu", "otpNode": "ns_1@172.23.104.109", "ports": { "direct": 11210, "httpsCAPI": 18092, "httpsMgmt": 18091, "proxy": 11211 }, "recoveryType": "none", "services": [ "index" ], "status": "healthy", "systemStats": { "cpu_utilization_rate": 7.828282828282828, "mem_free": 3252645888, "mem_total": 4220854272, "swap_total": 3758092288, "swap_used": 0 }, "uptime": "1338", "version": "6.0.0-35007-enterprise" }, { "clusterCompatibility": 327685, "clusterMembership": "active", "couchApiBase": "http://172.23.104.90:8092/", "couchApiBaseHTTPS": "https://172.23.104.90:18092/", "cpuCount": 4, "hostname": "172.23.104.90:8091", "interestingStats": {}, "mcdMemoryAllocated": 3220, "mcdMemoryReserved": 3220, "memoryFree": 3384590336, "memoryTotal": 4220846080, "os": "x86_64-unknown-linux-gnu", "otpNode": "ns_1@172.23.104.90", "ports": { "direct": 11210, "httpsCAPI": 18092, "httpsMgmt": 18091, "proxy": 11211 }, "recoveryType": "none", "services": [ "n1ql" ], "status": "healthy", "systemStats": { "cpu_utilization_rate": 1.256281407035176, "mem_free": 3384590336, "mem_total": 4220846080, "swap_total": 3758092288, "swap_used": 87597056 }, "uptime": "1340", "version": "6.0.0-35007-enterprise" } ], "rebalanceProgressUri": "/pools/default/rebalanceProgress", "rebalanceStatus": "running", "remoteClusters": { "uri": "/pools/default/remoteClusters?uuid=e98c9768a5e01a07af80b3b1a1234a2e", "validateURI": "/pools/default/remoteClusters?just_validate=1" }, "serverGroupsUri": "/pools/default/serverGroups?v=104001697", "stopRebalanceUri": "/controller/stopRebalance?uuid=e98c9768a5e01a07af80b3b1a1234a2e", "storageTotals": { "hdd": { "free": 29284752753, "quotaTotal": 33278128128, "total": 33278128128, "used": 3993375375, "usedByData": 149818718 }, "ram": { "quotaTotal": 2250244096, "quotaTotalPerNode": 2250244096, "quotaUsed": 419430400, "quotaUsedPerNode": 419430400, "total": 4220854272, "used": 2524655616, "usedByData": 96243008 } }, "tasks": { "uri": "/pools/default/tasks?v=133628798" } } 2018-09-04 04:05:34,509 - root - INFO - list of eventing nodes in cluster: [ip:172.23.104.108 port:8091 ssh_username:root] 2018-09-04 04:05:34,543 - root - INFO - Output of pools/default from eventing node after upgrade is { "alerts": [], "alertsSilenceURL": "/controller/resetAlerts?uuid=e98c9768a5e01a07af80b3b1a1234a2e&token=0", "autoCompactionSettings": { "databaseFragmentationThreshold": { "percentage": 30, "size": "undefined" }, "indexCircularCompaction": { "daysOfWeek": "Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday", "interval": { "abortOutside": false, "fromHour": 0, "fromMinute": 0, "toHour": 0, "toMinute": 0 } }, "indexCompactionMode": "circular", "indexFragmentationThreshold": { "percentage": 30 }, "parallelDBAndViewCompaction": false, "viewFragmentationThreshold": { "percentage": 30, "size": "undefined" } }, "balanced": true, "buckets": { "terseBucketsBase": "/pools/default/b/", "terseStreamingBucketsBase": "/pools/default/bs/", "uri": "/pools/default/buckets?v=81698835&uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "cbasMemoryQuota": 1117, "checkPermissionsURI": "/pools/default/checkPermissions?v=avFH6l7U1pA5nAqQRnhYonFTkQs%3D", "clusterName": "", "controllers": { "addNode": { "uri": "/controller/addNodeV2?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "clusterLogsCollection": { "cancelURI": "/controller/cancelLogsCollection?uuid=e98c9768a5e01a07af80b3b1a1234a2e", "startURI": "/controller/startLogsCollection?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "ejectNode": { "uri": "/controller/ejectNode?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "failOver": { "uri": "/controller/failOver?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "reAddNode": { "uri": "/controller/reAddNode?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "reFailOver": { "uri": "/controller/reFailOver?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "rebalance": { "uri": "/controller/rebalance?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "replication": { "createURI": "/controller/createReplication?uuid=e98c9768a5e01a07af80b3b1a1234a2e", "validateURI": "/controller/createReplication?just_validate=1" }, "setAutoCompaction": { "uri": "/controller/setAutoCompaction?uuid=e98c9768a5e01a07af80b3b1a1234a2e", "validateURI": "/controller/setAutoCompaction?just_validate=1" }, "setRecoveryType": { "uri": "/controller/setRecoveryType?uuid=e98c9768a5e01a07af80b3b1a1234a2e" }, "startGracefulFailover": { "uri": "/controller/startGracefulFailover?uuid=e98c9768a5e01a07af80b3b1a1234a2e" } }, "counters": { "failover": 4, "failover_complete": 4, "rebalance_start": 5, "rebalance_success": 4 }, "eventingMemoryQuota": 256, "ftsMemoryQuota": 256, "indexMemoryQuota": 512, "indexStatusURI": "/indexStatus?v=21137658", "maxBucketCount": 10, "memoryQuota": 2146, "name": "default", "nodeStatusesUri": "/nodeStatuses", "nodes": [ { "clusterCompatibility": 327685, "clusterMembership": "active", "couchApiBase": "http://172.23.104.107:8092/", "couchApiBaseHTTPS": "https://172.23.104.107:18092/", "cpuCount": 4, "hostname": "172.23.104.107:8091", "interestingStats": { "cmd_get": 874, "couch_docs_actual_disk_size": 151879006, "couch_docs_data_size": 124127091, "couch_spatial_data_size": 0, "couch_spatial_disk_size": 0, "couch_views_actual_disk_size": 0, "couch_views_data_size": 0, "curr_items": 9090, "curr_items_tot": 9090, "ep_bg_fetched": 0, "get_hits": 874, "mem_used": 95839328, "ops": 1520, "vb_active_num_non_resident": 0, "vb_replica_curr_items": 0 }, "mcdMemoryAllocated": 3220, "mcdMemoryReserved": 3220, "memoryFree": 3059044352, "memoryTotal": 4220854272, "os": "x86_64-unknown-linux-gnu", "otpNode": "ns_1@172.23.104.107", "ports": { "direct": 11210, "httpsCAPI": 18092, "httpsMgmt": 18091, "proxy": 11211 }, "recoveryType": "none", "services": [ "kv" ], "status": "healthy", "systemStats": { "cpu_utilization_rate": 57.66233766233766, "mem_free": 3059044352, "mem_total": 4220854272, "swap_total": 3758092288, "swap_used": 0 }, "uptime": "1337", "version": "6.0.0-35007-enterprise" }, { "clusterCompatibility": 327685, "clusterMembership": "active", "couchApiBase": "http://172.23.104.108:8092/", "couchApiBaseHTTPS": "https://172.23.104.108:18092/", "cpuCount": 4, "hostname": "172.23.104.108:8091", "interestingStats": {}, "mcdMemoryAllocated": 3220, "mcdMemoryReserved": 3220, "memoryFree": 3178827776, "memoryTotal": 4220854272, "os": "x86_64-unknown-linux-gnu", "otpNode": "ns_1@172.23.104.108", "ports": { "direct": 11210, "httpsCAPI": 18092, "httpsMgmt": 18091, "proxy": 11211 }, "recoveryType": "none", "services": [ "eventing" ], "status": "healthy", "systemStats": { "cpu_utilization_rate": 51.86170212765958, "mem_free": 3178827776, "mem_total": 4220854272, "swap_total": 3758092288, "swap_used": 0 }, "thisNode": true, "uptime": "1348", "version": "6.0.0-35007-enterprise" }, { "clusterCompatibility": 327685, "clusterMembership": "active", "couchApiBase": "http://172.23.104.109:8092/", "couchApiBaseHTTPS": "https://172.23.104.109:18092/", "cpuCount": 4, "hostname": "172.23.104.109:8091", "interestingStats": {}, "mcdMemoryAllocated": 3220, "mcdMemoryReserved": 3220, "memoryFree": 3252645888, "memoryTotal": 4220854272, "os": "x86_64-unknown-linux-gnu", "otpNode": "ns_1@172.23.104.109", "ports": { "direct": 11210, "httpsCAPI": 18092, "httpsMgmt": 18091, "proxy": 11211 }, "recoveryType": "none", "services": [ "index" ], "status": "healthy", "systemStats": { "cpu_utilization_rate": 7.828282828282828, "mem_free": 3252645888, "mem_total": 4220854272, "swap_total": 3758092288, "swap_used": 0 }, "uptime": "1338", "version": "6.0.0-35007-enterprise" }, { "clusterCompatibility": 327685, "clusterMembership": "active", "couchApiBase": "http://172.23.104.90:8092/", "couchApiBaseHTTPS": "https://172.23.104.90:18092/", "cpuCount": 4, "hostname": "172.23.104.90:8091", "interestingStats": {}, "mcdMemoryAllocated": 3220, "mcdMemoryReserved": 3220, "memoryFree": 3384590336, "memoryTotal": 4220846080, "os": "x86_64-unknown-linux-gnu", "otpNode": "ns_1@172.23.104.90", "ports": { "direct": 11210, "httpsCAPI": 18092, "httpsMgmt": 18091, "proxy": 11211 }, "recoveryType": "none", "services": [ "n1ql" ], "status": "healthy", "systemStats": { "cpu_utilization_rate": 1.256281407035176, "mem_free": 3384590336, "mem_total": 4220846080, "swap_total": 3758092288, "swap_used": 87597056 }, "uptime": "1340", "version": "6.0.0-35007-enterprise" } ], "rebalanceProgressUri": "/pools/default/rebalanceProgress", "rebalanceStatus": "running", "remoteClusters": { "uri": "/pools/default/remoteClusters?uuid=e98c9768a5e01a07af80b3b1a1234a2e", "validateURI": "/pools/default/remoteClusters?just_validate=1" }, "serverGroupsUri": "/pools/default/serverGroups?v=104001697", "stopRebalanceUri": "/controller/stopRebalance?uuid=e98c9768a5e01a07af80b3b1a1234a2e", "storageTotals": { "hdd": { "free": 29284752753, "quotaTotal": 33278128128, "total": 33278128128, "used": 3993375375, "usedByData": 151879006 }, "ram": { "quotaTotal": 2250244096, "quotaTotalPerNode": 2250244096, "quotaUsed": 419430400, "quotaUsedPerNode": 419430400, "total": 4220854272, "used": 2524655616, "usedByData": 95839328 } }, "tasks": { "uri": "/pools/default/tasks?v=133628798" } } 2018-09-04 04:05:34,561 - root - INFO - list of eventing nodes in cluster: [ip:172.23.104.108 port:8091 ssh_username:root] 2018-09-04 04:05:34,596 - root - INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.107 **** 2018-09-04 04:05:34,737 - root - INFO - sleep for 10 secs. ... 2018-09-04 04:05:44,748 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2018-09-04 04:05:44,799 - root - INFO - sleep for 10 secs. ... 2018-09-04 04:05:57,069 - root - ERROR - POST http://172.23.104.108:8091/_p/event/setApplication/?name=test_import_function_2 body: {"depcfg": {"buckets": [{"alias": "dst_bucket", "bucket_name": "dst_bucket1"}], "source_bucket": "src_bucket", "metadata_bucket": "metadata"}, "appcode": "function OnUpdate(doc,meta) {\n var expiry = new Date();\n expiry.setSeconds(expiry.getSeconds() + 5);\n\n var context = {docID : meta.id};\n createTimer(NDtimerCallback, expiry, meta.id, context);\n}\nfunction NDtimerCallback(context) {\n dst_bucket[context.docID] = 'from NDtimerCallback';\n}", "id": 0, "settings": {"enable_recursive_mutation": false, "app_log_max_files": 10, "curl_timeout": 500, "skip_timer_threshold": 86400, "dcp_stream_boundary": "everything", "use_memory_manager": true, "persist_interval": 5000, "sock_batch_size": 100, "dcp_num_connections": 1, "enable_snapshot_smr": false, "log_level": "TRACE", "min_page_items": 50, "fuzz_offset": 0, "max_delta_chain_len": 200, "xattr_doc_timer_entry_prune_threshold": 100, "worker_feedback_queue_cap": 10000, "tick_duration": 60000, "deadline_timeout": 3, "app_log_max_size": 10485760, "max_page_items": 400, "worker_count": 3, "lss_read_ahead_size": 1048576, "deployment_status": true, "lss_cleaner_threshold": 30, "description": "", "dcp_gen_chan_size": 10000, "lss_cleaner_max_threshold": 70, "feedback_batch_size": 100, "auto_swapper": true, "worker_queue_cap": 100000, "cpp_worker_thread_count": 2, "cron_timers_per_doc": 1000, "feedback_read_buffer_size": 65536, "execution_timeout": 1, "processing_status": true, "cleanup_timers": false, "timer_processing_tick_interval": 500, "breakpad_on": true, "lcb_inst_capacity": 5, "vb_ownership_giveup_routine_count": 3, "data_chan_size": 10000, "vb_ownership_takeover_routine_count": 3, "checkpoint_interval": 10000}, "appname": "test_import_function_2"} headers: {'Content-type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 406 reason: unknown {"name":"ERR_CLUSTER_VERSION","code":42,"description":"This function syntax is unsupported on current cluster version","attributes":null,"runtime_info":{"code":42,"info":"Function requires 6.0 but cluster is at 5.5"}} auth: Administrator:password ERROR 2018-09-04 04:05:57,070 - root - WARNING - CLEANUP WAS SKIPPED DUE TO FAILURES IN UPGRADE TEST Cluster instance shutdown with force 2018-09-04 04:05:57,071 - root - INFO - Test Input params were: {'case_number': 1, 'cluster_name': 'upgrade3', 'conf_file': 'conf/eventing/eventing_upgrade.conf', 'dataset': 'default', 'doc-per-day': '2', 'get-cbcollect-info': 'True', 'groups': 'simple', 'ini': '/tmp/upgrade3.ini', 'initial_version': '5.5.0-2958', 'logs_folder': '/data/workspace/test_bala_upgrade_new1/logs/testrunner-18-Sep-04_03-43-48/test_1', 'nodes_init': '4', 'num_nodes': 8, 'skip_cleanup': 'True', 'spec': 'eventing_upgrade', 'upgrade_version': '6.0.0-1567'} 2018-09-04 04:05:57,072 - root - INFO - sleep for 20 secs. sleep 20 seconds before run next test ... ====================================================================== ERROR: test_online_upgrade_with_failover_rebalance_with_eventing (eventing.eventing_upgrade.EventingUpgrade) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/eventing/eventing_upgrade.py", line 137, in test_online_upgrade_with_failover_rebalance_with_eventing self.import_function(EXPORTED_FUNCTION.BUCKET_OP_WITH_TIMER) File "pytests/eventing/eventing_upgrade.py", line 147, in import_function self.rest.deploy_function(body["appname"], body) File "./lib/membase/api/rest_client.py", line 4116, in deploy_function raise Exception(content) Exception: {"name":"ERR_CLUSTER_VERSION","code":42,"description":"This function syntax is unsupported on current cluster version","attributes":null,"runtime_info":{"code":42,"info":"Function requires 6.0 but cluster is at 5.5"}} ---------------------------------------------------------------------- Ran 1 test in 1348.134s FAILED (errors=1) grabbing cbcollect from 172.23.104.103 2018-09-04 04:06:17,098 - root - INFO - connecting to 172.23.104.103 with username:root password:couchbase ssh_key: 2018-09-04 04:06:17,235 - root - INFO - Connected to 172.23.104.103 2018-09-04 04:06:17,555 - root - INFO - It could be other operating system. Go to check at other location Collecting logs from 172.23.104.103 2018-09-04 04:06:17,903 - root - INFO - running command.raw on 172.23.104.103: /opt/couchbase/bin/cbcollect_info 172.23.104.103-20180904-0406-diag.zip 2018-09-04 04:07:44,341 - root - INFO - command executed successfully Downloading zipped logs from 172.23.104.103 2018-09-04 04:07:46,208 - root - INFO - running command.raw on 172.23.104.103: rm -f /root/172.23.104.103-20180904-0406-diag.zip 2018-09-04 04:07:46,292 - root - INFO - command executed successfully grabbing cbcollect from 172.23.104.104 2018-09-04 04:07:46,413 - root - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: 2018-09-04 04:07:46,547 - root - INFO - Connected to 172.23.104.104 2018-09-04 04:07:46,886 - root - INFO - It could be other operating system. Go to check at other location Collecting logs from 172.23.104.104 2018-09-04 04:07:47,234 - root - INFO - running command.raw on 172.23.104.104: /opt/couchbase/bin/cbcollect_info 172.23.104.104-20180904-0407-diag.zip 2018-09-04 04:08:59,825 - root - INFO - command executed successfully Downloading zipped logs from 172.23.104.104 2018-09-04 04:09:00,406 - root - INFO - running command.raw on 172.23.104.104: rm -f /root/172.23.104.104-20180904-0407-diag.zip 2018-09-04 04:09:00,486 - root - INFO - command executed successfully grabbing cbcollect from 172.23.104.105 2018-09-04 04:09:00,492 - root - INFO - connecting to 172.23.104.105 with username:root password:couchbase ssh_key: 2018-09-04 04:09:00,625 - root - INFO - Connected to 172.23.104.105 2018-09-04 04:09:00,925 - root - INFO - It could be other operating system. Go to check at other location Collecting logs from 172.23.104.105 2018-09-04 04:09:01,312 - root - INFO - running command.raw on 172.23.104.105: /opt/couchbase/bin/cbcollect_info 172.23.104.105-20180904-0409-diag.zip 2018-09-04 04:10:10,874 - root - INFO - command executed successfully Downloading zipped logs from 172.23.104.105 2018-09-04 04:10:11,182 - root - INFO - running command.raw on 172.23.104.105: rm -f /root/172.23.104.105-20180904-0409-diag.zip 2018-09-04 04:10:11,199 - root - INFO - command executed successfully grabbing cbcollect from 172.23.104.106 2018-09-04 04:10:11,205 - root - INFO - connecting to 172.23.104.106 with username:root password:couchbase ssh_key: 2018-09-04 04:10:11,407 - root - INFO - Connected to 172.23.104.106 2018-09-04 04:10:11,757 - root - INFO - It could be other operating system. Go to check at other location Collecting logs from 172.23.104.106 2018-09-04 04:10:12,217 - root - INFO - running command.raw on 172.23.104.106: /opt/couchbase/bin/cbcollect_info 172.23.104.106-20180904-0410-diag.zip 2018-09-04 04:12:13,509 - root - INFO - command executed successfully Downloading zipped logs from 172.23.104.106 2018-09-04 04:12:14,110 - root - INFO - running command.raw on 172.23.104.106: rm -f /root/172.23.104.106-20180904-0410-diag.zip 2018-09-04 04:12:14,203 - root - INFO - command executed successfully grabbing cbcollect from 172.23.104.107 2018-09-04 04:12:14,212 - root - INFO - connecting to 172.23.104.107 with username:root password:couchbase ssh_key: 2018-09-04 04:12:14,446 - root - INFO - Connected to 172.23.104.107 2018-09-04 04:12:15,214 - root - INFO - It could be other operating system. Go to check at other location Collecting logs from 172.23.104.107 2018-09-04 04:12:16,032 - root - INFO - running command.raw on 172.23.104.107: /opt/couchbase/bin/cbcollect_info 172.23.104.107-20180904-0412-diag.zip 2018-09-04 04:17:23,952 - root - INFO - command executed successfully Downloading zipped logs from 172.23.104.107 2018-09-04 04:17:24,751 - root - INFO - running command.raw on 172.23.104.107: rm -f /root/172.23.104.107-20180904-0412-diag.zip 2018-09-04 04:17:24,884 - root - INFO - command executed successfully grabbing cbcollect from 172.23.104.108 2018-09-04 04:17:24,890 - root - INFO - connecting to 172.23.104.108 with username:root password:couchbase ssh_key: 2018-09-04 04:17:25,090 - root - INFO - Connected to 172.23.104.108 2018-09-04 04:17:25,769 - root - INFO - It could be other operating system. Go to check at other location Collecting logs from 172.23.104.108 2018-09-04 04:17:26,454 - root - INFO - running command.raw on 172.23.104.108: /opt/couchbase/bin/cbcollect_info 172.23.104.108-20180904-0417-diag.zip 2018-09-04 04:19:11,525 - root - INFO - command executed successfully Downloading zipped logs from 172.23.104.108 2018-09-04 04:19:12,602 - root - INFO - running command.raw on 172.23.104.108: rm -f /root/172.23.104.108-20180904-0417-diag.zip 2018-09-04 04:19:12,737 - root - INFO - command executed successfully grabbing cbcollect from 172.23.104.109 2018-09-04 04:19:12,856 - root - INFO - connecting to 172.23.104.109 with username:root password:couchbase ssh_key: 2018-09-04 04:19:13,052 - root - INFO - Connected to 172.23.104.109 2018-09-04 04:19:13,760 - root - INFO - It could be other operating system. Go to check at other location Collecting logs from 172.23.104.109 2018-09-04 04:19:14,459 - root - INFO - running command.raw on 172.23.104.109: /opt/couchbase/bin/cbcollect_info 172.23.104.109-20180904-0419-diag.zip 2018-09-04 04:20:48,270 - root - INFO - command executed successfully Downloading zipped logs from 172.23.104.109 2018-09-04 04:20:48,747 - root - INFO - running command.raw on 172.23.104.109: rm -f /root/172.23.104.109-20180904-0419-diag.zip 2018-09-04 04:20:48,882 - root - INFO - command executed successfully grabbing cbcollect from 172.23.104.90 2018-09-04 04:20:48,888 - root - INFO - connecting to 172.23.104.90 with username:root password:couchbase ssh_key: 2018-09-04 04:20:49,085 - root - INFO - Connected to 172.23.104.90 2018-09-04 04:20:49,643 - root - INFO - It could be other operating system. Go to check at other location Collecting logs from 172.23.104.90 2018-09-04 04:20:50,107 - root - INFO - running command.raw on 172.23.104.90: /opt/couchbase/bin/cbcollect_info 172.23.104.90-20180904-0420-diag.zip 2018-09-04 04:22:23,563 - root - INFO - command executed successfully Downloading zipped logs from 172.23.104.90 2018-09-04 04:22:23,902 - root - INFO - running command.raw on 172.23.104.90: rm -f /root/172.23.104.90-20180904-0420-diag.zip 2018-09-04 04:22:23,928 - root - INFO - command executed successfully summary so far suite eventing.eventing_upgrade.EventingUpgrade , pass 0 , fail 1 failures so far... eventing.eventing_upgrade.EventingUpgrade.test_online_upgrade_with_failover_rebalance_with_eventing testrunner logs, diags and results are available under /data/workspace/test_bala_upgrade_new1/logs/testrunner-18-Sep-04_03-43-48/test_1 Run after suite setup for eventing.eventing_upgrade.EventingUpgrade.test_online_upgrade_with_failover_rebalance_with_eventing Thread Thread-63 was not properly terminated, will be terminated now. Thread Thread-31 was not properly terminated, will be terminated now. Thread Thread-83 was not properly terminated, will be terminated now. Thread Thread-67 was not properly terminated, will be terminated now. Thread Thread-35 was not properly terminated, will be terminated now. Thread Thread-39 was not properly terminated, will be terminated now. Thread Thread-43 was not properly terminated, will be terminated now. Thread Thread-7 was not properly terminated, will be terminated now. Thread Cluster_Thread was not properly terminated, will be terminated now. Thread Thread-81 was not properly terminated, will be terminated now. Thread Thread-79 was not properly terminated, will be terminated now. Thread Thread-65 was not properly terminated, will be terminated now. Thread Thread-85 was not properly terminated, will be terminated now. Thread Thread-69 was not properly terminated, will be terminated now. Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Description set: 6.0.0-35007,product=cb,parallel=True,url=http://server.jenkins.couchbase.com/view/Toys/job/toy-unix/3227/artifact/couchbase-server-enterprise-6.0.0-35007-centos7.x86_64.rpm [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: UNSTABLE