2013-01-04 16:26:20,869 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:26:20,979 - root - INFO - command executed successfully 2013-01-04 16:26:20,981 - root - INFO - running command.raw hostname 2013-01-04 16:26:20,985 - root - INFO - command executed successfully 2013-01-04 16:26:20,986 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:26:21,096 - root - INFO - command executed successfully 2013-01-04 16:26:21,102 - root - INFO - command executed successfully 2013-01-04 16:26:21,103 - root - INFO - running command.raw hostname 2013-01-04 16:26:21,194 - root - INFO - command executed successfully 2013-01-04 16:27:02,379 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:27:02,704 - root - INFO - Connected 2013-01-04 16:27:03,633 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:27:03,796 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:03,995 - root - INFO - Connected 2013-01-04 16:27:04,026 - root - INFO - command executed successfully 2013-01-04 16:27:04,028 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,138 - root - INFO - command executed successfully 2013-01-04 16:27:04,139 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,233 - root - INFO - command executed successfully 2013-01-04 16:27:04,235 - root - INFO - running command.raw hostname 2013-01-04 16:27:04,307 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:04,354 - root - INFO - command executed successfully 2013-01-04 16:27:04,418 - root - INFO - command executed successfully 2013-01-04 16:27:04,420 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,548 - root - INFO - command executed successfully 2013-01-04 16:27:04,549 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,635 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:04,660 - root - INFO - command executed successfully 2013-01-04 16:27:04,662 - root - INFO - running command.raw hostname 2013-01-04 16:27:04,720 - root - INFO - command executed successfully 2013-01-04 16:27:04,721 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,772 - root - INFO - command executed successfully 2013-01-04 16:27:04,805 - root - INFO - command executed successfully 2013-01-04 16:27:04,806 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,919 - root - INFO - command executed successfully 2013-01-04 16:27:04,921 - root - INFO - running command.raw hostname 2013-01-04 16:27:05,036 - root - INFO - command executed successfully 2013-01-04 16:27:05,040 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:27:05,059 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:05,145 - root - INFO - command executed successfully 2013-01-04 16:27:05,147 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:05,160 - root - INFO - command executed successfully 2013-01-04 16:27:05,162 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm': No such file or directory 2013-01-04 16:27:05,162 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:27:05,163 - root - ERROR - rm: cannot remove `cb_1275': Is a directory 2013-01-04 16:27:05,164 - root - ERROR - mv: cannot stat `cb_1275/*': No such file or directory 2013-01-04 16:27:05,165 - root - INFO - get md5 sum for local and remote 2013-01-04 16:27:05,166 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm > couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l 2013-01-04 16:27:05,260 - root - INFO - command executed successfully 2013-01-04 16:27:05,262 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:05,372 - root - INFO - command executed successfully 2013-01-04 16:27:05,373 - root - INFO - running command.raw hostname 2013-01-04 16:27:05,483 - root - INFO - command executed successfully 2013-01-04 16:27:05,487 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:27:05,604 - root - INFO - command executed successfully 2013-01-04 16:27:05,606 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm': No such file or directory 2013-01-04 16:27:05,607 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:27:05,608 - root - ERROR - rm: cannot remove `cb_0006': Is a directory 2013-01-04 16:27:05,609 - root - ERROR - rm: cannot remove `default': Is a directory 2013-01-04 16:27:05,609 - root - ERROR - mv: cannot stat `cb_0006/*': No such file or directory 2013-01-04 16:27:05,610 - root - INFO - get md5 sum for local and remote 2013-01-04 16:27:05,610 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm > couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l 2013-01-04 16:27:05,773 - root - INFO - command executed successfully 2013-01-04 16:27:05,774 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm: No such file or directory 2013-01-04 16:27:05,775 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:27:05,777 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:27:06,209 - root - INFO - command executed successfully 2013-01-04 16:27:06,210 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm: No such file or directory 2013-01-04 16:27:06,211 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:27:06,212 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:30:09,539 - root - INFO - command executed successfully 2013-01-04 16:30:09,540 - root - INFO - 1d0 2013-01-04 16:30:09,541 - root - INFO - < 4fceddd71c9a4468940d6b070fcda56a couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:09,822 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:09,933 - root - INFO - command executed successfully 2013-01-04 16:30:09,934 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:10,043 - root - INFO - command executed successfully 2013-01-04 16:30:10,046 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:10,156 - root - INFO - command executed successfully 2013-01-04 16:30:10,158 - root - INFO - running command.raw hostname 2013-01-04 16:30:10,267 - root - INFO - command executed successfully 2013-01-04 16:30:10,271 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:30:10,382 - root - INFO - command executed successfully 2013-01-04 16:30:10,654 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:10,738 - root - INFO - command executed successfully 2013-01-04 16:30:10,740 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:10,851 - root - INFO - command executed successfully 2013-01-04 16:30:10,853 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:10,962 - root - INFO - command executed successfully 2013-01-04 16:30:10,964 - root - INFO - running command.raw hostname 2013-01-04 16:30:11,074 - root - INFO - command executed successfully 2013-01-04 16:30:11,078 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:30:11,185 - root - INFO - command executed successfully 2013-01-04 16:30:11,515 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:11,624 - root - INFO - command executed successfully 2013-01-04 16:30:11,626 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:11,736 - root - INFO - command executed successfully 2013-01-04 16:30:11,738 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:11,849 - root - INFO - command executed successfully 2013-01-04 16:30:11,851 - root - INFO - running command.raw hostname 2013-01-04 16:30:11,962 - root - INFO - command executed successfully 2013-01-04 16:30:11,966 - root - INFO - deliverable_type : rpm 2013-01-04 16:30:11,966 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:30:11,966 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:22,831 - root - INFO - command executed successfully 2013-01-04 16:30:22,832 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:30:22,833 - root - INFO - 2013-01-04 16:30:22,834 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:30:22,835 - root - INFO - Please browse to http://kiwi-r101:8091/ to configure your server. 2013-01-04 16:30:22,836 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:30:22,836 - root - INFO - 2013-01-04 16:30:22,837 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:30:22,837 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:30:22,838 - root - INFO - 8091 and from 21100 to 21299. 2013-01-04 16:30:22,838 - root - INFO - 2013-01-04 16:30:22,838 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:30:22,839 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:30:22,839 - root - INFO - 2013-01-04 16:30:22,839 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:30:22,923 - root - INFO - command executed successfully 2013-01-04 16:30:22,925 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:30:23,035 - root - INFO - command executed successfully 2013-01-04 16:30:23,037 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:30:24,392 - root - INFO - command executed successfully 2013-01-04 16:30:24,393 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:30:24,394 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.2 finished 2013-01-04 16:30:31,632 - root - INFO - command executed successfully 2013-01-04 16:30:31,633 - root - INFO - 1d0 2013-01-04 16:30:31,635 - root - INFO - < 4fceddd71c9a4468940d6b070fcda56a couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:31,912 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:32,022 - root - INFO - command executed successfully 2013-01-04 16:30:32,024 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:32,134 - root - INFO - command executed successfully 2013-01-04 16:30:32,135 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:32,246 - root - INFO - command executed successfully 2013-01-04 16:30:32,248 - root - INFO - running command.raw hostname 2013-01-04 16:30:32,358 - root - INFO - command executed successfully 2013-01-04 16:30:32,361 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:30:32,471 - root - INFO - command executed successfully 2013-01-04 16:30:32,747 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:32,858 - root - INFO - command executed successfully 2013-01-04 16:30:32,859 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:32,969 - root - INFO - command executed successfully 2013-01-04 16:30:32,971 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:33,056 - root - INFO - command executed successfully 2013-01-04 16:30:33,058 - root - INFO - running command.raw hostname 2013-01-04 16:30:33,168 - root - INFO - command executed successfully 2013-01-04 16:30:33,172 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:30:33,279 - root - INFO - command executed successfully 2013-01-04 16:30:33,667 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:33,777 - root - INFO - command executed successfully 2013-01-04 16:30:33,779 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:33,889 - root - INFO - command executed successfully 2013-01-04 16:30:33,892 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:34,002 - root - INFO - command executed successfully 2013-01-04 16:30:34,003 - root - INFO - running command.raw hostname 2013-01-04 16:30:34,113 - root - INFO - command executed successfully 2013-01-04 16:30:34,117 - root - INFO - deliverable_type : rpm 2013-01-04 16:30:34,117 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:30:34,118 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:46,417 - root - INFO - command executed successfully 2013-01-04 16:30:46,419 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:30:46,420 - root - INFO - 2013-01-04 16:30:46,422 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:30:46,422 - root - INFO - Please browse to http://kiwi-r103:8091/ to configure your server. 2013-01-04 16:30:46,423 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:30:46,423 - root - INFO - 2013-01-04 16:30:46,423 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:30:46,424 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:30:46,424 - root - INFO - 8091 and from 21100 to 21299. 2013-01-04 16:30:46,424 - root - INFO - 2013-01-04 16:30:46,425 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:30:46,425 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:30:46,426 - root - INFO - 2013-01-04 16:30:46,426 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:30:46,542 - root - INFO - command executed successfully 2013-01-04 16:30:46,544 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:30:46,653 - root - INFO - command executed successfully 2013-01-04 16:30:46,655 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:30:47,985 - root - INFO - command executed successfully 2013-01-04 16:30:47,986 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:30:47,988 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.4 finished 2013-01-04 16:30:52,999 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:30:53,000 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:30:53,551 - root - INFO - Connected lib/membase/api/rest_client.py:186: DeprecationWarning: object.__new__() takes no parameters obj = object.__new__(self, serverInfo) 2013-01-04 16:30:53,577 - root - INFO - Connected 2013-01-04 16:30:56,803 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:56,825 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:56,889 - root - INFO - command executed successfully 2013-01-04 16:30:56,890 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:56,917 - root - INFO - command executed successfully 2013-01-04 16:30:56,919 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:57,001 - root - INFO - command executed successfully 2013-01-04 16:30:57,002 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:57,029 - root - INFO - command executed successfully 2013-01-04 16:30:57,031 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:57,114 - root - INFO - command executed successfully 2013-01-04 16:30:57,115 - root - INFO - running command.raw hostname 2013-01-04 16:30:57,140 - root - INFO - command executed successfully 2013-01-04 16:30:57,141 - root - INFO - running command.raw hostname 2013-01-04 16:30:57,225 - root - INFO - command executed successfully 2013-01-04 16:30:57,228 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:30:57,252 - root - INFO - command executed successfully 2013-01-04 16:30:57,256 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:30:57,354 - root - INFO - command executed successfully 2013-01-04 16:30:57,355 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:30:57,387 - root - INFO - command executed successfully 2013-01-04 16:30:57,387 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:30:57,469 - root - INFO - command executed successfully 2013-01-04 16:30:57,471 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:30:57,498 - root - INFO - command executed successfully 2013-01-04 16:30:57,500 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:30:59,348 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:30:59,755 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:31:02,351 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:02,365 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.2 finished 2013-01-04 16:31:02,757 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:02,770 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.4 finished 2013-01-04 16:31:02,813 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:31:03,134 - root - INFO - Connected 2013-01-04 16:31:03,453 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:03,563 - root - INFO - command executed successfully 2013-01-04 16:31:03,565 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:03,675 - root - INFO - command executed successfully 2013-01-04 16:31:03,677 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:03,787 - root - INFO - command executed successfully 2013-01-04 16:31:03,789 - root - INFO - running command.raw hostname 2013-01-04 16:31:03,899 - root - INFO - command executed successfully 2013-01-04 16:31:04,021 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:31:04,344 - root - INFO - Connected 2013-01-04 16:31:04,655 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:04,766 - root - INFO - command executed successfully 2013-01-04 16:31:04,767 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:04,853 - root - INFO - command executed successfully 2013-01-04 16:31:04,854 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:04,965 - root - INFO - command executed successfully 2013-01-04 16:31:04,966 - root - INFO - running command.raw hostname 2013-01-04 16:31:05,076 - root - INFO - command executed successfully 2013-01-04 16:31:05,206 - root - INFO - Installation of old version is done. Wait for 10 sec for upgrade 2013-01-04 16:31:05,382 - root - INFO - adding node 172.27.33.4:8091 to cluster 2013-01-04 16:31:05,382 - root - INFO - adding remote node @172.27.33.4:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:31:05,913 - root - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.27.33.2%2Cns_1%40172.27.33.4 2013-01-04 16:31:05,918 - root - INFO - rebalance operation started 2013-01-04 16:31:05,927 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:31:15,943 - root - INFO - rebalancing was completed with progress: 100% in 10.0244619846 sec 2013-01-04 16:31:16,953 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:16,964 - root - INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password 2013-01-04 16:31:16,988 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:16,999 - root - INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password 2013-01-04 16:31:18,012 - root - INFO - http://172.27.33.2:8091/pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=None&flushEnabled=1&replicaNumber=1&ramQuotaMB=1456 2013-01-04 16:31:18,026 - root - INFO - 0.0136511325836 seconds to create bucket default 2013-01-04 16:31:18,026 - root - INFO - waiting for memcached bucket : default in 172.27.33.2 to accept set ops 2013-01-04 16:31:19,215 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,309 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:19,439 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,525 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:19,666 - root - INFO - bucket 'default' was created with per node RAM quota: 1456 2013-01-04 16:31:19,734 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,817 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:20,669 - root - INFO - Batch create documents done #: 0 2013-01-04 16:31:21,093 - root - INFO - Batch create documents done #: 1000 2013-01-04 16:31:21,687 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:21,759 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.27.33.2:8091' 2013-01-04 16:31:21,773 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:21,852 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '172.27.33.2:8091' 2013-01-04 16:31:21,867 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:21,944 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.27.33.4:8091' 2013-01-04 16:31:21,961 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:22,038 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '172.27.33.4:8091' 2013-01-04 16:31:32,051 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,052 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,053 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,813 - root - INFO - Connected 2013-01-04 16:31:32,815 - root - INFO - Connected 2013-01-04 16:31:32,815 - root - INFO - Connected 2013-01-04 16:31:33,168 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,172 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,196 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,255 - root - INFO - command executed successfully 2013-01-04 16:31:33,257 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,261 - root - INFO - command executed successfully 2013-01-04 16:31:33,262 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,312 - root - INFO - command executed successfully 2013-01-04 16:31:33,314 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,367 - root - INFO - command executed successfully 2013-01-04 16:31:33,369 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,375 - root - INFO - command executed successfully 2013-01-04 16:31:33,376 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,425 - root - INFO - command executed successfully 2013-01-04 16:31:33,426 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,466 - root - INFO - command executed successfully 2013-01-04 16:31:33,467 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,481 - root - INFO - command executed successfully 2013-01-04 16:31:33,482 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,514 - root - INFO - command executed successfully 2013-01-04 16:31:33,515 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,583 - root - INFO - command executed successfully 2013-01-04 16:31:33,588 - root - INFO - CentOS 2013-01-04 16:31:33,589 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,591 - root - INFO - command executed successfully 2013-01-04 16:31:33,595 - root - INFO - CentOS 2013-01-04 16:31:33,596 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,627 - root - INFO - command executed successfully 2013-01-04 16:31:33,630 - root - INFO - CentOS 2013-01-04 16:31:33,631 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,991 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,996 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:34,008 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:34,080 - root - INFO - command executed successfully 2013-01-04 16:31:34,081 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,084 - root - INFO - command executed successfully 2013-01-04 16:31:34,085 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,123 - root - INFO - command executed successfully 2013-01-04 16:31:34,124 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,172 - root - INFO - command executed successfully 2013-01-04 16:31:34,173 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,175 - root - INFO - command executed successfully 2013-01-04 16:31:34,176 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,237 - root - INFO - command executed successfully 2013-01-04 16:31:34,239 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,265 - root - INFO - command executed successfully 2013-01-04 16:31:34,266 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,268 - root - INFO - command executed successfully 2013-01-04 16:31:34,269 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,351 - root - INFO - command executed successfully 2013-01-04 16:31:34,357 - root - INFO - command executed successfully 2013-01-04 16:31:34,358 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,359 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,370 - root - INFO - command executed successfully 2013-01-04 16:31:34,373 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,459 - root - INFO - command executed successfully 2013-01-04 16:31:34,461 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,461 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,465 - root - INFO - command executed successfully 2013-01-04 16:31:34,469 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,502 - root - INFO - command executed successfully 2013-01-04 16:31:34,504 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,505 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,574 - root - INFO - command executed successfully 2013-01-04 16:31:34,576 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,577 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,583 - root - INFO - command executed successfully 2013-01-04 16:31:34,584 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,585 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,617 - root - INFO - command executed successfully 2013-01-04 16:31:34,618 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,619 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,689 - root - INFO - command executed successfully 2013-01-04 16:31:34,691 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,699 - root - INFO - command executed successfully 2013-01-04 16:31:34,700 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,701 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,707 - root - INFO - command executed successfully 2013-01-04 16:31:34,708 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,800 - root - INFO - command executed successfully 2013-01-04 16:31:34,801 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,813 - root - INFO - command executed successfully 2013-01-04 16:31:34,814 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,820 - root - INFO - command executed successfully 2013-01-04 16:31:34,821 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,911 - root - INFO - command executed successfully 2013-01-04 16:31:34,912 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:34,914 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:34,924 - root - INFO - command executed successfully 2013-01-04 16:31:34,925 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,936 - root - INFO - command executed successfully 2013-01-04 16:31:34,937 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:34,938 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:35,015 - root - INFO - command executed successfully 2013-01-04 16:31:35,016 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:35,018 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:35,025 - root - INFO - command executed successfully 2013-01-04 16:31:35,026 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,027 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,053 - root - INFO - command executed successfully 2013-01-04 16:31:35,054 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,055 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,130 - root - INFO - command executed successfully 2013-01-04 16:31:35,131 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,132 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,140 - root - INFO - command executed successfully 2013-01-04 16:31:35,142 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,169 - root - INFO - command executed successfully 2013-01-04 16:31:35,170 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,219 - root - INFO - command executed successfully 2013-01-04 16:31:35,221 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,251 - root - INFO - command executed successfully 2013-01-04 16:31:35,252 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,281 - root - INFO - command executed successfully 2013-01-04 16:31:35,282 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,337 - root - INFO - command executed successfully 2013-01-04 16:31:35,338 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,362 - root - INFO - command executed successfully 2013-01-04 16:31:35,364 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,394 - root - INFO - command executed successfully 2013-01-04 16:31:35,395 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,454 - root - INFO - command executed successfully 2013-01-04 16:31:35,456 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,475 - root - INFO - command executed successfully 2013-01-04 16:31:35,476 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,511 - root - INFO - command executed successfully 2013-01-04 16:31:35,512 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,570 - root - INFO - command executed successfully 2013-01-04 16:31:35,571 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,587 - root - INFO - command executed successfully 2013-01-04 16:31:35,588 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,602 - root - INFO - command executed successfully 2013-01-04 16:31:35,603 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,686 - root - INFO - command executed successfully 2013-01-04 16:31:35,688 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,698 - root - INFO - command executed successfully 2013-01-04 16:31:35,718 - root - INFO - command executed successfully 2013-01-04 16:31:35,815 - root - INFO - command executed successfully 2013-01-04 16:31:36,006 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,074 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,089 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,095 - root - INFO - command executed successfully 2013-01-04 16:31:36,097 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,180 - root - INFO - command executed successfully 2013-01-04 16:31:36,181 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,189 - root - INFO - command executed successfully 2013-01-04 16:31:36,189 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,207 - root - INFO - command executed successfully 2013-01-04 16:31:36,208 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,271 - root - INFO - command executed successfully 2013-01-04 16:31:36,272 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,281 - root - INFO - command executed successfully 2013-01-04 16:31:36,283 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,318 - root - INFO - command executed successfully 2013-01-04 16:31:36,319 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,385 - root - INFO - command executed successfully 2013-01-04 16:31:36,388 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,395 - root - INFO - command executed successfully 2013-01-04 16:31:36,396 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,430 - root - INFO - command executed successfully 2013-01-04 16:31:36,434 - root - INFO - CentOS 2013-01-04 16:31:36,435 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,435 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:36,504 - root - INFO - command executed successfully 2013-01-04 16:31:36,509 - root - INFO - CentOS 2013-01-04 16:31:36,509 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,509 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:36,512 - root - INFO - command executed successfully 2013-01-04 16:31:36,515 - root - INFO - CentOS 2013-01-04 16:31:36,516 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,517 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:39,326 - root - INFO - command executed successfully 2013-01-04 16:31:39,327 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,328 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,329 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,330 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,331 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,332 - root - INFO - 2013-01-04 16:31:39,332 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,341 - root - INFO - command executed successfully 2013-01-04 16:31:39,342 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,342 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,343 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,343 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,344 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,345 - root - INFO - 2013-01-04 16:31:39,346 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,381 - root - INFO - command executed successfully 2013-01-04 16:31:39,382 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,382 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,384 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,386 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,386 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,387 - root - INFO - 2013-01-04 16:31:39,387 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,447 - root - INFO - command executed successfully 2013-01-04 16:31:39,449 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,458 - root - INFO - command executed successfully 2013-01-04 16:31:39,460 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,502 - root - INFO - command executed successfully 2013-01-04 16:31:39,504 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,559 - root - INFO - command executed successfully 2013-01-04 16:31:39,561 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,571 - root - INFO - command executed successfully 2013-01-04 16:31:39,573 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,613 - root - INFO - command executed successfully 2013-01-04 16:31:39,614 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,674 - root - INFO - command executed successfully 2013-01-04 16:31:39,676 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,677 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,683 - root - INFO - command executed successfully 2013-01-04 16:31:39,685 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,686 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,727 - root - INFO - command executed successfully 2013-01-04 16:31:39,729 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,730 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,787 - root - INFO - command executed successfully 2013-01-04 16:31:39,788 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,790 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,796 - root - INFO - command executed successfully 2013-01-04 16:31:39,798 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,799 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,842 - root - INFO - command executed successfully 2013-01-04 16:31:39,843 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,844 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,905 - root - INFO - command executed successfully 2013-01-04 16:31:39,907 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,908 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:39,909 - root - INFO - command executed successfully 2013-01-04 16:31:39,912 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,912 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:39,962 - root - INFO - command executed successfully 2013-01-04 16:31:39,963 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,964 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:40,024 - root - INFO - command executed successfully 2013-01-04 16:31:40,026 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,029 - root - INFO - command executed successfully 2013-01-04 16:31:40,031 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,082 - root - INFO - command executed successfully 2013-01-04 16:31:40,083 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,137 - root - INFO - command executed successfully 2013-01-04 16:31:40,138 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,140 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,143 - root - INFO - command executed successfully 2013-01-04 16:31:40,144 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,145 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,195 - root - INFO - command executed successfully 2013-01-04 16:31:40,196 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,196 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,253 - root - INFO - command executed successfully 2013-01-04 16:31:40,254 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,255 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,260 - root - INFO - command executed successfully 2013-01-04 16:31:40,261 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,263 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,307 - root - INFO - command executed successfully 2013-01-04 16:31:40,308 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,309 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,367 - root - INFO - command executed successfully 2013-01-04 16:31:40,368 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,377 - root - INFO - command executed successfully 2013-01-04 16:31:40,379 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,424 - root - INFO - command executed successfully 2013-01-04 16:31:40,425 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,481 - root - INFO - command executed successfully 2013-01-04 16:31:40,483 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,496 - root - INFO - command executed successfully 2013-01-04 16:31:40,497 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,543 - root - INFO - command executed successfully 2013-01-04 16:31:40,544 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,580 - root - INFO - command executed successfully 2013-01-04 16:31:40,581 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,593 - root - INFO - command executed successfully 2013-01-04 16:31:40,594 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,659 - root - INFO - command executed successfully 2013-01-04 16:31:40,660 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,697 - root - INFO - command executed successfully 2013-01-04 16:31:40,699 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,707 - root - INFO - command executed successfully 2013-01-04 16:31:40,708 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,777 - root - INFO - command executed successfully 2013-01-04 16:31:40,778 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,815 - root - INFO - command executed successfully 2013-01-04 16:31:40,817 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,821 - root - INFO - command executed successfully 2013-01-04 16:31:40,823 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,892 - root - INFO - command executed successfully 2013-01-04 16:31:40,894 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,934 - root - INFO - command executed successfully 2013-01-04 16:31:40,937 - root - INFO - command executed successfully 2013-01-04 16:31:41,008 - root - INFO - command executed successfully 2013-01-04 16:31:41,317 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,321 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,345 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,402 - root - INFO - command executed successfully 2013-01-04 16:31:41,404 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,437 - root - INFO - command executed successfully 2013-01-04 16:31:41,439 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,459 - root - INFO - command executed successfully 2013-01-04 16:31:41,461 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,515 - root - INFO - command executed successfully 2013-01-04 16:31:41,517 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,553 - root - INFO - command executed successfully 2013-01-04 16:31:41,555 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,575 - root - INFO - command executed successfully 2013-01-04 16:31:41,576 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,628 - root - INFO - command executed successfully 2013-01-04 16:31:41,629 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,669 - root - INFO - command executed successfully 2013-01-04 16:31:41,672 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,693 - root - INFO - command executed successfully 2013-01-04 16:31:41,694 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,741 - root - INFO - command executed successfully 2013-01-04 16:31:41,745 - root - INFO - CentOS 2013-01-04 16:31:41,746 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,746 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,786 - root - INFO - command executed successfully 2013-01-04 16:31:41,790 - root - INFO - CentOS 2013-01-04 16:31:41,790 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,791 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,809 - root - INFO - command executed successfully 2013-01-04 16:31:41,811 - root - INFO - CentOS 2013-01-04 16:31:41,812 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,813 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,869 - root - INFO - command executed successfully 2013-01-04 16:31:41,871 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,872 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,909 - root - INFO - command executed successfully 2013-01-04 16:31:41,911 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,912 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,929 - root - INFO - command executed successfully 2013-01-04 16:31:41,931 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,932 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,980 - root - INFO - command executed successfully 2013-01-04 16:31:41,982 - root - ERROR - beam: no process killed 2013-01-04 16:31:41,983 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,027 - root - INFO - command executed successfully 2013-01-04 16:31:42,029 - root - ERROR - beam: no process killed 2013-01-04 16:31:42,030 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,046 - root - INFO - command executed successfully 2013-01-04 16:31:42,047 - root - ERROR - beam: no process killed 2013-01-04 16:31:42,049 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,094 - root - INFO - command executed successfully 2013-01-04 16:31:42,096 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,097 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,147 - root - INFO - command executed successfully 2013-01-04 16:31:42,149 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,149 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,159 - root - INFO - command executed successfully 2013-01-04 16:31:42,160 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,161 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,208 - root - INFO - command executed successfully 2013-01-04 16:31:42,263 - root - INFO - command executed successfully 2013-01-04 16:31:42,276 - root - INFO - command executed successfully thread uninstaller-thread-172.27.33.3 finished thread uninstaller-thread-172.27.33.6 finished thread uninstaller-thread-172.27.33.5 finished 2013-01-04 16:31:42,278 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,279 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,279 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,852 - root - INFO - Connected 2013-01-04 16:31:42,853 - root - INFO - Connected 2013-01-04 16:31:43,045 - root - INFO - Connected 2013-01-04 16:31:43,351 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,371 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,418 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,453 - root - INFO - command executed successfully 2013-01-04 16:31:43,455 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,462 - root - INFO - command executed successfully 2013-01-04 16:31:43,463 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,505 - root - INFO - command executed successfully 2013-01-04 16:31:43,507 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,568 - root - INFO - command executed successfully 2013-01-04 16:31:43,569 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,580 - root - INFO - command executed successfully 2013-01-04 16:31:43,581 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,622 - root - INFO - command executed successfully 2013-01-04 16:31:43,623 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,680 - root - INFO - command executed successfully 2013-01-04 16:31:43,681 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,699 - root - INFO - command executed successfully 2013-01-04 16:31:43,700 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,738 - root - INFO - command executed successfully 2013-01-04 16:31:43,739 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,787 - root - INFO - command executed successfully 2013-01-04 16:31:43,793 - root - INFO - command executed successfully 2013-01-04 16:31:43,828 - root - INFO - command executed successfully 2013-01-04 16:32:46,827 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:32:47,158 - root - INFO - Connected 2013-01-04 16:32:47,469 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:48,298 - root - INFO - command executed successfully 2013-01-04 16:32:48,325 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:48,804 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:32:48,819 - root - INFO - command executed successfully 2013-01-04 16:32:48,845 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:49,056 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:32:49,172 - root - INFO - command executed successfully 2013-01-04 16:32:49,173 - root - INFO - running command.raw hostname 2013-01-04 16:32:49,380 - root - INFO - Connected 2013-01-04 16:32:49,567 - root - INFO - command executed successfully 2013-01-04 16:32:49,581 - root - INFO - Connected 2013-01-04 16:32:49,923 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:49,930 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:49,937 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,013 - root - INFO - command executed successfully 2013-01-04 16:32:50,014 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,039 - root - INFO - command executed successfully 2013-01-04 16:32:50,040 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,053 - root - INFO - command executed successfully 2013-01-04 16:32:50,054 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,125 - root - INFO - command executed successfully 2013-01-04 16:32:50,126 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,153 - root - INFO - command executed successfully 2013-01-04 16:32:50,155 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,166 - root - INFO - command executed successfully 2013-01-04 16:32:50,167 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,213 - root - INFO - command executed successfully 2013-01-04 16:32:50,215 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,270 - root - INFO - command executed successfully 2013-01-04 16:32:50,271 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,283 - root - INFO - command executed successfully 2013-01-04 16:32:50,285 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,325 - root - INFO - command executed successfully 2013-01-04 16:32:50,330 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:50,384 - root - INFO - command executed successfully 2013-01-04 16:32:50,402 - root - INFO - command executed successfully 2013-01-04 16:32:50,450 - root - INFO - command executed successfully 2013-01-04 16:32:50,452 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:50,453 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:50,454 - root - ERROR - rm: cannot remove `cb_8077': Is a directory 2013-01-04 16:32:50,455 - root - ERROR - mv: cannot stat `cb_8077/*': No such file or directory 2013-01-04 16:32:50,456 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:50,457 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:50,702 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,716 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,804 - root - INFO - command executed successfully 2013-01-04 16:32:50,805 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,814 - root - INFO - command executed successfully 2013-01-04 16:32:50,816 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,920 - root - INFO - command executed successfully 2013-01-04 16:32:50,922 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,928 - root - INFO - command executed successfully 2013-01-04 16:32:50,930 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:51,038 - root - INFO - command executed successfully 2013-01-04 16:32:51,040 - root - INFO - running command.raw hostname 2013-01-04 16:32:51,043 - root - INFO - command executed successfully 2013-01-04 16:32:51,045 - root - INFO - running command.raw hostname 2013-01-04 16:32:51,059 - root - INFO - command executed successfully 2013-01-04 16:32:51,060 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,062 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,063 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:32:51,157 - root - INFO - command executed successfully 2013-01-04 16:32:51,161 - root - INFO - command executed successfully 2013-01-04 16:32:51,165 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:51,168 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:51,254 - root - INFO - command executed successfully 2013-01-04 16:32:51,256 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:51,257 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:51,258 - root - ERROR - rm: cannot remove `cb_1951': Is a directory 2013-01-04 16:32:51,258 - root - ERROR - mv: cannot stat `cb_1951/*': No such file or directory 2013-01-04 16:32:51,259 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:51,259 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:51,287 - root - INFO - command executed successfully 2013-01-04 16:32:51,289 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:51,290 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:51,291 - root - ERROR - rm: cannot remove `cb_1410': Is a directory 2013-01-04 16:32:51,291 - root - ERROR - mv: cannot stat `cb_1410/*': No such file or directory 2013-01-04 16:32:51,292 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:51,292 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:51,842 - root - INFO - command executed successfully 2013-01-04 16:32:51,843 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,845 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,846 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:32:51,869 - root - INFO - command executed successfully 2013-01-04 16:32:51,870 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,872 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,872 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:36:32,684 - root - INFO - command executed successfully 2013-01-04 16:36:32,686 - root - INFO - 1d0 2013-01-04 16:36:32,687 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:32,951 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:33,068 - root - INFO - command executed successfully 2013-01-04 16:36:33,069 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:33,184 - root - INFO - command executed successfully 2013-01-04 16:36:33,186 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:33,300 - root - INFO - command executed successfully 2013-01-04 16:36:33,302 - root - INFO - running command.raw hostname 2013-01-04 16:36:33,417 - root - INFO - command executed successfully 2013-01-04 16:36:33,421 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:33,528 - root - INFO - command executed successfully 2013-01-04 16:36:33,779 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:33,893 - root - INFO - command executed successfully 2013-01-04 16:36:33,894 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:34,008 - root - INFO - command executed successfully 2013-01-04 16:36:34,010 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:34,125 - root - INFO - command executed successfully 2013-01-04 16:36:34,126 - root - INFO - running command.raw hostname 2013-01-04 16:36:34,236 - root - INFO - command executed successfully 2013-01-04 16:36:34,240 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:34,347 - root - INFO - command executed successfully 2013-01-04 16:36:34,645 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:34,736 - root - INFO - command executed successfully 2013-01-04 16:36:34,737 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:34,853 - root - INFO - command executed successfully 2013-01-04 16:36:34,854 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:34,967 - root - INFO - command executed successfully 2013-01-04 16:36:34,969 - root - INFO - running command.raw hostname 2013-01-04 16:36:35,082 - root - INFO - command executed successfully 2013-01-04 16:36:35,086 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:35,087 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:35,088 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:44,640 - root - INFO - command executed successfully 2013-01-04 16:36:44,641 - root - INFO - 1d0 2013-01-04 16:36:44,642 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:44,920 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:45,034 - root - INFO - command executed successfully 2013-01-04 16:36:45,035 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:45,152 - root - INFO - command executed successfully 2013-01-04 16:36:45,155 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:45,273 - root - INFO - command executed successfully 2013-01-04 16:36:45,274 - root - INFO - running command.raw hostname 2013-01-04 16:36:45,387 - root - INFO - command executed successfully 2013-01-04 16:36:45,391 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:45,499 - root - INFO - command executed successfully 2013-01-04 16:36:45,748 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:45,864 - root - INFO - command executed successfully 2013-01-04 16:36:45,865 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:45,980 - root - INFO - command executed successfully 2013-01-04 16:36:45,981 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:45,985 - root - INFO - command executed successfully 2013-01-04 16:36:45,986 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:36:45,988 - root - INFO - 2013-01-04 16:36:45,989 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:36:45,989 - root - INFO - Please browse to http://kiwi-r102:8091/ to configure your server. 2013-01-04 16:36:45,991 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:36:45,992 - root - INFO - 2013-01-04 16:36:45,992 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:36:45,994 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:36:45,995 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:36:45,995 - root - INFO - 2013-01-04 16:36:45,996 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:36:45,996 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:36:45,996 - root - INFO - 2013-01-04 16:36:45,997 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:36:46,074 - root - INFO - command executed successfully 2013-01-04 16:36:46,075 - root - INFO - running command.raw hostname 2013-01-04 16:36:46,080 - root - INFO - command executed successfully 2013-01-04 16:36:46,081 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:36:46,187 - root - INFO - command executed successfully 2013-01-04 16:36:46,190 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:46,197 - root - INFO - command executed successfully 2013-01-04 16:36:46,197 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:36:46,305 - root - INFO - command executed successfully 2013-01-04 16:36:46,616 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:46,704 - root - INFO - command executed successfully 2013-01-04 16:36:46,705 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:46,815 - root - INFO - command executed successfully 2013-01-04 16:36:46,817 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:46,930 - root - INFO - command executed successfully 2013-01-04 16:36:46,931 - root - INFO - running command.raw hostname 2013-01-04 16:36:47,046 - root - INFO - command executed successfully 2013-01-04 16:36:47,050 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:47,050 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:47,052 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:47,500 - root - INFO - command executed successfully 2013-01-04 16:36:47,501 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:36:47,503 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.3 finished 2013-01-04 16:36:56,036 - root - INFO - command executed successfully 2013-01-04 16:36:56,037 - root - INFO - 1d0 2013-01-04 16:36:56,039 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:56,319 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:56,432 - root - INFO - command executed successfully 2013-01-04 16:36:56,433 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:56,544 - root - INFO - command executed successfully 2013-01-04 16:36:56,545 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:56,632 - root - INFO - command executed successfully 2013-01-04 16:36:56,633 - root - INFO - running command.raw hostname 2013-01-04 16:36:56,719 - root - INFO - command executed successfully 2013-01-04 16:36:56,723 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:56,830 - root - INFO - command executed successfully 2013-01-04 16:36:57,103 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:57,188 - root - INFO - command executed successfully 2013-01-04 16:36:57,189 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:57,299 - root - INFO - command executed successfully 2013-01-04 16:36:57,301 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:57,412 - root - INFO - command executed successfully 2013-01-04 16:36:57,414 - root - INFO - running command.raw hostname 2013-01-04 16:36:57,524 - root - INFO - command executed successfully 2013-01-04 16:36:57,528 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:57,636 - root - INFO - command executed successfully 2013-01-04 16:36:57,963 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:58,048 - root - INFO - command executed successfully 2013-01-04 16:36:58,049 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:58,160 - root - INFO - command executed successfully 2013-01-04 16:36:58,161 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:58,271 - root - INFO - command executed successfully 2013-01-04 16:36:58,272 - root - INFO - running command.raw hostname 2013-01-04 16:36:58,383 - root - INFO - command executed successfully 2013-01-04 16:36:58,386 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:58,386 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:58,387 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:58,479 - root - INFO - command executed successfully 2013-01-04 16:36:58,480 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:36:58,481 - root - INFO - 2013-01-04 16:36:58,482 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:36:58,482 - root - INFO - Please browse to http://kiwi-r105:8091/ to configure your server. 2013-01-04 16:36:58,483 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:36:58,484 - root - INFO - 2013-01-04 16:36:58,485 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:36:58,486 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:36:58,486 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:36:58,486 - root - INFO - 2013-01-04 16:36:58,487 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:36:58,488 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:36:58,488 - root - INFO - 2013-01-04 16:36:58,489 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:36:58,569 - root - INFO - command executed successfully 2013-01-04 16:36:58,571 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:36:58,685 - root - INFO - command executed successfully 2013-01-04 16:36:58,687 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:37:00,004 - root - INFO - command executed successfully 2013-01-04 16:37:00,005 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:37:00,006 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.6 finished 2013-01-04 16:37:09,052 - root - INFO - command executed successfully 2013-01-04 16:37:09,054 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:37:09,055 - root - INFO - 2013-01-04 16:37:09,056 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:37:09,057 - root - INFO - Please browse to http://kiwi-r104:8091/ to configure your server. 2013-01-04 16:37:09,058 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:37:09,058 - root - INFO - 2013-01-04 16:37:09,059 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:37:09,059 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:37:09,059 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:37:09,060 - root - INFO - 2013-01-04 16:37:09,060 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:37:09,060 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:37:09,061 - root - INFO - 2013-01-04 16:37:09,061 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:37:09,177 - root - INFO - command executed successfully 2013-01-04 16:37:09,179 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:37:09,289 - root - INFO - command executed successfully 2013-01-04 16:37:09,291 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:37:10,642 - root - INFO - command executed successfully 2013-01-04 16:37:10,644 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:37:10,645 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.5 finished 2013-01-04 16:37:15,657 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:37:15,658 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:37:15,658 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:37:16,228 - root - INFO - Connected 2013-01-04 16:37:16,235 - root - INFO - Connected 2013-01-04 16:37:16,430 - root - INFO - Connected 2013-01-04 16:37:19,746 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,749 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,759 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,838 - root - INFO - command executed successfully 2013-01-04 16:37:19,840 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,845 - root - INFO - command executed successfully 2013-01-04 16:37:19,846 - root - INFO - command executed successfully 2013-01-04 16:37:19,849 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,848 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,935 - root - INFO - command executed successfully 2013-01-04 16:37:19,936 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:19,962 - root - INFO - command executed successfully 2013-01-04 16:37:19,964 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:19,968 - root - INFO - command executed successfully 2013-01-04 16:37:19,969 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:20,047 - root - INFO - command executed successfully 2013-01-04 16:37:20,048 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,078 - root - INFO - command executed successfully 2013-01-04 16:37:20,080 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,081 - root - INFO - command executed successfully 2013-01-04 16:37:20,086 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,163 - root - INFO - command executed successfully 2013-01-04 16:37:20,167 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,198 - root - INFO - command executed successfully 2013-01-04 16:37:20,200 - root - INFO - command executed successfully 2013-01-04 16:37:20,206 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,208 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,288 - root - INFO - command executed successfully 2013-01-04 16:37:20,289 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,299 - root - INFO - command executed successfully 2013-01-04 16:37:20,301 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,303 - root - INFO - command executed successfully 2013-01-04 16:37:20,304 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,377 - root - INFO - command executed successfully 2013-01-04 16:37:20,378 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:20,393 - root - INFO - command executed successfully 2013-01-04 16:37:20,394 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:20,416 - root - INFO - command executed successfully 2013-01-04 16:37:20,417 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:22,371 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:22,375 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:22,510 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:25,374 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,378 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,390 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password 2013-01-04 16:37:25,396 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.3 finished thread initializer-thread-172.27.33.6 finished 2013-01-04 16:37:25,514 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,528 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.5 finished 2013-01-04 16:37:25,607 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:37:25,932 - root - INFO - Connected 2013-01-04 16:37:26,243 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:26,359 - root - INFO - command executed successfully 2013-01-04 16:37:26,360 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:26,474 - root - INFO - command executed successfully 2013-01-04 16:37:26,476 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:26,586 - root - INFO - command executed successfully 2013-01-04 16:37:26,589 - root - INFO - running command.raw hostname 2013-01-04 16:37:26,698 - root - INFO - command executed successfully 2013-01-04 16:37:26,818 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:37:27,142 - root - INFO - Connected 2013-01-04 16:37:27,416 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:27,504 - root - INFO - command executed successfully 2013-01-04 16:37:27,505 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:27,619 - root - INFO - command executed successfully 2013-01-04 16:37:27,620 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:27,734 - root - INFO - command executed successfully 2013-01-04 16:37:27,736 - root - INFO - running command.raw hostname 2013-01-04 16:37:27,849 - root - INFO - command executed successfully 2013-01-04 16:37:27,972 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:37:28,296 - root - INFO - Connected 2013-01-04 16:37:28,614 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:28,698 - root - INFO - command executed successfully 2013-01-04 16:37:28,700 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:28,809 - root - INFO - command executed successfully 2013-01-04 16:37:28,811 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:28,922 - root - INFO - command executed successfully 2013-01-04 16:37:28,924 - root - INFO - running command.raw hostname 2013-01-04 16:37:29,033 - root - INFO - command executed successfully 2013-01-04 16:37:29,156 - root - INFO - Installation of new version is done. Wait for 10 sec for rebalance 2013-01-04 16:51:41,243 - root - INFO - fail_over successful 2013-01-04 16:52:36,258 - root - INFO - adding node 172.27.33.3:8091 to cluster 2013-01-04 16:52:36,258 - root - INFO - adding remote node @172.27.33.3:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:37,575 - root - INFO - adding node 172.27.33.6:8091 to cluster 2013-01-04 16:52:37,576 - root - INFO - adding remote node @172.27.33.6:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:38,548 - root - INFO - adding node 172.27.33.5:8091 to cluster 2013-01-04 16:52:38,549 - root - INFO - adding remote node @172.27.33.5:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:39,763 - root - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5%2Cns_1%40172.27.33.2%2Cns_1%40172.27.33.4 2013-01-04 16:52:39,773 - root - INFO - rebalance operation started 2013-01-04 16:52:39,779 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:52:49,796 - root - INFO - rebalancing was completed with progress: 100% in 10.02257514 sec 2013-01-04 16:53:36,118 - root - INFO - Rebalance in all 2.0 Nodes 2013-01-04 16:53:47,639 - root - INFO - /diag/eval status: True content: 'ns_1@172.27.33.2' command: node(global:whereis_name(ns_orchestrator)) ERROR 2013-01-04 16:55:40,065 - root - INFO - ============== basetestcase cleanup was started for test #1 online_upgrade_rebalance_in_out ============== 2013-01-04 16:55:40,101 - root - INFO - deleting existing buckets on [ip:172.27.33.2 port:8091 ssh_username:root, ip:172.27.33.3 port:8091 ssh_username:root, ip:172.27.33.6 port:8091 ssh_username:root, ip:172.27.33.5 port:8091 ssh_username:root] 2013-01-04 16:55:40,402 - root - INFO - deleted bucket : default from 172.27.33.2 2013-01-04 16:55:40,402 - root - INFO - waiting for bucket deletion to complete.... 2013-01-04 16:55:40,407 - root - INFO - existing buckets : [] 2013-01-04 16:55:40,480 - root - INFO - rebalancing all nodes in order to remove nodes 2013-01-04 16:55:40,485 - root - INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5&user=Administrator&knownNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5%2Cns_1%40172.27.33.2 2013-01-04 16:55:40,491 - root - INFO - rebalance operation started 2013-01-04 16:55:40,499 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:55:44,511 - root - INFO - rebalance progress took 4.02038812637 seconds 2013-01-04 16:55:44,512 - root - INFO - sleep for 4.02038812637 seconds after rebalance... 2013-01-04 16:55:48,574 - root - INFO - removed all the nodes from cluster associated with ip:172.27.33.2 port:8091 ssh_username:root ? [(u'ns_1@172.27.33.6', 8091), (u'ns_1@172.27.33.3', 8091), (u'ns_1@172.27.33.5', 8091)] 2013-01-04 16:55:58,593 - root - INFO - waiting for ns_server @ 172.27.33.2:8091 2013-01-04 16:55:58,602 - root - INFO - ns_server @ 172.27.33.2:8091 is running 2013-01-04 16:55:58,609 - root - INFO - waiting for ns_server @ 172.27.33.3:8091 2013-01-04 16:55:58,616 - root - INFO - ns_server @ 172.27.33.3:8091 is running 2013-01-04 16:55:58,623 - root - INFO - waiting for ns_server @ 172.27.33.6:8091 2013-01-04 16:55:58,629 - root - INFO - ns_server @ 172.27.33.6:8091 is running 2013-01-04 16:55:58,637 - root - INFO - waiting for ns_server @ 172.27.33.5:8091 2013-01-04 16:55:58,644 - root - INFO - ns_server @ 172.27.33.5:8091 is running 2013-01-04 16:55:58,645 - root - INFO - ============== basetestcase cleanup was finished for test #1 online_upgrade_rebalance_in_out ============== ====================================================================== ERROR: online_upgrade_rebalance_in_out (newupgradetests.MultiNodesUpgradeTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/newupgradetests.py", line 195, in online_upgrade_rebalance_in_out self.online_upgrade() File "pytests/newupgradetests.py", line 240, in online_upgrade raise Exception("After rebalance in 2.0 Nodes, 2.0 doesn't become the master") Exception: After rebalance in 2.0 Nodes, 2.0 doesn't become the master ---------------------------------------------------------------------- Ran 1 test in 1788.471s FAILED (errors=1) summary so far suite newupgradetests.MultiNodesUpgradeTests , pass 0 , fail 1 failures so far... newupgradetests.MultiNodesUpgradeTests.online_upgrade_rebalance_in_out testrunner logs, diags and results are available under logs/testrunner-13-Jan-04_16-26-102013-01-04 16:26:20,869 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:26:20,979 - root - INFO - command executed successfully 2013-01-04 16:26:20,981 - root - INFO - running command.raw hostname 2013-01-04 16:26:20,985 - root - INFO - command executed successfully 2013-01-04 16:26:20,986 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:26:21,096 - root - INFO - command executed successfully 2013-01-04 16:26:21,102 - root - INFO - command executed successfully 2013-01-04 16:26:21,103 - root - INFO - running command.raw hostname 2013-01-04 16:26:21,194 - root - INFO - command executed successfully 2013-01-04 16:27:02,379 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:27:02,704 - root - INFO - Connected 2013-01-04 16:27:03,633 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:27:03,796 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:03,995 - root - INFO - Connected 2013-01-04 16:27:04,026 - root - INFO - command executed successfully 2013-01-04 16:27:04,028 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,138 - root - INFO - command executed successfully 2013-01-04 16:27:04,139 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,233 - root - INFO - command executed successfully 2013-01-04 16:27:04,235 - root - INFO - running command.raw hostname 2013-01-04 16:27:04,307 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:04,354 - root - INFO - command executed successfully 2013-01-04 16:27:04,418 - root - INFO - command executed successfully 2013-01-04 16:27:04,420 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,548 - root - INFO - command executed successfully 2013-01-04 16:27:04,549 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,635 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:04,660 - root - INFO - command executed successfully 2013-01-04 16:27:04,662 - root - INFO - running command.raw hostname 2013-01-04 16:27:04,720 - root - INFO - command executed successfully 2013-01-04 16:27:04,721 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,772 - root - INFO - command executed successfully 2013-01-04 16:27:04,805 - root - INFO - command executed successfully 2013-01-04 16:27:04,806 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,919 - root - INFO - command executed successfully 2013-01-04 16:27:04,921 - root - INFO - running command.raw hostname 2013-01-04 16:27:05,036 - root - INFO - command executed successfully 2013-01-04 16:27:05,040 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:27:05,059 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:05,145 - root - INFO - command executed successfully 2013-01-04 16:27:05,147 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:05,160 - root - INFO - command executed successfully 2013-01-04 16:27:05,162 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm': No such file or directory 2013-01-04 16:27:05,162 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:27:05,163 - root - ERROR - rm: cannot remove `cb_1275': Is a directory 2013-01-04 16:27:05,164 - root - ERROR - mv: cannot stat `cb_1275/*': No such file or directory 2013-01-04 16:27:05,165 - root - INFO - get md5 sum for local and remote 2013-01-04 16:27:05,166 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm > couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l 2013-01-04 16:27:05,260 - root - INFO - command executed successfully 2013-01-04 16:27:05,262 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:05,372 - root - INFO - command executed successfully 2013-01-04 16:27:05,373 - root - INFO - running command.raw hostname 2013-01-04 16:27:05,483 - root - INFO - command executed successfully 2013-01-04 16:27:05,487 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:27:05,604 - root - INFO - command executed successfully 2013-01-04 16:27:05,606 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm': No such file or directory 2013-01-04 16:27:05,607 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:27:05,608 - root - ERROR - rm: cannot remove `cb_0006': Is a directory 2013-01-04 16:27:05,609 - root - ERROR - rm: cannot remove `default': Is a directory 2013-01-04 16:27:05,609 - root - ERROR - mv: cannot stat `cb_0006/*': No such file or directory 2013-01-04 16:27:05,610 - root - INFO - get md5 sum for local and remote 2013-01-04 16:27:05,610 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm > couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l 2013-01-04 16:27:05,773 - root - INFO - command executed successfully 2013-01-04 16:27:05,774 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm: No such file or directory 2013-01-04 16:27:05,775 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:27:05,777 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:27:06,209 - root - INFO - command executed successfully 2013-01-04 16:27:06,210 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm: No such file or directory 2013-01-04 16:27:06,211 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:27:06,212 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:30:09,539 - root - INFO - command executed successfully 2013-01-04 16:30:09,540 - root - INFO - 1d0 2013-01-04 16:30:09,541 - root - INFO - < 4fceddd71c9a4468940d6b070fcda56a couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:09,822 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:09,933 - root - INFO - command executed successfully 2013-01-04 16:30:09,934 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:10,043 - root - INFO - command executed successfully 2013-01-04 16:30:10,046 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:10,156 - root - INFO - command executed successfully 2013-01-04 16:30:10,158 - root - INFO - running command.raw hostname 2013-01-04 16:30:10,267 - root - INFO - command executed successfully 2013-01-04 16:30:10,271 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:30:10,382 - root - INFO - command executed successfully 2013-01-04 16:30:10,654 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:10,738 - root - INFO - command executed successfully 2013-01-04 16:30:10,740 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:10,851 - root - INFO - command executed successfully 2013-01-04 16:30:10,853 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:10,962 - root - INFO - command executed successfully 2013-01-04 16:30:10,964 - root - INFO - running command.raw hostname 2013-01-04 16:30:11,074 - root - INFO - command executed successfully 2013-01-04 16:30:11,078 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:30:11,185 - root - INFO - command executed successfully 2013-01-04 16:30:11,515 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:11,624 - root - INFO - command executed successfully 2013-01-04 16:30:11,626 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:11,736 - root - INFO - command executed successfully 2013-01-04 16:30:11,738 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:11,849 - root - INFO - command executed successfully 2013-01-04 16:30:11,851 - root - INFO - running command.raw hostname 2013-01-04 16:30:11,962 - root - INFO - command executed successfully 2013-01-04 16:30:11,966 - root - INFO - deliverable_type : rpm 2013-01-04 16:30:11,966 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:30:11,966 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:22,831 - root - INFO - command executed successfully 2013-01-04 16:30:22,832 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:30:22,833 - root - INFO - 2013-01-04 16:30:22,834 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:30:22,835 - root - INFO - Please browse to http://kiwi-r101:8091/ to configure your server. 2013-01-04 16:30:22,836 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:30:22,836 - root - INFO - 2013-01-04 16:30:22,837 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:30:22,837 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:30:22,838 - root - INFO - 8091 and from 21100 to 21299. 2013-01-04 16:30:22,838 - root - INFO - 2013-01-04 16:30:22,838 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:30:22,839 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:30:22,839 - root - INFO - 2013-01-04 16:30:22,839 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:30:22,923 - root - INFO - command executed successfully 2013-01-04 16:30:22,925 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:30:23,035 - root - INFO - command executed successfully 2013-01-04 16:30:23,037 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:30:24,392 - root - INFO - command executed successfully 2013-01-04 16:30:24,393 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:30:24,394 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.2 finished 2013-01-04 16:30:31,632 - root - INFO - command executed successfully 2013-01-04 16:30:31,633 - root - INFO - 1d0 2013-01-04 16:30:31,635 - root - INFO - < 4fceddd71c9a4468940d6b070fcda56a couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:31,912 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:32,022 - root - INFO - command executed successfully 2013-01-04 16:30:32,024 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:32,134 - root - INFO - command executed successfully 2013-01-04 16:30:32,135 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:32,246 - root - INFO - command executed successfully 2013-01-04 16:30:32,248 - root - INFO - running command.raw hostname 2013-01-04 16:30:32,358 - root - INFO - command executed successfully 2013-01-04 16:30:32,361 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:30:32,471 - root - INFO - command executed successfully 2013-01-04 16:30:32,747 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:32,858 - root - INFO - command executed successfully 2013-01-04 16:30:32,859 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:32,969 - root - INFO - command executed successfully 2013-01-04 16:30:32,971 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:33,056 - root - INFO - command executed successfully 2013-01-04 16:30:33,058 - root - INFO - running command.raw hostname 2013-01-04 16:30:33,168 - root - INFO - command executed successfully 2013-01-04 16:30:33,172 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:30:33,279 - root - INFO - command executed successfully 2013-01-04 16:30:33,667 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:33,777 - root - INFO - command executed successfully 2013-01-04 16:30:33,779 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:33,889 - root - INFO - command executed successfully 2013-01-04 16:30:33,892 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:34,002 - root - INFO - command executed successfully 2013-01-04 16:30:34,003 - root - INFO - running command.raw hostname 2013-01-04 16:30:34,113 - root - INFO - command executed successfully 2013-01-04 16:30:34,117 - root - INFO - deliverable_type : rpm 2013-01-04 16:30:34,117 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:30:34,118 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:46,417 - root - INFO - command executed successfully 2013-01-04 16:30:46,419 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:30:46,420 - root - INFO - 2013-01-04 16:30:46,422 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:30:46,422 - root - INFO - Please browse to http://kiwi-r103:8091/ to configure your server. 2013-01-04 16:30:46,423 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:30:46,423 - root - INFO - 2013-01-04 16:30:46,423 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:30:46,424 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:30:46,424 - root - INFO - 8091 and from 21100 to 21299. 2013-01-04 16:30:46,424 - root - INFO - 2013-01-04 16:30:46,425 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:30:46,425 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:30:46,426 - root - INFO - 2013-01-04 16:30:46,426 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:30:46,542 - root - INFO - command executed successfully 2013-01-04 16:30:46,544 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:30:46,653 - root - INFO - command executed successfully 2013-01-04 16:30:46,655 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:30:47,985 - root - INFO - command executed successfully 2013-01-04 16:30:47,986 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:30:47,988 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.4 finished 2013-01-04 16:30:52,999 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:30:53,000 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:30:53,551 - root - INFO - Connected lib/membase/api/rest_client.py:186: DeprecationWarning: object.__new__() takes no parameters obj = object.__new__(self, serverInfo) 2013-01-04 16:30:53,577 - root - INFO - Connected 2013-01-04 16:30:56,803 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:56,825 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:56,889 - root - INFO - command executed successfully 2013-01-04 16:30:56,890 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:56,917 - root - INFO - command executed successfully 2013-01-04 16:30:56,919 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:57,001 - root - INFO - command executed successfully 2013-01-04 16:30:57,002 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:57,029 - root - INFO - command executed successfully 2013-01-04 16:30:57,031 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:57,114 - root - INFO - command executed successfully 2013-01-04 16:30:57,115 - root - INFO - running command.raw hostname 2013-01-04 16:30:57,140 - root - INFO - command executed successfully 2013-01-04 16:30:57,141 - root - INFO - running command.raw hostname 2013-01-04 16:30:57,225 - root - INFO - command executed successfully 2013-01-04 16:30:57,228 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:30:57,252 - root - INFO - command executed successfully 2013-01-04 16:30:57,256 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:30:57,354 - root - INFO - command executed successfully 2013-01-04 16:30:57,355 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:30:57,387 - root - INFO - command executed successfully 2013-01-04 16:30:57,387 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:30:57,469 - root - INFO - command executed successfully 2013-01-04 16:30:57,471 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:30:57,498 - root - INFO - command executed successfully 2013-01-04 16:30:57,500 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:30:59,348 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:30:59,755 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:31:02,351 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:02,365 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.2 finished 2013-01-04 16:31:02,757 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:02,770 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.4 finished 2013-01-04 16:31:02,813 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:31:03,134 - root - INFO - Connected 2013-01-04 16:31:03,453 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:03,563 - root - INFO - command executed successfully 2013-01-04 16:31:03,565 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:03,675 - root - INFO - command executed successfully 2013-01-04 16:31:03,677 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:03,787 - root - INFO - command executed successfully 2013-01-04 16:31:03,789 - root - INFO - running command.raw hostname 2013-01-04 16:31:03,899 - root - INFO - command executed successfully 2013-01-04 16:31:04,021 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:31:04,344 - root - INFO - Connected 2013-01-04 16:31:04,655 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:04,766 - root - INFO - command executed successfully 2013-01-04 16:31:04,767 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:04,853 - root - INFO - command executed successfully 2013-01-04 16:31:04,854 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:04,965 - root - INFO - command executed successfully 2013-01-04 16:31:04,966 - root - INFO - running command.raw hostname 2013-01-04 16:31:05,076 - root - INFO - command executed successfully 2013-01-04 16:31:05,206 - root - INFO - Installation of old version is done. Wait for 10 sec for upgrade 2013-01-04 16:31:05,382 - root - INFO - adding node 172.27.33.4:8091 to cluster 2013-01-04 16:31:05,382 - root - INFO - adding remote node @172.27.33.4:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:31:05,913 - root - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.27.33.2%2Cns_1%40172.27.33.4 2013-01-04 16:31:05,918 - root - INFO - rebalance operation started 2013-01-04 16:31:05,927 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:31:15,943 - root - INFO - rebalancing was completed with progress: 100% in 10.0244619846 sec 2013-01-04 16:31:16,953 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:16,964 - root - INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password 2013-01-04 16:31:16,988 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:16,999 - root - INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password 2013-01-04 16:31:18,012 - root - INFO - http://172.27.33.2:8091/pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=None&flushEnabled=1&replicaNumber=1&ramQuotaMB=1456 2013-01-04 16:31:18,026 - root - INFO - 0.0136511325836 seconds to create bucket default 2013-01-04 16:31:18,026 - root - INFO - waiting for memcached bucket : default in 172.27.33.2 to accept set ops 2013-01-04 16:31:19,215 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,309 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:19,439 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,525 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:19,666 - root - INFO - bucket 'default' was created with per node RAM quota: 1456 2013-01-04 16:31:19,734 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,817 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:20,669 - root - INFO - Batch create documents done #: 0 2013-01-04 16:31:21,093 - root - INFO - Batch create documents done #: 1000 2013-01-04 16:31:21,687 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:21,759 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.27.33.2:8091' 2013-01-04 16:31:21,773 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:21,852 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '172.27.33.2:8091' 2013-01-04 16:31:21,867 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:21,944 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.27.33.4:8091' 2013-01-04 16:31:21,961 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:22,038 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '172.27.33.4:8091' 2013-01-04 16:31:32,051 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,052 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,053 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,813 - root - INFO - Connected 2013-01-04 16:31:32,815 - root - INFO - Connected 2013-01-04 16:31:32,815 - root - INFO - Connected 2013-01-04 16:31:33,168 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,172 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,196 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,255 - root - INFO - command executed successfully 2013-01-04 16:31:33,257 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,261 - root - INFO - command executed successfully 2013-01-04 16:31:33,262 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,312 - root - INFO - command executed successfully 2013-01-04 16:31:33,314 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,367 - root - INFO - command executed successfully 2013-01-04 16:31:33,369 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,375 - root - INFO - command executed successfully 2013-01-04 16:31:33,376 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,425 - root - INFO - command executed successfully 2013-01-04 16:31:33,426 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,466 - root - INFO - command executed successfully 2013-01-04 16:31:33,467 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,481 - root - INFO - command executed successfully 2013-01-04 16:31:33,482 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,514 - root - INFO - command executed successfully 2013-01-04 16:31:33,515 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,583 - root - INFO - command executed successfully 2013-01-04 16:31:33,588 - root - INFO - CentOS 2013-01-04 16:31:33,589 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,591 - root - INFO - command executed successfully 2013-01-04 16:31:33,595 - root - INFO - CentOS 2013-01-04 16:31:33,596 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,627 - root - INFO - command executed successfully 2013-01-04 16:31:33,630 - root - INFO - CentOS 2013-01-04 16:31:33,631 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,991 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,996 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:34,008 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:34,080 - root - INFO - command executed successfully 2013-01-04 16:31:34,081 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,084 - root - INFO - command executed successfully 2013-01-04 16:31:34,085 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,123 - root - INFO - command executed successfully 2013-01-04 16:31:34,124 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,172 - root - INFO - command executed successfully 2013-01-04 16:31:34,173 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,175 - root - INFO - command executed successfully 2013-01-04 16:31:34,176 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,237 - root - INFO - command executed successfully 2013-01-04 16:31:34,239 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,265 - root - INFO - command executed successfully 2013-01-04 16:31:34,266 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,268 - root - INFO - command executed successfully 2013-01-04 16:31:34,269 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,351 - root - INFO - command executed successfully 2013-01-04 16:31:34,357 - root - INFO - command executed successfully 2013-01-04 16:31:34,358 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,359 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,370 - root - INFO - command executed successfully 2013-01-04 16:31:34,373 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,459 - root - INFO - command executed successfully 2013-01-04 16:31:34,461 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,461 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,465 - root - INFO - command executed successfully 2013-01-04 16:31:34,469 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,502 - root - INFO - command executed successfully 2013-01-04 16:31:34,504 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,505 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,574 - root - INFO - command executed successfully 2013-01-04 16:31:34,576 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,577 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,583 - root - INFO - command executed successfully 2013-01-04 16:31:34,584 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,585 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,617 - root - INFO - command executed successfully 2013-01-04 16:31:34,618 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,619 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,689 - root - INFO - command executed successfully 2013-01-04 16:31:34,691 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,699 - root - INFO - command executed successfully 2013-01-04 16:31:34,700 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,701 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,707 - root - INFO - command executed successfully 2013-01-04 16:31:34,708 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,800 - root - INFO - command executed successfully 2013-01-04 16:31:34,801 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,813 - root - INFO - command executed successfully 2013-01-04 16:31:34,814 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,820 - root - INFO - command executed successfully 2013-01-04 16:31:34,821 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,911 - root - INFO - command executed successfully 2013-01-04 16:31:34,912 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:34,914 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:34,924 - root - INFO - command executed successfully 2013-01-04 16:31:34,925 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,936 - root - INFO - command executed successfully 2013-01-04 16:31:34,937 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:34,938 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:35,015 - root - INFO - command executed successfully 2013-01-04 16:31:35,016 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:35,018 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:35,025 - root - INFO - command executed successfully 2013-01-04 16:31:35,026 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,027 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,053 - root - INFO - command executed successfully 2013-01-04 16:31:35,054 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,055 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,130 - root - INFO - command executed successfully 2013-01-04 16:31:35,131 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,132 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,140 - root - INFO - command executed successfully 2013-01-04 16:31:35,142 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,169 - root - INFO - command executed successfully 2013-01-04 16:31:35,170 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,219 - root - INFO - command executed successfully 2013-01-04 16:31:35,221 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,251 - root - INFO - command executed successfully 2013-01-04 16:31:35,252 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,281 - root - INFO - command executed successfully 2013-01-04 16:31:35,282 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,337 - root - INFO - command executed successfully 2013-01-04 16:31:35,338 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,362 - root - INFO - command executed successfully 2013-01-04 16:31:35,364 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,394 - root - INFO - command executed successfully 2013-01-04 16:31:35,395 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,454 - root - INFO - command executed successfully 2013-01-04 16:31:35,456 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,475 - root - INFO - command executed successfully 2013-01-04 16:31:35,476 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,511 - root - INFO - command executed successfully 2013-01-04 16:31:35,512 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,570 - root - INFO - command executed successfully 2013-01-04 16:31:35,571 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,587 - root - INFO - command executed successfully 2013-01-04 16:31:35,588 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,602 - root - INFO - command executed successfully 2013-01-04 16:31:35,603 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,686 - root - INFO - command executed successfully 2013-01-04 16:31:35,688 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,698 - root - INFO - command executed successfully 2013-01-04 16:31:35,718 - root - INFO - command executed successfully 2013-01-04 16:31:35,815 - root - INFO - command executed successfully 2013-01-04 16:31:36,006 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,074 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,089 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,095 - root - INFO - command executed successfully 2013-01-04 16:31:36,097 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,180 - root - INFO - command executed successfully 2013-01-04 16:31:36,181 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,189 - root - INFO - command executed successfully 2013-01-04 16:31:36,189 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,207 - root - INFO - command executed successfully 2013-01-04 16:31:36,208 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,271 - root - INFO - command executed successfully 2013-01-04 16:31:36,272 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,281 - root - INFO - command executed successfully 2013-01-04 16:31:36,283 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,318 - root - INFO - command executed successfully 2013-01-04 16:31:36,319 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,385 - root - INFO - command executed successfully 2013-01-04 16:31:36,388 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,395 - root - INFO - command executed successfully 2013-01-04 16:31:36,396 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,430 - root - INFO - command executed successfully 2013-01-04 16:31:36,434 - root - INFO - CentOS 2013-01-04 16:31:36,435 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,435 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:36,504 - root - INFO - command executed successfully 2013-01-04 16:31:36,509 - root - INFO - CentOS 2013-01-04 16:31:36,509 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,509 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:36,512 - root - INFO - command executed successfully 2013-01-04 16:31:36,515 - root - INFO - CentOS 2013-01-04 16:31:36,516 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,517 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:39,326 - root - INFO - command executed successfully 2013-01-04 16:31:39,327 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,328 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,329 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,330 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,331 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,332 - root - INFO - 2013-01-04 16:31:39,332 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,341 - root - INFO - command executed successfully 2013-01-04 16:31:39,342 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,342 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,343 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,343 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,344 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,345 - root - INFO - 2013-01-04 16:31:39,346 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,381 - root - INFO - command executed successfully 2013-01-04 16:31:39,382 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,382 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,384 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,386 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,386 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,387 - root - INFO - 2013-01-04 16:31:39,387 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,447 - root - INFO - command executed successfully 2013-01-04 16:31:39,449 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,458 - root - INFO - command executed successfully 2013-01-04 16:31:39,460 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,502 - root - INFO - command executed successfully 2013-01-04 16:31:39,504 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,559 - root - INFO - command executed successfully 2013-01-04 16:31:39,561 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,571 - root - INFO - command executed successfully 2013-01-04 16:31:39,573 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,613 - root - INFO - command executed successfully 2013-01-04 16:31:39,614 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,674 - root - INFO - command executed successfully 2013-01-04 16:31:39,676 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,677 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,683 - root - INFO - command executed successfully 2013-01-04 16:31:39,685 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,686 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,727 - root - INFO - command executed successfully 2013-01-04 16:31:39,729 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,730 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,787 - root - INFO - command executed successfully 2013-01-04 16:31:39,788 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,790 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,796 - root - INFO - command executed successfully 2013-01-04 16:31:39,798 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,799 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,842 - root - INFO - command executed successfully 2013-01-04 16:31:39,843 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,844 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,905 - root - INFO - command executed successfully 2013-01-04 16:31:39,907 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,908 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:39,909 - root - INFO - command executed successfully 2013-01-04 16:31:39,912 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,912 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:39,962 - root - INFO - command executed successfully 2013-01-04 16:31:39,963 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,964 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:40,024 - root - INFO - command executed successfully 2013-01-04 16:31:40,026 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,029 - root - INFO - command executed successfully 2013-01-04 16:31:40,031 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,082 - root - INFO - command executed successfully 2013-01-04 16:31:40,083 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,137 - root - INFO - command executed successfully 2013-01-04 16:31:40,138 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,140 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,143 - root - INFO - command executed successfully 2013-01-04 16:31:40,144 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,145 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,195 - root - INFO - command executed successfully 2013-01-04 16:31:40,196 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,196 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,253 - root - INFO - command executed successfully 2013-01-04 16:31:40,254 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,255 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,260 - root - INFO - command executed successfully 2013-01-04 16:31:40,261 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,263 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,307 - root - INFO - command executed successfully 2013-01-04 16:31:40,308 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,309 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,367 - root - INFO - command executed successfully 2013-01-04 16:31:40,368 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,377 - root - INFO - command executed successfully 2013-01-04 16:31:40,379 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,424 - root - INFO - command executed successfully 2013-01-04 16:31:40,425 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,481 - root - INFO - command executed successfully 2013-01-04 16:31:40,483 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,496 - root - INFO - command executed successfully 2013-01-04 16:31:40,497 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,543 - root - INFO - command executed successfully 2013-01-04 16:31:40,544 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,580 - root - INFO - command executed successfully 2013-01-04 16:31:40,581 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,593 - root - INFO - command executed successfully 2013-01-04 16:31:40,594 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,659 - root - INFO - command executed successfully 2013-01-04 16:31:40,660 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,697 - root - INFO - command executed successfully 2013-01-04 16:31:40,699 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,707 - root - INFO - command executed successfully 2013-01-04 16:31:40,708 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,777 - root - INFO - command executed successfully 2013-01-04 16:31:40,778 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,815 - root - INFO - command executed successfully 2013-01-04 16:31:40,817 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,821 - root - INFO - command executed successfully 2013-01-04 16:31:40,823 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,892 - root - INFO - command executed successfully 2013-01-04 16:31:40,894 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,934 - root - INFO - command executed successfully 2013-01-04 16:31:40,937 - root - INFO - command executed successfully 2013-01-04 16:31:41,008 - root - INFO - command executed successfully 2013-01-04 16:31:41,317 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,321 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,345 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,402 - root - INFO - command executed successfully 2013-01-04 16:31:41,404 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,437 - root - INFO - command executed successfully 2013-01-04 16:31:41,439 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,459 - root - INFO - command executed successfully 2013-01-04 16:31:41,461 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,515 - root - INFO - command executed successfully 2013-01-04 16:31:41,517 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,553 - root - INFO - command executed successfully 2013-01-04 16:31:41,555 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,575 - root - INFO - command executed successfully 2013-01-04 16:31:41,576 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,628 - root - INFO - command executed successfully 2013-01-04 16:31:41,629 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,669 - root - INFO - command executed successfully 2013-01-04 16:31:41,672 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,693 - root - INFO - command executed successfully 2013-01-04 16:31:41,694 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,741 - root - INFO - command executed successfully 2013-01-04 16:31:41,745 - root - INFO - CentOS 2013-01-04 16:31:41,746 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,746 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,786 - root - INFO - command executed successfully 2013-01-04 16:31:41,790 - root - INFO - CentOS 2013-01-04 16:31:41,790 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,791 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,809 - root - INFO - command executed successfully 2013-01-04 16:31:41,811 - root - INFO - CentOS 2013-01-04 16:31:41,812 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,813 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,869 - root - INFO - command executed successfully 2013-01-04 16:31:41,871 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,872 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,909 - root - INFO - command executed successfully 2013-01-04 16:31:41,911 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,912 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,929 - root - INFO - command executed successfully 2013-01-04 16:31:41,931 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,932 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,980 - root - INFO - command executed successfully 2013-01-04 16:31:41,982 - root - ERROR - beam: no process killed 2013-01-04 16:31:41,983 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,027 - root - INFO - command executed successfully 2013-01-04 16:31:42,029 - root - ERROR - beam: no process killed 2013-01-04 16:31:42,030 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,046 - root - INFO - command executed successfully 2013-01-04 16:31:42,047 - root - ERROR - beam: no process killed 2013-01-04 16:31:42,049 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,094 - root - INFO - command executed successfully 2013-01-04 16:31:42,096 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,097 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,147 - root - INFO - command executed successfully 2013-01-04 16:31:42,149 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,149 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,159 - root - INFO - command executed successfully 2013-01-04 16:31:42,160 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,161 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,208 - root - INFO - command executed successfully 2013-01-04 16:31:42,263 - root - INFO - command executed successfully 2013-01-04 16:31:42,276 - root - INFO - command executed successfully thread uninstaller-thread-172.27.33.3 finished thread uninstaller-thread-172.27.33.6 finished thread uninstaller-thread-172.27.33.5 finished 2013-01-04 16:31:42,278 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,279 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,279 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,852 - root - INFO - Connected 2013-01-04 16:31:42,853 - root - INFO - Connected 2013-01-04 16:31:43,045 - root - INFO - Connected 2013-01-04 16:31:43,351 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,371 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,418 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,453 - root - INFO - command executed successfully 2013-01-04 16:31:43,455 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,462 - root - INFO - command executed successfully 2013-01-04 16:31:43,463 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,505 - root - INFO - command executed successfully 2013-01-04 16:31:43,507 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,568 - root - INFO - command executed successfully 2013-01-04 16:31:43,569 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,580 - root - INFO - command executed successfully 2013-01-04 16:31:43,581 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,622 - root - INFO - command executed successfully 2013-01-04 16:31:43,623 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,680 - root - INFO - command executed successfully 2013-01-04 16:31:43,681 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,699 - root - INFO - command executed successfully 2013-01-04 16:31:43,700 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,738 - root - INFO - command executed successfully 2013-01-04 16:31:43,739 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,787 - root - INFO - command executed successfully 2013-01-04 16:31:43,793 - root - INFO - command executed successfully 2013-01-04 16:31:43,828 - root - INFO - command executed successfully 2013-01-04 16:32:46,827 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:32:47,158 - root - INFO - Connected 2013-01-04 16:32:47,469 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:48,298 - root - INFO - command executed successfully 2013-01-04 16:32:48,325 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:48,804 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:32:48,819 - root - INFO - command executed successfully 2013-01-04 16:32:48,845 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:49,056 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:32:49,172 - root - INFO - command executed successfully 2013-01-04 16:32:49,173 - root - INFO - running command.raw hostname 2013-01-04 16:32:49,380 - root - INFO - Connected 2013-01-04 16:32:49,567 - root - INFO - command executed successfully 2013-01-04 16:32:49,581 - root - INFO - Connected 2013-01-04 16:32:49,923 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:49,930 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:49,937 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,013 - root - INFO - command executed successfully 2013-01-04 16:32:50,014 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,039 - root - INFO - command executed successfully 2013-01-04 16:32:50,040 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,053 - root - INFO - command executed successfully 2013-01-04 16:32:50,054 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,125 - root - INFO - command executed successfully 2013-01-04 16:32:50,126 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,153 - root - INFO - command executed successfully 2013-01-04 16:32:50,155 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,166 - root - INFO - command executed successfully 2013-01-04 16:32:50,167 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,213 - root - INFO - command executed successfully 2013-01-04 16:32:50,215 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,270 - root - INFO - command executed successfully 2013-01-04 16:32:50,271 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,283 - root - INFO - command executed successfully 2013-01-04 16:32:50,285 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,325 - root - INFO - command executed successfully 2013-01-04 16:32:50,330 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:50,384 - root - INFO - command executed successfully 2013-01-04 16:32:50,402 - root - INFO - command executed successfully 2013-01-04 16:32:50,450 - root - INFO - command executed successfully 2013-01-04 16:32:50,452 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:50,453 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:50,454 - root - ERROR - rm: cannot remove `cb_8077': Is a directory 2013-01-04 16:32:50,455 - root - ERROR - mv: cannot stat `cb_8077/*': No such file or directory 2013-01-04 16:32:50,456 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:50,457 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:50,702 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,716 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,804 - root - INFO - command executed successfully 2013-01-04 16:32:50,805 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,814 - root - INFO - command executed successfully 2013-01-04 16:32:50,816 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,920 - root - INFO - command executed successfully 2013-01-04 16:32:50,922 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,928 - root - INFO - command executed successfully 2013-01-04 16:32:50,930 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:51,038 - root - INFO - command executed successfully 2013-01-04 16:32:51,040 - root - INFO - running command.raw hostname 2013-01-04 16:32:51,043 - root - INFO - command executed successfully 2013-01-04 16:32:51,045 - root - INFO - running command.raw hostname 2013-01-04 16:32:51,059 - root - INFO - command executed successfully 2013-01-04 16:32:51,060 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,062 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,063 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:32:51,157 - root - INFO - command executed successfully 2013-01-04 16:32:51,161 - root - INFO - command executed successfully 2013-01-04 16:32:51,165 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:51,168 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:51,254 - root - INFO - command executed successfully 2013-01-04 16:32:51,256 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:51,257 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:51,258 - root - ERROR - rm: cannot remove `cb_1951': Is a directory 2013-01-04 16:32:51,258 - root - ERROR - mv: cannot stat `cb_1951/*': No such file or directory 2013-01-04 16:32:51,259 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:51,259 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:51,287 - root - INFO - command executed successfully 2013-01-04 16:32:51,289 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:51,290 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:51,291 - root - ERROR - rm: cannot remove `cb_1410': Is a directory 2013-01-04 16:32:51,291 - root - ERROR - mv: cannot stat `cb_1410/*': No such file or directory 2013-01-04 16:32:51,292 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:51,292 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:51,842 - root - INFO - command executed successfully 2013-01-04 16:32:51,843 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,845 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,846 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:32:51,869 - root - INFO - command executed successfully 2013-01-04 16:32:51,870 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,872 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,872 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:36:32,684 - root - INFO - command executed successfully 2013-01-04 16:36:32,686 - root - INFO - 1d0 2013-01-04 16:36:32,687 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:32,951 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:33,068 - root - INFO - command executed successfully 2013-01-04 16:36:33,069 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:33,184 - root - INFO - command executed successfully 2013-01-04 16:36:33,186 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:33,300 - root - INFO - command executed successfully 2013-01-04 16:36:33,302 - root - INFO - running command.raw hostname 2013-01-04 16:36:33,417 - root - INFO - command executed successfully 2013-01-04 16:36:33,421 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:33,528 - root - INFO - command executed successfully 2013-01-04 16:36:33,779 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:33,893 - root - INFO - command executed successfully 2013-01-04 16:36:33,894 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:34,008 - root - INFO - command executed successfully 2013-01-04 16:36:34,010 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:34,125 - root - INFO - command executed successfully 2013-01-04 16:36:34,126 - root - INFO - running command.raw hostname 2013-01-04 16:36:34,236 - root - INFO - command executed successfully 2013-01-04 16:36:34,240 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:34,347 - root - INFO - command executed successfully 2013-01-04 16:36:34,645 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:34,736 - root - INFO - command executed successfully 2013-01-04 16:36:34,737 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:34,853 - root - INFO - command executed successfully 2013-01-04 16:36:34,854 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:34,967 - root - INFO - command executed successfully 2013-01-04 16:36:34,969 - root - INFO - running command.raw hostname 2013-01-04 16:36:35,082 - root - INFO - command executed successfully 2013-01-04 16:36:35,086 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:35,087 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:35,088 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:44,640 - root - INFO - command executed successfully 2013-01-04 16:36:44,641 - root - INFO - 1d0 2013-01-04 16:36:44,642 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:44,920 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:45,034 - root - INFO - command executed successfully 2013-01-04 16:36:45,035 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:45,152 - root - INFO - command executed successfully 2013-01-04 16:36:45,155 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:45,273 - root - INFO - command executed successfully 2013-01-04 16:36:45,274 - root - INFO - running command.raw hostname 2013-01-04 16:36:45,387 - root - INFO - command executed successfully 2013-01-04 16:36:45,391 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:45,499 - root - INFO - command executed successfully 2013-01-04 16:36:45,748 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:45,864 - root - INFO - command executed successfully 2013-01-04 16:36:45,865 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:45,980 - root - INFO - command executed successfully 2013-01-04 16:36:45,981 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:45,985 - root - INFO - command executed successfully 2013-01-04 16:36:45,986 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:36:45,988 - root - INFO - 2013-01-04 16:36:45,989 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:36:45,989 - root - INFO - Please browse to http://kiwi-r102:8091/ to configure your server. 2013-01-04 16:36:45,991 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:36:45,992 - root - INFO - 2013-01-04 16:36:45,992 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:36:45,994 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:36:45,995 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:36:45,995 - root - INFO - 2013-01-04 16:36:45,996 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:36:45,996 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:36:45,996 - root - INFO - 2013-01-04 16:36:45,997 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:36:46,074 - root - INFO - command executed successfully 2013-01-04 16:36:46,075 - root - INFO - running command.raw hostname 2013-01-04 16:36:46,080 - root - INFO - command executed successfully 2013-01-04 16:36:46,081 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:36:46,187 - root - INFO - command executed successfully 2013-01-04 16:36:46,190 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:46,197 - root - INFO - command executed successfully 2013-01-04 16:36:46,197 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:36:46,305 - root - INFO - command executed successfully 2013-01-04 16:36:46,616 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:46,704 - root - INFO - command executed successfully 2013-01-04 16:36:46,705 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:46,815 - root - INFO - command executed successfully 2013-01-04 16:36:46,817 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:46,930 - root - INFO - command executed successfully 2013-01-04 16:36:46,931 - root - INFO - running command.raw hostname 2013-01-04 16:36:47,046 - root - INFO - command executed successfully 2013-01-04 16:36:47,050 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:47,050 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:47,052 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:47,500 - root - INFO - command executed successfully 2013-01-04 16:36:47,501 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:36:47,503 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.3 finished 2013-01-04 16:36:56,036 - root - INFO - command executed successfully 2013-01-04 16:36:56,037 - root - INFO - 1d0 2013-01-04 16:36:56,039 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:56,319 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:56,432 - root - INFO - command executed successfully 2013-01-04 16:36:56,433 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:56,544 - root - INFO - command executed successfully 2013-01-04 16:36:56,545 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:56,632 - root - INFO - command executed successfully 2013-01-04 16:36:56,633 - root - INFO - running command.raw hostname 2013-01-04 16:36:56,719 - root - INFO - command executed successfully 2013-01-04 16:36:56,723 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:56,830 - root - INFO - command executed successfully 2013-01-04 16:36:57,103 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:57,188 - root - INFO - command executed successfully 2013-01-04 16:36:57,189 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:57,299 - root - INFO - command executed successfully 2013-01-04 16:36:57,301 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:57,412 - root - INFO - command executed successfully 2013-01-04 16:36:57,414 - root - INFO - running command.raw hostname 2013-01-04 16:36:57,524 - root - INFO - command executed successfully 2013-01-04 16:36:57,528 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:57,636 - root - INFO - command executed successfully 2013-01-04 16:36:57,963 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:58,048 - root - INFO - command executed successfully 2013-01-04 16:36:58,049 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:58,160 - root - INFO - command executed successfully 2013-01-04 16:36:58,161 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:58,271 - root - INFO - command executed successfully 2013-01-04 16:36:58,272 - root - INFO - running command.raw hostname 2013-01-04 16:36:58,383 - root - INFO - command executed successfully 2013-01-04 16:36:58,386 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:58,386 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:58,387 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:58,479 - root - INFO - command executed successfully 2013-01-04 16:36:58,480 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:36:58,481 - root - INFO - 2013-01-04 16:36:58,482 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:36:58,482 - root - INFO - Please browse to http://kiwi-r105:8091/ to configure your server. 2013-01-04 16:36:58,483 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:36:58,484 - root - INFO - 2013-01-04 16:36:58,485 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:36:58,486 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:36:58,486 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:36:58,486 - root - INFO - 2013-01-04 16:36:58,487 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:36:58,488 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:36:58,488 - root - INFO - 2013-01-04 16:36:58,489 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:36:58,569 - root - INFO - command executed successfully 2013-01-04 16:36:58,571 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:36:58,685 - root - INFO - command executed successfully 2013-01-04 16:36:58,687 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:37:00,004 - root - INFO - command executed successfully 2013-01-04 16:37:00,005 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:37:00,006 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.6 finished 2013-01-04 16:37:09,052 - root - INFO - command executed successfully 2013-01-04 16:37:09,054 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:37:09,055 - root - INFO - 2013-01-04 16:37:09,056 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:37:09,057 - root - INFO - Please browse to http://kiwi-r104:8091/ to configure your server. 2013-01-04 16:37:09,058 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:37:09,058 - root - INFO - 2013-01-04 16:37:09,059 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:37:09,059 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:37:09,059 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:37:09,060 - root - INFO - 2013-01-04 16:37:09,060 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:37:09,060 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:37:09,061 - root - INFO - 2013-01-04 16:37:09,061 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:37:09,177 - root - INFO - command executed successfully 2013-01-04 16:37:09,179 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:37:09,289 - root - INFO - command executed successfully 2013-01-04 16:37:09,291 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:37:10,642 - root - INFO - command executed successfully 2013-01-04 16:37:10,644 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:37:10,645 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.5 finished 2013-01-04 16:37:15,657 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:37:15,658 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:37:15,658 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:37:16,228 - root - INFO - Connected 2013-01-04 16:37:16,235 - root - INFO - Connected 2013-01-04 16:37:16,430 - root - INFO - Connected 2013-01-04 16:37:19,746 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,749 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,759 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,838 - root - INFO - command executed successfully 2013-01-04 16:37:19,840 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,845 - root - INFO - command executed successfully 2013-01-04 16:37:19,846 - root - INFO - command executed successfully 2013-01-04 16:37:19,849 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,848 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,935 - root - INFO - command executed successfully 2013-01-04 16:37:19,936 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:19,962 - root - INFO - command executed successfully 2013-01-04 16:37:19,964 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:19,968 - root - INFO - command executed successfully 2013-01-04 16:37:19,969 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:20,047 - root - INFO - command executed successfully 2013-01-04 16:37:20,048 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,078 - root - INFO - command executed successfully 2013-01-04 16:37:20,080 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,081 - root - INFO - command executed successfully 2013-01-04 16:37:20,086 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,163 - root - INFO - command executed successfully 2013-01-04 16:37:20,167 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,198 - root - INFO - command executed successfully 2013-01-04 16:37:20,200 - root - INFO - command executed successfully 2013-01-04 16:37:20,206 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,208 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,288 - root - INFO - command executed successfully 2013-01-04 16:37:20,289 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,299 - root - INFO - command executed successfully 2013-01-04 16:37:20,301 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,303 - root - INFO - command executed successfully 2013-01-04 16:37:20,304 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,377 - root - INFO - command executed successfully 2013-01-04 16:37:20,378 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:20,393 - root - INFO - command executed successfully 2013-01-04 16:37:20,394 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:20,416 - root - INFO - command executed successfully 2013-01-04 16:37:20,417 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:22,371 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:22,375 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:22,510 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:25,374 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,378 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,390 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password 2013-01-04 16:37:25,396 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.3 finished thread initializer-thread-172.27.33.6 finished 2013-01-04 16:37:25,514 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,528 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.5 finished 2013-01-04 16:37:25,607 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:37:25,932 - root - INFO - Connected 2013-01-04 16:37:26,243 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:26,359 - root - INFO - command executed successfully 2013-01-04 16:37:26,360 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:26,474 - root - INFO - command executed successfully 2013-01-04 16:37:26,476 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:26,586 - root - INFO - command executed successfully 2013-01-04 16:37:26,589 - root - INFO - running command.raw hostname 2013-01-04 16:37:26,698 - root - INFO - command executed successfully 2013-01-04 16:37:26,818 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:37:27,142 - root - INFO - Connected 2013-01-04 16:37:27,416 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:27,504 - root - INFO - command executed successfully 2013-01-04 16:37:27,505 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:27,619 - root - INFO - command executed successfully 2013-01-04 16:37:27,620 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:27,734 - root - INFO - command executed successfully 2013-01-04 16:37:27,736 - root - INFO - running command.raw hostname 2013-01-04 16:37:27,849 - root - INFO - command executed successfully 2013-01-04 16:37:27,972 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:37:28,296 - root - INFO - Connected 2013-01-04 16:37:28,614 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:28,698 - root - INFO - command executed successfully 2013-01-04 16:37:28,700 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:28,809 - root - INFO - command executed successfully 2013-01-04 16:37:28,811 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:28,922 - root - INFO - command executed successfully 2013-01-04 16:37:28,924 - root - INFO - running command.raw hostname 2013-01-04 16:37:29,033 - root - INFO - command executed successfully 2013-01-04 16:37:29,156 - root - INFO - Installation of new version is done. Wait for 10 sec for rebalance 2013-01-04 16:51:41,243 - root - INFO - fail_over successful 2013-01-04 16:52:36,258 - root - INFO - adding node 172.27.33.3:8091 to cluster 2013-01-04 16:52:36,258 - root - INFO - adding remote node @172.27.33.3:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:37,575 - root - INFO - adding node 172.27.33.6:8091 to cluster 2013-01-04 16:52:37,576 - root - INFO - adding remote node @172.27.33.6:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:38,548 - root - INFO - adding node 172.27.33.5:8091 to cluster 2013-01-04 16:52:38,549 - root - INFO - adding remote node @172.27.33.5:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:39,763 - root - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5%2Cns_1%40172.27.33.2%2Cns_1%40172.27.33.4 2013-01-04 16:52:39,773 - root - INFO - rebalance operation started 2013-01-04 16:52:39,779 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:52:49,796 - root - INFO - rebalancing was completed with progress: 100% in 10.02257514 sec 2013-01-04 16:53:36,118 - root - INFO - Rebalance in all 2.0 Nodes 2013-01-04 16:53:47,639 - root - INFO - /diag/eval status: True content: 'ns_1@172.27.33.2' command: node(global:whereis_name(ns_orchestrator)) ERROR 2013-01-04 16:55:40,065 - root - INFO - ============== basetestcase cleanup was started for test #1 online_upgrade_rebalance_in_out ============== 2013-01-04 16:55:40,101 - root - INFO - deleting existing buckets on [ip:172.27.33.2 port:8091 ssh_username:root, ip:172.27.33.3 port:8091 ssh_username:root, ip:172.27.33.6 port:8091 ssh_username:root, ip:172.27.33.5 port:8091 ssh_username:root] 2013-01-04 16:55:40,402 - root - INFO - deleted bucket : default from 172.27.33.2 2013-01-04 16:55:40,402 - root - INFO - waiting for bucket deletion to complete.... 2013-01-04 16:55:40,407 - root - INFO - existing buckets : [] 2013-01-04 16:55:40,480 - root - INFO - rebalancing all nodes in order to remove nodes 2013-01-04 16:55:40,485 - root - INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5&user=Administrator&knownNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5%2Cns_1%40172.27.33.2 2013-01-04 16:55:40,491 - root - INFO - rebalance operation started 2013-01-04 16:55:40,499 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:55:44,511 - root - INFO - rebalance progress took 4.02038812637 seconds 2013-01-04 16:55:44,512 - root - INFO - sleep for 4.02038812637 seconds after rebalance... 2013-01-04 16:55:48,574 - root - INFO - removed all the nodes from cluster associated with ip:172.27.33.2 port:8091 ssh_username:root ? [(u'ns_1@172.27.33.6', 8091), (u'ns_1@172.27.33.3', 8091), (u'ns_1@172.27.33.5', 8091)] 2013-01-04 16:55:58,593 - root - INFO - waiting for ns_server @ 172.27.33.2:8091 2013-01-04 16:55:58,602 - root - INFO - ns_server @ 172.27.33.2:8091 is running 2013-01-04 16:55:58,609 - root - INFO - waiting for ns_server @ 172.27.33.3:8091 2013-01-04 16:55:58,616 - root - INFO - ns_server @ 172.27.33.3:8091 is running 2013-01-04 16:55:58,623 - root - INFO - waiting for ns_server @ 172.27.33.6:8091 2013-01-04 16:55:58,629 - root - INFO - ns_server @ 172.27.33.6:8091 is running 2013-01-04 16:55:58,637 - root - INFO - waiting for ns_server @ 172.27.33.5:8091 2013-01-04 16:55:58,644 - root - INFO - ns_server @ 172.27.33.5:8091 is running 2013-01-04 16:55:58,645 - root - INFO - ============== basetestcase cleanup was finished for test #1 online_upgrade_rebalance_in_out ============== ====================================================================== ERROR: online_upgrade_rebalance_in_out (newupgradetests.MultiNodesUpgradeTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/newupgradetests.py", line 195, in online_upgrade_rebalance_in_out self.online_upgrade() File "pytests/newupgradetests.py", line 240, in online_upgrade raise Exception("After rebalance in 2.0 Nodes, 2.0 doesn't become the master") Exception: After rebalance in 2.0 Nodes, 2.0 doesn't become the master ---------------------------------------------------------------------- Ran 1 test in 1788.471s FAILED (errors=1) summary so far suite newupgradetests.MultiNodesUpgradeTests , pass 0 , fail 1 failures so far... newupgradetests.MultiNodesUpgradeTests.online_upgrade_rebalance_in_out testrunner logs, diags and results are available under logs/testrunner-13-Jan-04_16-26-102013-01-04 16:26:20,869 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:26:20,979 - root - INFO - command executed successfully 2013-01-04 16:26:20,981 - root - INFO - running command.raw hostname 2013-01-04 16:26:20,985 - root - INFO - command executed successfully 2013-01-04 16:26:20,986 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:26:21,096 - root - INFO - command executed successfully 2013-01-04 16:26:21,102 - root - INFO - command executed successfully 2013-01-04 16:26:21,103 - root - INFO - running command.raw hostname 2013-01-04 16:26:21,194 - root - INFO - command executed successfully 2013-01-04 16:27:02,379 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:27:02,704 - root - INFO - Connected 2013-01-04 16:27:03,633 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:27:03,796 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:03,995 - root - INFO - Connected 2013-01-04 16:27:04,026 - root - INFO - command executed successfully 2013-01-04 16:27:04,028 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,138 - root - INFO - command executed successfully 2013-01-04 16:27:04,139 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,233 - root - INFO - command executed successfully 2013-01-04 16:27:04,235 - root - INFO - running command.raw hostname 2013-01-04 16:27:04,307 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:04,354 - root - INFO - command executed successfully 2013-01-04 16:27:04,418 - root - INFO - command executed successfully 2013-01-04 16:27:04,420 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,548 - root - INFO - command executed successfully 2013-01-04 16:27:04,549 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,635 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:04,660 - root - INFO - command executed successfully 2013-01-04 16:27:04,662 - root - INFO - running command.raw hostname 2013-01-04 16:27:04,720 - root - INFO - command executed successfully 2013-01-04 16:27:04,721 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,772 - root - INFO - command executed successfully 2013-01-04 16:27:04,805 - root - INFO - command executed successfully 2013-01-04 16:27:04,806 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,919 - root - INFO - command executed successfully 2013-01-04 16:27:04,921 - root - INFO - running command.raw hostname 2013-01-04 16:27:05,036 - root - INFO - command executed successfully 2013-01-04 16:27:05,040 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:27:05,059 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:05,145 - root - INFO - command executed successfully 2013-01-04 16:27:05,147 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:05,160 - root - INFO - command executed successfully 2013-01-04 16:27:05,162 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm': No such file or directory 2013-01-04 16:27:05,162 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:27:05,163 - root - ERROR - rm: cannot remove `cb_1275': Is a directory 2013-01-04 16:27:05,164 - root - ERROR - mv: cannot stat `cb_1275/*': No such file or directory 2013-01-04 16:27:05,165 - root - INFO - get md5 sum for local and remote 2013-01-04 16:27:05,166 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm > couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l 2013-01-04 16:27:05,260 - root - INFO - command executed successfully 2013-01-04 16:27:05,262 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:05,372 - root - INFO - command executed successfully 2013-01-04 16:27:05,373 - root - INFO - running command.raw hostname 2013-01-04 16:27:05,483 - root - INFO - command executed successfully 2013-01-04 16:27:05,487 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:27:05,604 - root - INFO - command executed successfully 2013-01-04 16:27:05,606 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm': No such file or directory 2013-01-04 16:27:05,607 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:27:05,608 - root - ERROR - rm: cannot remove `cb_0006': Is a directory 2013-01-04 16:27:05,609 - root - ERROR - rm: cannot remove `default': Is a directory 2013-01-04 16:27:05,609 - root - ERROR - mv: cannot stat `cb_0006/*': No such file or directory 2013-01-04 16:27:05,610 - root - INFO - get md5 sum for local and remote 2013-01-04 16:27:05,610 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm > couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l 2013-01-04 16:27:05,773 - root - INFO - command executed successfully 2013-01-04 16:27:05,774 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm: No such file or directory 2013-01-04 16:27:05,775 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:27:05,777 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:27:06,209 - root - INFO - command executed successfully 2013-01-04 16:27:06,210 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm: No such file or directory 2013-01-04 16:27:06,211 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:27:06,212 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:30:09,539 - root - INFO - command executed successfully 2013-01-04 16:30:09,540 - root - INFO - 1d0 2013-01-04 16:30:09,541 - root - INFO - < 4fceddd71c9a4468940d6b070fcda56a couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:09,822 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:09,933 - root - INFO - command executed successfully 2013-01-04 16:30:09,934 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:10,043 - root - INFO - command executed successfully 2013-01-04 16:30:10,046 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:10,156 - root - INFO - command executed successfully 2013-01-04 16:30:10,158 - root - INFO - running command.raw hostname 2013-01-04 16:30:10,267 - root - INFO - command executed successfully 2013-01-04 16:30:10,271 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:30:10,382 - root - INFO - command executed successfully 2013-01-04 16:30:10,654 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:10,738 - root - INFO - command executed successfully 2013-01-04 16:30:10,740 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:10,851 - root - INFO - command executed successfully 2013-01-04 16:30:10,853 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:10,962 - root - INFO - command executed successfully 2013-01-04 16:30:10,964 - root - INFO - running command.raw hostname 2013-01-04 16:30:11,074 - root - INFO - command executed successfully 2013-01-04 16:30:11,078 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:30:11,185 - root - INFO - command executed successfully 2013-01-04 16:30:11,515 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:11,624 - root - INFO - command executed successfully 2013-01-04 16:30:11,626 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:11,736 - root - INFO - command executed successfully 2013-01-04 16:30:11,738 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:11,849 - root - INFO - command executed successfully 2013-01-04 16:30:11,851 - root - INFO - running command.raw hostname 2013-01-04 16:30:11,962 - root - INFO - command executed successfully 2013-01-04 16:30:11,966 - root - INFO - deliverable_type : rpm 2013-01-04 16:30:11,966 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:30:11,966 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:22,831 - root - INFO - command executed successfully 2013-01-04 16:30:22,832 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:30:22,833 - root - INFO - 2013-01-04 16:30:22,834 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:30:22,835 - root - INFO - Please browse to http://kiwi-r101:8091/ to configure your server. 2013-01-04 16:30:22,836 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:30:22,836 - root - INFO - 2013-01-04 16:30:22,837 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:30:22,837 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:30:22,838 - root - INFO - 8091 and from 21100 to 21299. 2013-01-04 16:30:22,838 - root - INFO - 2013-01-04 16:30:22,838 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:30:22,839 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:30:22,839 - root - INFO - 2013-01-04 16:30:22,839 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:30:22,923 - root - INFO - command executed successfully 2013-01-04 16:30:22,925 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:30:23,035 - root - INFO - command executed successfully 2013-01-04 16:30:23,037 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:30:24,392 - root - INFO - command executed successfully 2013-01-04 16:30:24,393 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:30:24,394 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.2 finished 2013-01-04 16:30:31,632 - root - INFO - command executed successfully 2013-01-04 16:30:31,633 - root - INFO - 1d0 2013-01-04 16:30:31,635 - root - INFO - < 4fceddd71c9a4468940d6b070fcda56a couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:31,912 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:32,022 - root - INFO - command executed successfully 2013-01-04 16:30:32,024 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:32,134 - root - INFO - command executed successfully 2013-01-04 16:30:32,135 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:32,246 - root - INFO - command executed successfully 2013-01-04 16:30:32,248 - root - INFO - running command.raw hostname 2013-01-04 16:30:32,358 - root - INFO - command executed successfully 2013-01-04 16:30:32,361 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:30:32,471 - root - INFO - command executed successfully 2013-01-04 16:30:32,747 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:32,858 - root - INFO - command executed successfully 2013-01-04 16:30:32,859 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:32,969 - root - INFO - command executed successfully 2013-01-04 16:30:32,971 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:33,056 - root - INFO - command executed successfully 2013-01-04 16:30:33,058 - root - INFO - running command.raw hostname 2013-01-04 16:30:33,168 - root - INFO - command executed successfully 2013-01-04 16:30:33,172 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:30:33,279 - root - INFO - command executed successfully 2013-01-04 16:30:33,667 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:33,777 - root - INFO - command executed successfully 2013-01-04 16:30:33,779 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:33,889 - root - INFO - command executed successfully 2013-01-04 16:30:33,892 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:34,002 - root - INFO - command executed successfully 2013-01-04 16:30:34,003 - root - INFO - running command.raw hostname 2013-01-04 16:30:34,113 - root - INFO - command executed successfully 2013-01-04 16:30:34,117 - root - INFO - deliverable_type : rpm 2013-01-04 16:30:34,117 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:30:34,118 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:46,417 - root - INFO - command executed successfully 2013-01-04 16:30:46,419 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:30:46,420 - root - INFO - 2013-01-04 16:30:46,422 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:30:46,422 - root - INFO - Please browse to http://kiwi-r103:8091/ to configure your server. 2013-01-04 16:30:46,423 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:30:46,423 - root - INFO - 2013-01-04 16:30:46,423 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:30:46,424 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:30:46,424 - root - INFO - 8091 and from 21100 to 21299. 2013-01-04 16:30:46,424 - root - INFO - 2013-01-04 16:30:46,425 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:30:46,425 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:30:46,426 - root - INFO - 2013-01-04 16:30:46,426 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:30:46,542 - root - INFO - command executed successfully 2013-01-04 16:30:46,544 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:30:46,653 - root - INFO - command executed successfully 2013-01-04 16:30:46,655 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:30:47,985 - root - INFO - command executed successfully 2013-01-04 16:30:47,986 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:30:47,988 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.4 finished 2013-01-04 16:30:52,999 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:30:53,000 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:30:53,551 - root - INFO - Connected lib/membase/api/rest_client.py:186: DeprecationWarning: object.__new__() takes no parameters obj = object.__new__(self, serverInfo) 2013-01-04 16:30:53,577 - root - INFO - Connected 2013-01-04 16:30:56,803 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:56,825 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:56,889 - root - INFO - command executed successfully 2013-01-04 16:30:56,890 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:56,917 - root - INFO - command executed successfully 2013-01-04 16:30:56,919 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:57,001 - root - INFO - command executed successfully 2013-01-04 16:30:57,002 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:57,029 - root - INFO - command executed successfully 2013-01-04 16:30:57,031 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:57,114 - root - INFO - command executed successfully 2013-01-04 16:30:57,115 - root - INFO - running command.raw hostname 2013-01-04 16:30:57,140 - root - INFO - command executed successfully 2013-01-04 16:30:57,141 - root - INFO - running command.raw hostname 2013-01-04 16:30:57,225 - root - INFO - command executed successfully 2013-01-04 16:30:57,228 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:30:57,252 - root - INFO - command executed successfully 2013-01-04 16:30:57,256 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:30:57,354 - root - INFO - command executed successfully 2013-01-04 16:30:57,355 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:30:57,387 - root - INFO - command executed successfully 2013-01-04 16:30:57,387 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:30:57,469 - root - INFO - command executed successfully 2013-01-04 16:30:57,471 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:30:57,498 - root - INFO - command executed successfully 2013-01-04 16:30:57,500 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:30:59,348 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:30:59,755 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:31:02,351 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:02,365 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.2 finished 2013-01-04 16:31:02,757 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:02,770 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.4 finished 2013-01-04 16:31:02,813 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:31:03,134 - root - INFO - Connected 2013-01-04 16:31:03,453 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:03,563 - root - INFO - command executed successfully 2013-01-04 16:31:03,565 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:03,675 - root - INFO - command executed successfully 2013-01-04 16:31:03,677 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:03,787 - root - INFO - command executed successfully 2013-01-04 16:31:03,789 - root - INFO - running command.raw hostname 2013-01-04 16:31:03,899 - root - INFO - command executed successfully 2013-01-04 16:31:04,021 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:31:04,344 - root - INFO - Connected 2013-01-04 16:31:04,655 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:04,766 - root - INFO - command executed successfully 2013-01-04 16:31:04,767 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:04,853 - root - INFO - command executed successfully 2013-01-04 16:31:04,854 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:04,965 - root - INFO - command executed successfully 2013-01-04 16:31:04,966 - root - INFO - running command.raw hostname 2013-01-04 16:31:05,076 - root - INFO - command executed successfully 2013-01-04 16:31:05,206 - root - INFO - Installation of old version is done. Wait for 10 sec for upgrade 2013-01-04 16:31:05,382 - root - INFO - adding node 172.27.33.4:8091 to cluster 2013-01-04 16:31:05,382 - root - INFO - adding remote node @172.27.33.4:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:31:05,913 - root - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.27.33.2%2Cns_1%40172.27.33.4 2013-01-04 16:31:05,918 - root - INFO - rebalance operation started 2013-01-04 16:31:05,927 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:31:15,943 - root - INFO - rebalancing was completed with progress: 100% in 10.0244619846 sec 2013-01-04 16:31:16,953 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:16,964 - root - INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password 2013-01-04 16:31:16,988 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:16,999 - root - INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password 2013-01-04 16:31:18,012 - root - INFO - http://172.27.33.2:8091/pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=None&flushEnabled=1&replicaNumber=1&ramQuotaMB=1456 2013-01-04 16:31:18,026 - root - INFO - 0.0136511325836 seconds to create bucket default 2013-01-04 16:31:18,026 - root - INFO - waiting for memcached bucket : default in 172.27.33.2 to accept set ops 2013-01-04 16:31:19,215 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,309 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:19,439 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,525 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:19,666 - root - INFO - bucket 'default' was created with per node RAM quota: 1456 2013-01-04 16:31:19,734 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,817 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:20,669 - root - INFO - Batch create documents done #: 0 2013-01-04 16:31:21,093 - root - INFO - Batch create documents done #: 1000 2013-01-04 16:31:21,687 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:21,759 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.27.33.2:8091' 2013-01-04 16:31:21,773 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:21,852 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '172.27.33.2:8091' 2013-01-04 16:31:21,867 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:21,944 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.27.33.4:8091' 2013-01-04 16:31:21,961 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:22,038 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '172.27.33.4:8091' 2013-01-04 16:31:32,051 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,052 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,053 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,813 - root - INFO - Connected 2013-01-04 16:31:32,815 - root - INFO - Connected 2013-01-04 16:31:32,815 - root - INFO - Connected 2013-01-04 16:31:33,168 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,172 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,196 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,255 - root - INFO - command executed successfully 2013-01-04 16:31:33,257 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,261 - root - INFO - command executed successfully 2013-01-04 16:31:33,262 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,312 - root - INFO - command executed successfully 2013-01-04 16:31:33,314 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,367 - root - INFO - command executed successfully 2013-01-04 16:31:33,369 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,375 - root - INFO - command executed successfully 2013-01-04 16:31:33,376 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,425 - root - INFO - command executed successfully 2013-01-04 16:31:33,426 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,466 - root - INFO - command executed successfully 2013-01-04 16:31:33,467 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,481 - root - INFO - command executed successfully 2013-01-04 16:31:33,482 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,514 - root - INFO - command executed successfully 2013-01-04 16:31:33,515 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,583 - root - INFO - command executed successfully 2013-01-04 16:31:33,588 - root - INFO - CentOS 2013-01-04 16:31:33,589 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,591 - root - INFO - command executed successfully 2013-01-04 16:31:33,595 - root - INFO - CentOS 2013-01-04 16:31:33,596 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,627 - root - INFO - command executed successfully 2013-01-04 16:31:33,630 - root - INFO - CentOS 2013-01-04 16:31:33,631 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,991 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,996 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:34,008 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:34,080 - root - INFO - command executed successfully 2013-01-04 16:31:34,081 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,084 - root - INFO - command executed successfully 2013-01-04 16:31:34,085 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,123 - root - INFO - command executed successfully 2013-01-04 16:31:34,124 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,172 - root - INFO - command executed successfully 2013-01-04 16:31:34,173 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,175 - root - INFO - command executed successfully 2013-01-04 16:31:34,176 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,237 - root - INFO - command executed successfully 2013-01-04 16:31:34,239 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,265 - root - INFO - command executed successfully 2013-01-04 16:31:34,266 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,268 - root - INFO - command executed successfully 2013-01-04 16:31:34,269 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,351 - root - INFO - command executed successfully 2013-01-04 16:31:34,357 - root - INFO - command executed successfully 2013-01-04 16:31:34,358 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,359 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,370 - root - INFO - command executed successfully 2013-01-04 16:31:34,373 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,459 - root - INFO - command executed successfully 2013-01-04 16:31:34,461 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,461 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,465 - root - INFO - command executed successfully 2013-01-04 16:31:34,469 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,502 - root - INFO - command executed successfully 2013-01-04 16:31:34,504 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,505 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,574 - root - INFO - command executed successfully 2013-01-04 16:31:34,576 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,577 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,583 - root - INFO - command executed successfully 2013-01-04 16:31:34,584 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,585 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,617 - root - INFO - command executed successfully 2013-01-04 16:31:34,618 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,619 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,689 - root - INFO - command executed successfully 2013-01-04 16:31:34,691 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,699 - root - INFO - command executed successfully 2013-01-04 16:31:34,700 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,701 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,707 - root - INFO - command executed successfully 2013-01-04 16:31:34,708 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,800 - root - INFO - command executed successfully 2013-01-04 16:31:34,801 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,813 - root - INFO - command executed successfully 2013-01-04 16:31:34,814 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,820 - root - INFO - command executed successfully 2013-01-04 16:31:34,821 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,911 - root - INFO - command executed successfully 2013-01-04 16:31:34,912 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:34,914 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:34,924 - root - INFO - command executed successfully 2013-01-04 16:31:34,925 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,936 - root - INFO - command executed successfully 2013-01-04 16:31:34,937 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:34,938 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:35,015 - root - INFO - command executed successfully 2013-01-04 16:31:35,016 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:35,018 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:35,025 - root - INFO - command executed successfully 2013-01-04 16:31:35,026 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,027 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,053 - root - INFO - command executed successfully 2013-01-04 16:31:35,054 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,055 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,130 - root - INFO - command executed successfully 2013-01-04 16:31:35,131 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,132 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,140 - root - INFO - command executed successfully 2013-01-04 16:31:35,142 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,169 - root - INFO - command executed successfully 2013-01-04 16:31:35,170 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,219 - root - INFO - command executed successfully 2013-01-04 16:31:35,221 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,251 - root - INFO - command executed successfully 2013-01-04 16:31:35,252 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,281 - root - INFO - command executed successfully 2013-01-04 16:31:35,282 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,337 - root - INFO - command executed successfully 2013-01-04 16:31:35,338 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,362 - root - INFO - command executed successfully 2013-01-04 16:31:35,364 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,394 - root - INFO - command executed successfully 2013-01-04 16:31:35,395 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,454 - root - INFO - command executed successfully 2013-01-04 16:31:35,456 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,475 - root - INFO - command executed successfully 2013-01-04 16:31:35,476 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,511 - root - INFO - command executed successfully 2013-01-04 16:31:35,512 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,570 - root - INFO - command executed successfully 2013-01-04 16:31:35,571 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,587 - root - INFO - command executed successfully 2013-01-04 16:31:35,588 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,602 - root - INFO - command executed successfully 2013-01-04 16:31:35,603 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,686 - root - INFO - command executed successfully 2013-01-04 16:31:35,688 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,698 - root - INFO - command executed successfully 2013-01-04 16:31:35,718 - root - INFO - command executed successfully 2013-01-04 16:31:35,815 - root - INFO - command executed successfully 2013-01-04 16:31:36,006 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,074 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,089 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,095 - root - INFO - command executed successfully 2013-01-04 16:31:36,097 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,180 - root - INFO - command executed successfully 2013-01-04 16:31:36,181 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,189 - root - INFO - command executed successfully 2013-01-04 16:31:36,189 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,207 - root - INFO - command executed successfully 2013-01-04 16:31:36,208 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,271 - root - INFO - command executed successfully 2013-01-04 16:31:36,272 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,281 - root - INFO - command executed successfully 2013-01-04 16:31:36,283 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,318 - root - INFO - command executed successfully 2013-01-04 16:31:36,319 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,385 - root - INFO - command executed successfully 2013-01-04 16:31:36,388 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,395 - root - INFO - command executed successfully 2013-01-04 16:31:36,396 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,430 - root - INFO - command executed successfully 2013-01-04 16:31:36,434 - root - INFO - CentOS 2013-01-04 16:31:36,435 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,435 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:36,504 - root - INFO - command executed successfully 2013-01-04 16:31:36,509 - root - INFO - CentOS 2013-01-04 16:31:36,509 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,509 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:36,512 - root - INFO - command executed successfully 2013-01-04 16:31:36,515 - root - INFO - CentOS 2013-01-04 16:31:36,516 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,517 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:39,326 - root - INFO - command executed successfully 2013-01-04 16:31:39,327 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,328 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,329 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,330 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,331 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,332 - root - INFO - 2013-01-04 16:31:39,332 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,341 - root - INFO - command executed successfully 2013-01-04 16:31:39,342 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,342 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,343 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,343 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,344 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,345 - root - INFO - 2013-01-04 16:31:39,346 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,381 - root - INFO - command executed successfully 2013-01-04 16:31:39,382 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,382 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,384 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,386 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,386 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,387 - root - INFO - 2013-01-04 16:31:39,387 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,447 - root - INFO - command executed successfully 2013-01-04 16:31:39,449 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,458 - root - INFO - command executed successfully 2013-01-04 16:31:39,460 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,502 - root - INFO - command executed successfully 2013-01-04 16:31:39,504 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,559 - root - INFO - command executed successfully 2013-01-04 16:31:39,561 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,571 - root - INFO - command executed successfully 2013-01-04 16:31:39,573 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,613 - root - INFO - command executed successfully 2013-01-04 16:31:39,614 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,674 - root - INFO - command executed successfully 2013-01-04 16:31:39,676 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,677 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,683 - root - INFO - command executed successfully 2013-01-04 16:31:39,685 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,686 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,727 - root - INFO - command executed successfully 2013-01-04 16:31:39,729 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,730 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,787 - root - INFO - command executed successfully 2013-01-04 16:31:39,788 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,790 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,796 - root - INFO - command executed successfully 2013-01-04 16:31:39,798 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,799 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,842 - root - INFO - command executed successfully 2013-01-04 16:31:39,843 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,844 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,905 - root - INFO - command executed successfully 2013-01-04 16:31:39,907 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,908 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:39,909 - root - INFO - command executed successfully 2013-01-04 16:31:39,912 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,912 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:39,962 - root - INFO - command executed successfully 2013-01-04 16:31:39,963 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,964 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:40,024 - root - INFO - command executed successfully 2013-01-04 16:31:40,026 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,029 - root - INFO - command executed successfully 2013-01-04 16:31:40,031 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,082 - root - INFO - command executed successfully 2013-01-04 16:31:40,083 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,137 - root - INFO - command executed successfully 2013-01-04 16:31:40,138 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,140 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,143 - root - INFO - command executed successfully 2013-01-04 16:31:40,144 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,145 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,195 - root - INFO - command executed successfully 2013-01-04 16:31:40,196 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,196 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,253 - root - INFO - command executed successfully 2013-01-04 16:31:40,254 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,255 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,260 - root - INFO - command executed successfully 2013-01-04 16:31:40,261 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,263 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,307 - root - INFO - command executed successfully 2013-01-04 16:31:40,308 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,309 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,367 - root - INFO - command executed successfully 2013-01-04 16:31:40,368 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,377 - root - INFO - command executed successfully 2013-01-04 16:31:40,379 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,424 - root - INFO - command executed successfully 2013-01-04 16:31:40,425 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,481 - root - INFO - command executed successfully 2013-01-04 16:31:40,483 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,496 - root - INFO - command executed successfully 2013-01-04 16:31:40,497 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,543 - root - INFO - command executed successfully 2013-01-04 16:31:40,544 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,580 - root - INFO - command executed successfully 2013-01-04 16:31:40,581 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,593 - root - INFO - command executed successfully 2013-01-04 16:31:40,594 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,659 - root - INFO - command executed successfully 2013-01-04 16:31:40,660 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,697 - root - INFO - command executed successfully 2013-01-04 16:31:40,699 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,707 - root - INFO - command executed successfully 2013-01-04 16:31:40,708 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,777 - root - INFO - command executed successfully 2013-01-04 16:31:40,778 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,815 - root - INFO - command executed successfully 2013-01-04 16:31:40,817 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,821 - root - INFO - command executed successfully 2013-01-04 16:31:40,823 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,892 - root - INFO - command executed successfully 2013-01-04 16:31:40,894 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,934 - root - INFO - command executed successfully 2013-01-04 16:31:40,937 - root - INFO - command executed successfully 2013-01-04 16:31:41,008 - root - INFO - command executed successfully 2013-01-04 16:31:41,317 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,321 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,345 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,402 - root - INFO - command executed successfully 2013-01-04 16:31:41,404 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,437 - root - INFO - command executed successfully 2013-01-04 16:31:41,439 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,459 - root - INFO - command executed successfully 2013-01-04 16:31:41,461 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,515 - root - INFO - command executed successfully 2013-01-04 16:31:41,517 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,553 - root - INFO - command executed successfully 2013-01-04 16:31:41,555 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,575 - root - INFO - command executed successfully 2013-01-04 16:31:41,576 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,628 - root - INFO - command executed successfully 2013-01-04 16:31:41,629 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,669 - root - INFO - command executed successfully 2013-01-04 16:31:41,672 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,693 - root - INFO - command executed successfully 2013-01-04 16:31:41,694 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,741 - root - INFO - command executed successfully 2013-01-04 16:31:41,745 - root - INFO - CentOS 2013-01-04 16:31:41,746 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,746 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,786 - root - INFO - command executed successfully 2013-01-04 16:31:41,790 - root - INFO - CentOS 2013-01-04 16:31:41,790 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,791 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,809 - root - INFO - command executed successfully 2013-01-04 16:31:41,811 - root - INFO - CentOS 2013-01-04 16:31:41,812 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,813 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,869 - root - INFO - command executed successfully 2013-01-04 16:31:41,871 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,872 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,909 - root - INFO - command executed successfully 2013-01-04 16:31:41,911 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,912 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,929 - root - INFO - command executed successfully 2013-01-04 16:31:41,931 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,932 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,980 - root - INFO - command executed successfully 2013-01-04 16:31:41,982 - root - ERROR - beam: no process killed 2013-01-04 16:31:41,983 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,027 - root - INFO - command executed successfully 2013-01-04 16:31:42,029 - root - ERROR - beam: no process killed 2013-01-04 16:31:42,030 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,046 - root - INFO - command executed successfully 2013-01-04 16:31:42,047 - root - ERROR - beam: no process killed 2013-01-04 16:31:42,049 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,094 - root - INFO - command executed successfully 2013-01-04 16:31:42,096 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,097 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,147 - root - INFO - command executed successfully 2013-01-04 16:31:42,149 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,149 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,159 - root - INFO - command executed successfully 2013-01-04 16:31:42,160 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,161 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,208 - root - INFO - command executed successfully 2013-01-04 16:31:42,263 - root - INFO - command executed successfully 2013-01-04 16:31:42,276 - root - INFO - command executed successfully thread uninstaller-thread-172.27.33.3 finished thread uninstaller-thread-172.27.33.6 finished thread uninstaller-thread-172.27.33.5 finished 2013-01-04 16:31:42,278 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,279 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,279 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,852 - root - INFO - Connected 2013-01-04 16:31:42,853 - root - INFO - Connected 2013-01-04 16:31:43,045 - root - INFO - Connected 2013-01-04 16:31:43,351 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,371 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,418 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,453 - root - INFO - command executed successfully 2013-01-04 16:31:43,455 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,462 - root - INFO - command executed successfully 2013-01-04 16:31:43,463 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,505 - root - INFO - command executed successfully 2013-01-04 16:31:43,507 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,568 - root - INFO - command executed successfully 2013-01-04 16:31:43,569 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,580 - root - INFO - command executed successfully 2013-01-04 16:31:43,581 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,622 - root - INFO - command executed successfully 2013-01-04 16:31:43,623 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,680 - root - INFO - command executed successfully 2013-01-04 16:31:43,681 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,699 - root - INFO - command executed successfully 2013-01-04 16:31:43,700 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,738 - root - INFO - command executed successfully 2013-01-04 16:31:43,739 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,787 - root - INFO - command executed successfully 2013-01-04 16:31:43,793 - root - INFO - command executed successfully 2013-01-04 16:31:43,828 - root - INFO - command executed successfully 2013-01-04 16:32:46,827 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:32:47,158 - root - INFO - Connected 2013-01-04 16:32:47,469 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:48,298 - root - INFO - command executed successfully 2013-01-04 16:32:48,325 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:48,804 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:32:48,819 - root - INFO - command executed successfully 2013-01-04 16:32:48,845 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:49,056 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:32:49,172 - root - INFO - command executed successfully 2013-01-04 16:32:49,173 - root - INFO - running command.raw hostname 2013-01-04 16:32:49,380 - root - INFO - Connected 2013-01-04 16:32:49,567 - root - INFO - command executed successfully 2013-01-04 16:32:49,581 - root - INFO - Connected 2013-01-04 16:32:49,923 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:49,930 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:49,937 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,013 - root - INFO - command executed successfully 2013-01-04 16:32:50,014 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,039 - root - INFO - command executed successfully 2013-01-04 16:32:50,040 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,053 - root - INFO - command executed successfully 2013-01-04 16:32:50,054 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,125 - root - INFO - command executed successfully 2013-01-04 16:32:50,126 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,153 - root - INFO - command executed successfully 2013-01-04 16:32:50,155 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,166 - root - INFO - command executed successfully 2013-01-04 16:32:50,167 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,213 - root - INFO - command executed successfully 2013-01-04 16:32:50,215 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,270 - root - INFO - command executed successfully 2013-01-04 16:32:50,271 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,283 - root - INFO - command executed successfully 2013-01-04 16:32:50,285 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,325 - root - INFO - command executed successfully 2013-01-04 16:32:50,330 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:50,384 - root - INFO - command executed successfully 2013-01-04 16:32:50,402 - root - INFO - command executed successfully 2013-01-04 16:32:50,450 - root - INFO - command executed successfully 2013-01-04 16:32:50,452 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:50,453 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:50,454 - root - ERROR - rm: cannot remove `cb_8077': Is a directory 2013-01-04 16:32:50,455 - root - ERROR - mv: cannot stat `cb_8077/*': No such file or directory 2013-01-04 16:32:50,456 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:50,457 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:50,702 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,716 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,804 - root - INFO - command executed successfully 2013-01-04 16:32:50,805 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,814 - root - INFO - command executed successfully 2013-01-04 16:32:50,816 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,920 - root - INFO - command executed successfully 2013-01-04 16:32:50,922 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,928 - root - INFO - command executed successfully 2013-01-04 16:32:50,930 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:51,038 - root - INFO - command executed successfully 2013-01-04 16:32:51,040 - root - INFO - running command.raw hostname 2013-01-04 16:32:51,043 - root - INFO - command executed successfully 2013-01-04 16:32:51,045 - root - INFO - running command.raw hostname 2013-01-04 16:32:51,059 - root - INFO - command executed successfully 2013-01-04 16:32:51,060 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,062 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,063 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:32:51,157 - root - INFO - command executed successfully 2013-01-04 16:32:51,161 - root - INFO - command executed successfully 2013-01-04 16:32:51,165 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:51,168 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:51,254 - root - INFO - command executed successfully 2013-01-04 16:32:51,256 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:51,257 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:51,258 - root - ERROR - rm: cannot remove `cb_1951': Is a directory 2013-01-04 16:32:51,258 - root - ERROR - mv: cannot stat `cb_1951/*': No such file or directory 2013-01-04 16:32:51,259 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:51,259 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:51,287 - root - INFO - command executed successfully 2013-01-04 16:32:51,289 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:51,290 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:51,291 - root - ERROR - rm: cannot remove `cb_1410': Is a directory 2013-01-04 16:32:51,291 - root - ERROR - mv: cannot stat `cb_1410/*': No such file or directory 2013-01-04 16:32:51,292 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:51,292 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:51,842 - root - INFO - command executed successfully 2013-01-04 16:32:51,843 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,845 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,846 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:32:51,869 - root - INFO - command executed successfully 2013-01-04 16:32:51,870 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,872 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,872 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:36:32,684 - root - INFO - command executed successfully 2013-01-04 16:36:32,686 - root - INFO - 1d0 2013-01-04 16:36:32,687 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:32,951 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:33,068 - root - INFO - command executed successfully 2013-01-04 16:36:33,069 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:33,184 - root - INFO - command executed successfully 2013-01-04 16:36:33,186 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:33,300 - root - INFO - command executed successfully 2013-01-04 16:36:33,302 - root - INFO - running command.raw hostname 2013-01-04 16:36:33,417 - root - INFO - command executed successfully 2013-01-04 16:36:33,421 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:33,528 - root - INFO - command executed successfully 2013-01-04 16:36:33,779 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:33,893 - root - INFO - command executed successfully 2013-01-04 16:36:33,894 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:34,008 - root - INFO - command executed successfully 2013-01-04 16:36:34,010 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:34,125 - root - INFO - command executed successfully 2013-01-04 16:36:34,126 - root - INFO - running command.raw hostname 2013-01-04 16:36:34,236 - root - INFO - command executed successfully 2013-01-04 16:36:34,240 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:34,347 - root - INFO - command executed successfully 2013-01-04 16:36:34,645 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:34,736 - root - INFO - command executed successfully 2013-01-04 16:36:34,737 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:34,853 - root - INFO - command executed successfully 2013-01-04 16:36:34,854 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:34,967 - root - INFO - command executed successfully 2013-01-04 16:36:34,969 - root - INFO - running command.raw hostname 2013-01-04 16:36:35,082 - root - INFO - command executed successfully 2013-01-04 16:36:35,086 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:35,087 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:35,088 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:44,640 - root - INFO - command executed successfully 2013-01-04 16:36:44,641 - root - INFO - 1d0 2013-01-04 16:36:44,642 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:44,920 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:45,034 - root - INFO - command executed successfully 2013-01-04 16:36:45,035 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:45,152 - root - INFO - command executed successfully 2013-01-04 16:36:45,155 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:45,273 - root - INFO - command executed successfully 2013-01-04 16:36:45,274 - root - INFO - running command.raw hostname 2013-01-04 16:36:45,387 - root - INFO - command executed successfully 2013-01-04 16:36:45,391 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:45,499 - root - INFO - command executed successfully 2013-01-04 16:36:45,748 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:45,864 - root - INFO - command executed successfully 2013-01-04 16:36:45,865 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:45,980 - root - INFO - command executed successfully 2013-01-04 16:36:45,981 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:45,985 - root - INFO - command executed successfully 2013-01-04 16:36:45,986 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:36:45,988 - root - INFO - 2013-01-04 16:36:45,989 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:36:45,989 - root - INFO - Please browse to http://kiwi-r102:8091/ to configure your server. 2013-01-04 16:36:45,991 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:36:45,992 - root - INFO - 2013-01-04 16:36:45,992 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:36:45,994 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:36:45,995 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:36:45,995 - root - INFO - 2013-01-04 16:36:45,996 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:36:45,996 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:36:45,996 - root - INFO - 2013-01-04 16:36:45,997 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:36:46,074 - root - INFO - command executed successfully 2013-01-04 16:36:46,075 - root - INFO - running command.raw hostname 2013-01-04 16:36:46,080 - root - INFO - command executed successfully 2013-01-04 16:36:46,081 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:36:46,187 - root - INFO - command executed successfully 2013-01-04 16:36:46,190 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:46,197 - root - INFO - command executed successfully 2013-01-04 16:36:46,197 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:36:46,305 - root - INFO - command executed successfully 2013-01-04 16:36:46,616 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:46,704 - root - INFO - command executed successfully 2013-01-04 16:36:46,705 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:46,815 - root - INFO - command executed successfully 2013-01-04 16:36:46,817 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:46,930 - root - INFO - command executed successfully 2013-01-04 16:36:46,931 - root - INFO - running command.raw hostname 2013-01-04 16:36:47,046 - root - INFO - command executed successfully 2013-01-04 16:36:47,050 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:47,050 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:47,052 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:47,500 - root - INFO - command executed successfully 2013-01-04 16:36:47,501 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:36:47,503 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.3 finished 2013-01-04 16:36:56,036 - root - INFO - command executed successfully 2013-01-04 16:36:56,037 - root - INFO - 1d0 2013-01-04 16:36:56,039 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:56,319 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:56,432 - root - INFO - command executed successfully 2013-01-04 16:36:56,433 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:56,544 - root - INFO - command executed successfully 2013-01-04 16:36:56,545 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:56,632 - root - INFO - command executed successfully 2013-01-04 16:36:56,633 - root - INFO - running command.raw hostname 2013-01-04 16:36:56,719 - root - INFO - command executed successfully 2013-01-04 16:36:56,723 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:56,830 - root - INFO - command executed successfully 2013-01-04 16:36:57,103 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:57,188 - root - INFO - command executed successfully 2013-01-04 16:36:57,189 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:57,299 - root - INFO - command executed successfully 2013-01-04 16:36:57,301 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:57,412 - root - INFO - command executed successfully 2013-01-04 16:36:57,414 - root - INFO - running command.raw hostname 2013-01-04 16:36:57,524 - root - INFO - command executed successfully 2013-01-04 16:36:57,528 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:57,636 - root - INFO - command executed successfully 2013-01-04 16:36:57,963 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:58,048 - root - INFO - command executed successfully 2013-01-04 16:36:58,049 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:58,160 - root - INFO - command executed successfully 2013-01-04 16:36:58,161 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:58,271 - root - INFO - command executed successfully 2013-01-04 16:36:58,272 - root - INFO - running command.raw hostname 2013-01-04 16:36:58,383 - root - INFO - command executed successfully 2013-01-04 16:36:58,386 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:58,386 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:58,387 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:58,479 - root - INFO - command executed successfully 2013-01-04 16:36:58,480 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:36:58,481 - root - INFO - 2013-01-04 16:36:58,482 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:36:58,482 - root - INFO - Please browse to http://kiwi-r105:8091/ to configure your server. 2013-01-04 16:36:58,483 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:36:58,484 - root - INFO - 2013-01-04 16:36:58,485 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:36:58,486 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:36:58,486 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:36:58,486 - root - INFO - 2013-01-04 16:36:58,487 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:36:58,488 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:36:58,488 - root - INFO - 2013-01-04 16:36:58,489 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:36:58,569 - root - INFO - command executed successfully 2013-01-04 16:36:58,571 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:36:58,685 - root - INFO - command executed successfully 2013-01-04 16:36:58,687 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:37:00,004 - root - INFO - command executed successfully 2013-01-04 16:37:00,005 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:37:00,006 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.6 finished 2013-01-04 16:37:09,052 - root - INFO - command executed successfully 2013-01-04 16:37:09,054 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:37:09,055 - root - INFO - 2013-01-04 16:37:09,056 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:37:09,057 - root - INFO - Please browse to http://kiwi-r104:8091/ to configure your server. 2013-01-04 16:37:09,058 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:37:09,058 - root - INFO - 2013-01-04 16:37:09,059 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:37:09,059 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:37:09,059 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:37:09,060 - root - INFO - 2013-01-04 16:37:09,060 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:37:09,060 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:37:09,061 - root - INFO - 2013-01-04 16:37:09,061 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:37:09,177 - root - INFO - command executed successfully 2013-01-04 16:37:09,179 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:37:09,289 - root - INFO - command executed successfully 2013-01-04 16:37:09,291 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:37:10,642 - root - INFO - command executed successfully 2013-01-04 16:37:10,644 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:37:10,645 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.5 finished 2013-01-04 16:37:15,657 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:37:15,658 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:37:15,658 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:37:16,228 - root - INFO - Connected 2013-01-04 16:37:16,235 - root - INFO - Connected 2013-01-04 16:37:16,430 - root - INFO - Connected 2013-01-04 16:37:19,746 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,749 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,759 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,838 - root - INFO - command executed successfully 2013-01-04 16:37:19,840 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,845 - root - INFO - command executed successfully 2013-01-04 16:37:19,846 - root - INFO - command executed successfully 2013-01-04 16:37:19,849 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,848 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,935 - root - INFO - command executed successfully 2013-01-04 16:37:19,936 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:19,962 - root - INFO - command executed successfully 2013-01-04 16:37:19,964 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:19,968 - root - INFO - command executed successfully 2013-01-04 16:37:19,969 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:20,047 - root - INFO - command executed successfully 2013-01-04 16:37:20,048 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,078 - root - INFO - command executed successfully 2013-01-04 16:37:20,080 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,081 - root - INFO - command executed successfully 2013-01-04 16:37:20,086 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,163 - root - INFO - command executed successfully 2013-01-04 16:37:20,167 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,198 - root - INFO - command executed successfully 2013-01-04 16:37:20,200 - root - INFO - command executed successfully 2013-01-04 16:37:20,206 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,208 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,288 - root - INFO - command executed successfully 2013-01-04 16:37:20,289 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,299 - root - INFO - command executed successfully 2013-01-04 16:37:20,301 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,303 - root - INFO - command executed successfully 2013-01-04 16:37:20,304 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,377 - root - INFO - command executed successfully 2013-01-04 16:37:20,378 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:20,393 - root - INFO - command executed successfully 2013-01-04 16:37:20,394 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:20,416 - root - INFO - command executed successfully 2013-01-04 16:37:20,417 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:22,371 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:22,375 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:22,510 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:25,374 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,378 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,390 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password 2013-01-04 16:37:25,396 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.3 finished thread initializer-thread-172.27.33.6 finished 2013-01-04 16:37:25,514 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,528 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.5 finished 2013-01-04 16:37:25,607 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:37:25,932 - root - INFO - Connected 2013-01-04 16:37:26,243 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:26,359 - root - INFO - command executed successfully 2013-01-04 16:37:26,360 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:26,474 - root - INFO - command executed successfully 2013-01-04 16:37:26,476 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:26,586 - root - INFO - command executed successfully 2013-01-04 16:37:26,589 - root - INFO - running command.raw hostname 2013-01-04 16:37:26,698 - root - INFO - command executed successfully 2013-01-04 16:37:26,818 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:37:27,142 - root - INFO - Connected 2013-01-04 16:37:27,416 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:27,504 - root - INFO - command executed successfully 2013-01-04 16:37:27,505 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:27,619 - root - INFO - command executed successfully 2013-01-04 16:37:27,620 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:27,734 - root - INFO - command executed successfully 2013-01-04 16:37:27,736 - root - INFO - running command.raw hostname 2013-01-04 16:37:27,849 - root - INFO - command executed successfully 2013-01-04 16:37:27,972 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:37:28,296 - root - INFO - Connected 2013-01-04 16:37:28,614 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:28,698 - root - INFO - command executed successfully 2013-01-04 16:37:28,700 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:28,809 - root - INFO - command executed successfully 2013-01-04 16:37:28,811 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:28,922 - root - INFO - command executed successfully 2013-01-04 16:37:28,924 - root - INFO - running command.raw hostname 2013-01-04 16:37:29,033 - root - INFO - command executed successfully 2013-01-04 16:37:29,156 - root - INFO - Installation of new version is done. Wait for 10 sec for rebalance 2013-01-04 16:51:41,243 - root - INFO - fail_over successful 2013-01-04 16:52:36,258 - root - INFO - adding node 172.27.33.3:8091 to cluster 2013-01-04 16:52:36,258 - root - INFO - adding remote node @172.27.33.3:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:37,575 - root - INFO - adding node 172.27.33.6:8091 to cluster 2013-01-04 16:52:37,576 - root - INFO - adding remote node @172.27.33.6:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:38,548 - root - INFO - adding node 172.27.33.5:8091 to cluster 2013-01-04 16:52:38,549 - root - INFO - adding remote node @172.27.33.5:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:39,763 - root - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5%2Cns_1%40172.27.33.2%2Cns_1%40172.27.33.4 2013-01-04 16:52:39,773 - root - INFO - rebalance operation started 2013-01-04 16:52:39,779 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:52:49,796 - root - INFO - rebalancing was completed with progress: 100% in 10.02257514 sec 2013-01-04 16:53:36,118 - root - INFO - Rebalance in all 2.0 Nodes 2013-01-04 16:53:47,639 - root - INFO - /diag/eval status: True content: 'ns_1@172.27.33.2' command: node(global:whereis_name(ns_orchestrator)) ERROR 2013-01-04 16:55:40,065 - root - INFO - ============== basetestcase cleanup was started for test #1 online_upgrade_rebalance_in_out ============== 2013-01-04 16:55:40,101 - root - INFO - deleting existing buckets on [ip:172.27.33.2 port:8091 ssh_username:root, ip:172.27.33.3 port:8091 ssh_username:root, ip:172.27.33.6 port:8091 ssh_username:root, ip:172.27.33.5 port:8091 ssh_username:root] 2013-01-04 16:55:40,402 - root - INFO - deleted bucket : default from 172.27.33.2 2013-01-04 16:55:40,402 - root - INFO - waiting for bucket deletion to complete.... 2013-01-04 16:55:40,407 - root - INFO - existing buckets : [] 2013-01-04 16:55:40,480 - root - INFO - rebalancing all nodes in order to remove nodes 2013-01-04 16:55:40,485 - root - INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5&user=Administrator&knownNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5%2Cns_1%40172.27.33.2 2013-01-04 16:55:40,491 - root - INFO - rebalance operation started 2013-01-04 16:55:40,499 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:55:44,511 - root - INFO - rebalance progress took 4.02038812637 seconds 2013-01-04 16:55:44,512 - root - INFO - sleep for 4.02038812637 seconds after rebalance... 2013-01-04 16:55:48,574 - root - INFO - removed all the nodes from cluster associated with ip:172.27.33.2 port:8091 ssh_username:root ? [(u'ns_1@172.27.33.6', 8091), (u'ns_1@172.27.33.3', 8091), (u'ns_1@172.27.33.5', 8091)] 2013-01-04 16:55:58,593 - root - INFO - waiting for ns_server @ 172.27.33.2:8091 2013-01-04 16:55:58,602 - root - INFO - ns_server @ 172.27.33.2:8091 is running 2013-01-04 16:55:58,609 - root - INFO - waiting for ns_server @ 172.27.33.3:8091 2013-01-04 16:55:58,616 - root - INFO - ns_server @ 172.27.33.3:8091 is running 2013-01-04 16:55:58,623 - root - INFO - waiting for ns_server @ 172.27.33.6:8091 2013-01-04 16:55:58,629 - root - INFO - ns_server @ 172.27.33.6:8091 is running 2013-01-04 16:55:58,637 - root - INFO - waiting for ns_server @ 172.27.33.5:8091 2013-01-04 16:55:58,644 - root - INFO - ns_server @ 172.27.33.5:8091 is running 2013-01-04 16:55:58,645 - root - INFO - ============== basetestcase cleanup was finished for test #1 online_upgrade_rebalance_in_out ============== ====================================================================== ERROR: online_upgrade_rebalance_in_out (newupgradetests.MultiNodesUpgradeTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/newupgradetests.py", line 195, in online_upgrade_rebalance_in_out self.online_upgrade() File "pytests/newupgradetests.py", line 240, in online_upgrade raise Exception("After rebalance in 2.0 Nodes, 2.0 doesn't become the master") Exception: After rebalance in 2.0 Nodes, 2.0 doesn't become the master ---------------------------------------------------------------------- Ran 1 test in 1788.471s FAILED (errors=1) summary so far suite newupgradetests.MultiNodesUpgradeTests , pass 0 , fail 1 failures so far... newupgradetests.MultiNodesUpgradeTests.online_upgrade_rebalance_in_out testrunner logs, diags and results are available under logs/testrunner-13-Jan-04_16-26-102013-01-04 16:26:20,869 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:26:20,979 - root - INFO - command executed successfully 2013-01-04 16:26:20,981 - root - INFO - running command.raw hostname 2013-01-04 16:26:20,985 - root - INFO - command executed successfully 2013-01-04 16:26:20,986 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:26:21,096 - root - INFO - command executed successfully 2013-01-04 16:26:21,102 - root - INFO - command executed successfully 2013-01-04 16:26:21,103 - root - INFO - running command.raw hostname 2013-01-04 16:26:21,194 - root - INFO - command executed successfully 2013-01-04 16:27:02,379 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:27:02,704 - root - INFO - Connected 2013-01-04 16:27:03,633 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:27:03,796 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:03,995 - root - INFO - Connected 2013-01-04 16:27:04,026 - root - INFO - command executed successfully 2013-01-04 16:27:04,028 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,138 - root - INFO - command executed successfully 2013-01-04 16:27:04,139 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,233 - root - INFO - command executed successfully 2013-01-04 16:27:04,235 - root - INFO - running command.raw hostname 2013-01-04 16:27:04,307 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:04,354 - root - INFO - command executed successfully 2013-01-04 16:27:04,418 - root - INFO - command executed successfully 2013-01-04 16:27:04,420 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,548 - root - INFO - command executed successfully 2013-01-04 16:27:04,549 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,635 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:04,660 - root - INFO - command executed successfully 2013-01-04 16:27:04,662 - root - INFO - running command.raw hostname 2013-01-04 16:27:04,720 - root - INFO - command executed successfully 2013-01-04 16:27:04,721 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:04,772 - root - INFO - command executed successfully 2013-01-04 16:27:04,805 - root - INFO - command executed successfully 2013-01-04 16:27:04,806 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:04,919 - root - INFO - command executed successfully 2013-01-04 16:27:04,921 - root - INFO - running command.raw hostname 2013-01-04 16:27:05,036 - root - INFO - command executed successfully 2013-01-04 16:27:05,040 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:27:05,059 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:27:05,145 - root - INFO - command executed successfully 2013-01-04 16:27:05,147 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:27:05,160 - root - INFO - command executed successfully 2013-01-04 16:27:05,162 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm': No such file or directory 2013-01-04 16:27:05,162 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:27:05,163 - root - ERROR - rm: cannot remove `cb_1275': Is a directory 2013-01-04 16:27:05,164 - root - ERROR - mv: cannot stat `cb_1275/*': No such file or directory 2013-01-04 16:27:05,165 - root - INFO - get md5 sum for local and remote 2013-01-04 16:27:05,166 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm > couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l 2013-01-04 16:27:05,260 - root - INFO - command executed successfully 2013-01-04 16:27:05,262 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:27:05,372 - root - INFO - command executed successfully 2013-01-04 16:27:05,373 - root - INFO - running command.raw hostname 2013-01-04 16:27:05,483 - root - INFO - command executed successfully 2013-01-04 16:27:05,487 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:27:05,604 - root - INFO - command executed successfully 2013-01-04 16:27:05,606 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm': No such file or directory 2013-01-04 16:27:05,607 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:27:05,608 - root - ERROR - rm: cannot remove `cb_0006': Is a directory 2013-01-04 16:27:05,609 - root - ERROR - rm: cannot remove `default': Is a directory 2013-01-04 16:27:05,609 - root - ERROR - mv: cannot stat `cb_0006/*': No such file or directory 2013-01-04 16:27:05,610 - root - INFO - get md5 sum for local and remote 2013-01-04 16:27:05,610 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm > couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l 2013-01-04 16:27:05,773 - root - INFO - command executed successfully 2013-01-04 16:27:05,774 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm: No such file or directory 2013-01-04 16:27:05,775 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:27:05,777 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:27:06,209 - root - INFO - command executed successfully 2013-01-04 16:27:06,210 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm: No such file or directory 2013-01-04 16:27:06,211 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:27:06,212 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:30:09,539 - root - INFO - command executed successfully 2013-01-04 16:30:09,540 - root - INFO - 1d0 2013-01-04 16:30:09,541 - root - INFO - < 4fceddd71c9a4468940d6b070fcda56a couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:09,822 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:09,933 - root - INFO - command executed successfully 2013-01-04 16:30:09,934 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:10,043 - root - INFO - command executed successfully 2013-01-04 16:30:10,046 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:10,156 - root - INFO - command executed successfully 2013-01-04 16:30:10,158 - root - INFO - running command.raw hostname 2013-01-04 16:30:10,267 - root - INFO - command executed successfully 2013-01-04 16:30:10,271 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:30:10,382 - root - INFO - command executed successfully 2013-01-04 16:30:10,654 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:10,738 - root - INFO - command executed successfully 2013-01-04 16:30:10,740 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:10,851 - root - INFO - command executed successfully 2013-01-04 16:30:10,853 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:10,962 - root - INFO - command executed successfully 2013-01-04 16:30:10,964 - root - INFO - running command.raw hostname 2013-01-04 16:30:11,074 - root - INFO - command executed successfully 2013-01-04 16:30:11,078 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:30:11,185 - root - INFO - command executed successfully 2013-01-04 16:30:11,515 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:11,624 - root - INFO - command executed successfully 2013-01-04 16:30:11,626 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:11,736 - root - INFO - command executed successfully 2013-01-04 16:30:11,738 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:11,849 - root - INFO - command executed successfully 2013-01-04 16:30:11,851 - root - INFO - running command.raw hostname 2013-01-04 16:30:11,962 - root - INFO - command executed successfully 2013-01-04 16:30:11,966 - root - INFO - deliverable_type : rpm 2013-01-04 16:30:11,966 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:30:11,966 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:22,831 - root - INFO - command executed successfully 2013-01-04 16:30:22,832 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:30:22,833 - root - INFO - 2013-01-04 16:30:22,834 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:30:22,835 - root - INFO - Please browse to http://kiwi-r101:8091/ to configure your server. 2013-01-04 16:30:22,836 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:30:22,836 - root - INFO - 2013-01-04 16:30:22,837 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:30:22,837 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:30:22,838 - root - INFO - 8091 and from 21100 to 21299. 2013-01-04 16:30:22,838 - root - INFO - 2013-01-04 16:30:22,838 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:30:22,839 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:30:22,839 - root - INFO - 2013-01-04 16:30:22,839 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:30:22,923 - root - INFO - command executed successfully 2013-01-04 16:30:22,925 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:30:23,035 - root - INFO - command executed successfully 2013-01-04 16:30:23,037 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:30:24,392 - root - INFO - command executed successfully 2013-01-04 16:30:24,393 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:30:24,394 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.2 finished 2013-01-04 16:30:31,632 - root - INFO - command executed successfully 2013-01-04 16:30:31,633 - root - INFO - 1d0 2013-01-04 16:30:31,635 - root - INFO - < 4fceddd71c9a4468940d6b070fcda56a couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:31,912 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:32,022 - root - INFO - command executed successfully 2013-01-04 16:30:32,024 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:32,134 - root - INFO - command executed successfully 2013-01-04 16:30:32,135 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:32,246 - root - INFO - command executed successfully 2013-01-04 16:30:32,248 - root - INFO - running command.raw hostname 2013-01-04 16:30:32,358 - root - INFO - command executed successfully 2013-01-04 16:30:32,361 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:30:32,471 - root - INFO - command executed successfully 2013-01-04 16:30:32,747 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:32,858 - root - INFO - command executed successfully 2013-01-04 16:30:32,859 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:32,969 - root - INFO - command executed successfully 2013-01-04 16:30:32,971 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:33,056 - root - INFO - command executed successfully 2013-01-04 16:30:33,058 - root - INFO - running command.raw hostname 2013-01-04 16:30:33,168 - root - INFO - command executed successfully 2013-01-04 16:30:33,172 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:30:33,279 - root - INFO - command executed successfully 2013-01-04 16:30:33,667 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:33,777 - root - INFO - command executed successfully 2013-01-04 16:30:33,779 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:33,889 - root - INFO - command executed successfully 2013-01-04 16:30:33,892 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:34,002 - root - INFO - command executed successfully 2013-01-04 16:30:34,003 - root - INFO - running command.raw hostname 2013-01-04 16:30:34,113 - root - INFO - command executed successfully 2013-01-04 16:30:34,117 - root - INFO - deliverable_type : rpm 2013-01-04 16:30:34,117 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:30:34,118 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.0-1978-rel.rpm 2013-01-04 16:30:46,417 - root - INFO - command executed successfully 2013-01-04 16:30:46,419 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:30:46,420 - root - INFO - 2013-01-04 16:30:46,422 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:30:46,422 - root - INFO - Please browse to http://kiwi-r103:8091/ to configure your server. 2013-01-04 16:30:46,423 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:30:46,423 - root - INFO - 2013-01-04 16:30:46,423 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:30:46,424 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:30:46,424 - root - INFO - 8091 and from 21100 to 21299. 2013-01-04 16:30:46,424 - root - INFO - 2013-01-04 16:30:46,425 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:30:46,425 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:30:46,426 - root - INFO - 2013-01-04 16:30:46,426 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:30:46,542 - root - INFO - command executed successfully 2013-01-04 16:30:46,544 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:30:46,653 - root - INFO - command executed successfully 2013-01-04 16:30:46,655 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:30:47,985 - root - INFO - command executed successfully 2013-01-04 16:30:47,986 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:30:47,988 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.4 finished 2013-01-04 16:30:52,999 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:30:53,000 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:30:53,551 - root - INFO - Connected lib/membase/api/rest_client.py:186: DeprecationWarning: object.__new__() takes no parameters obj = object.__new__(self, serverInfo) 2013-01-04 16:30:53,577 - root - INFO - Connected 2013-01-04 16:30:56,803 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:56,825 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:30:56,889 - root - INFO - command executed successfully 2013-01-04 16:30:56,890 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:56,917 - root - INFO - command executed successfully 2013-01-04 16:30:56,919 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:30:57,001 - root - INFO - command executed successfully 2013-01-04 16:30:57,002 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:57,029 - root - INFO - command executed successfully 2013-01-04 16:30:57,031 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:30:57,114 - root - INFO - command executed successfully 2013-01-04 16:30:57,115 - root - INFO - running command.raw hostname 2013-01-04 16:30:57,140 - root - INFO - command executed successfully 2013-01-04 16:30:57,141 - root - INFO - running command.raw hostname 2013-01-04 16:30:57,225 - root - INFO - command executed successfully 2013-01-04 16:30:57,228 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:30:57,252 - root - INFO - command executed successfully 2013-01-04 16:30:57,256 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:30:57,354 - root - INFO - command executed successfully 2013-01-04 16:30:57,355 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:30:57,387 - root - INFO - command executed successfully 2013-01-04 16:30:57,387 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:30:57,469 - root - INFO - command executed successfully 2013-01-04 16:30:57,471 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:30:57,498 - root - INFO - command executed successfully 2013-01-04 16:30:57,500 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:30:59,348 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:30:59,755 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:31:02,351 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:02,365 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.2 finished 2013-01-04 16:31:02,757 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:02,770 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.4 finished 2013-01-04 16:31:02,813 - root - INFO - connecting to 172.27.33.2 with username : root password : couchbase ssh_key: 2013-01-04 16:31:03,134 - root - INFO - Connected 2013-01-04 16:31:03,453 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:03,563 - root - INFO - command executed successfully 2013-01-04 16:31:03,565 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:03,675 - root - INFO - command executed successfully 2013-01-04 16:31:03,677 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:03,787 - root - INFO - command executed successfully 2013-01-04 16:31:03,789 - root - INFO - running command.raw hostname 2013-01-04 16:31:03,899 - root - INFO - command executed successfully 2013-01-04 16:31:04,021 - root - INFO - connecting to 172.27.33.4 with username : root password : couchbase ssh_key: 2013-01-04 16:31:04,344 - root - INFO - Connected 2013-01-04 16:31:04,655 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:04,766 - root - INFO - command executed successfully 2013-01-04 16:31:04,767 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:04,853 - root - INFO - command executed successfully 2013-01-04 16:31:04,854 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:04,965 - root - INFO - command executed successfully 2013-01-04 16:31:04,966 - root - INFO - running command.raw hostname 2013-01-04 16:31:05,076 - root - INFO - command executed successfully 2013-01-04 16:31:05,206 - root - INFO - Installation of old version is done. Wait for 10 sec for upgrade 2013-01-04 16:31:05,382 - root - INFO - adding node 172.27.33.4:8091 to cluster 2013-01-04 16:31:05,382 - root - INFO - adding remote node @172.27.33.4:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:31:05,913 - root - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.27.33.2%2Cns_1%40172.27.33.4 2013-01-04 16:31:05,918 - root - INFO - rebalance operation started 2013-01-04 16:31:05,927 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:31:15,943 - root - INFO - rebalancing was completed with progress: 100% in 10.0244619846 sec 2013-01-04 16:31:16,953 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:16,964 - root - INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password 2013-01-04 16:31:16,988 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:31:16,999 - root - INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password 2013-01-04 16:31:18,012 - root - INFO - http://172.27.33.2:8091/pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=None&flushEnabled=1&replicaNumber=1&ramQuotaMB=1456 2013-01-04 16:31:18,026 - root - INFO - 0.0136511325836 seconds to create bucket default 2013-01-04 16:31:18,026 - root - INFO - waiting for memcached bucket : default in 172.27.33.2 to accept set ops 2013-01-04 16:31:19,215 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,309 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:19,439 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,525 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:19,666 - root - INFO - bucket 'default' was created with per node RAM quota: 1456 2013-01-04 16:31:19,734 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:19,817 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:20,669 - root - INFO - Batch create documents done #: 0 2013-01-04 16:31:21,093 - root - INFO - Batch create documents done #: 1000 2013-01-04 16:31:21,687 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:21,759 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.27.33.2:8091' 2013-01-04 16:31:21,773 - root - INFO - creating direct client 172.27.33.2:11210 default 2013-01-04 16:31:21,852 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '172.27.33.2:8091' 2013-01-04 16:31:21,867 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:21,944 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.27.33.4:8091' 2013-01-04 16:31:21,961 - root - INFO - creating direct client 172.27.33.4:11210 default 2013-01-04 16:31:22,038 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '172.27.33.4:8091' 2013-01-04 16:31:32,051 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,052 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,053 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:31:32,813 - root - INFO - Connected 2013-01-04 16:31:32,815 - root - INFO - Connected 2013-01-04 16:31:32,815 - root - INFO - Connected 2013-01-04 16:31:33,168 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,172 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,196 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,255 - root - INFO - command executed successfully 2013-01-04 16:31:33,257 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,261 - root - INFO - command executed successfully 2013-01-04 16:31:33,262 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,312 - root - INFO - command executed successfully 2013-01-04 16:31:33,314 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:33,367 - root - INFO - command executed successfully 2013-01-04 16:31:33,369 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,375 - root - INFO - command executed successfully 2013-01-04 16:31:33,376 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,425 - root - INFO - command executed successfully 2013-01-04 16:31:33,426 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:33,466 - root - INFO - command executed successfully 2013-01-04 16:31:33,467 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,481 - root - INFO - command executed successfully 2013-01-04 16:31:33,482 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,514 - root - INFO - command executed successfully 2013-01-04 16:31:33,515 - root - INFO - running command.raw hostname 2013-01-04 16:31:33,583 - root - INFO - command executed successfully 2013-01-04 16:31:33,588 - root - INFO - CentOS 2013-01-04 16:31:33,589 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,591 - root - INFO - command executed successfully 2013-01-04 16:31:33,595 - root - INFO - CentOS 2013-01-04 16:31:33,596 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,627 - root - INFO - command executed successfully 2013-01-04 16:31:33,630 - root - INFO - CentOS 2013-01-04 16:31:33,631 - root - INFO - running rpm -e to remove membase-server 2013-01-04 16:31:33,991 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:33,996 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:34,008 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:34,080 - root - INFO - command executed successfully 2013-01-04 16:31:34,081 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,084 - root - INFO - command executed successfully 2013-01-04 16:31:34,085 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,123 - root - INFO - command executed successfully 2013-01-04 16:31:34,124 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:34,172 - root - INFO - command executed successfully 2013-01-04 16:31:34,173 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,175 - root - INFO - command executed successfully 2013-01-04 16:31:34,176 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,237 - root - INFO - command executed successfully 2013-01-04 16:31:34,239 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:34,265 - root - INFO - command executed successfully 2013-01-04 16:31:34,266 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,268 - root - INFO - command executed successfully 2013-01-04 16:31:34,269 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,351 - root - INFO - command executed successfully 2013-01-04 16:31:34,357 - root - INFO - command executed successfully 2013-01-04 16:31:34,358 - root - INFO - running command.raw hostname 2013-01-04 16:31:34,359 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,370 - root - INFO - command executed successfully 2013-01-04 16:31:34,373 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,459 - root - INFO - command executed successfully 2013-01-04 16:31:34,461 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,461 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,465 - root - INFO - command executed successfully 2013-01-04 16:31:34,469 - root - INFO - running command.raw rpm -e membase-server 2013-01-04 16:31:34,502 - root - INFO - command executed successfully 2013-01-04 16:31:34,504 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,505 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,574 - root - INFO - command executed successfully 2013-01-04 16:31:34,576 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,577 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,583 - root - INFO - command executed successfully 2013-01-04 16:31:34,584 - root - ERROR - error: package membase-server is not installed 2013-01-04 16:31:34,585 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:34,617 - root - INFO - command executed successfully 2013-01-04 16:31:34,618 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,619 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,689 - root - INFO - command executed successfully 2013-01-04 16:31:34,691 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,699 - root - INFO - command executed successfully 2013-01-04 16:31:34,700 - root - ERROR - beam: no process killed 2013-01-04 16:31:34,701 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:34,707 - root - INFO - command executed successfully 2013-01-04 16:31:34,708 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,800 - root - INFO - command executed successfully 2013-01-04 16:31:34,801 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,813 - root - INFO - command executed successfully 2013-01-04 16:31:34,814 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:34,820 - root - INFO - command executed successfully 2013-01-04 16:31:34,821 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,911 - root - INFO - command executed successfully 2013-01-04 16:31:34,912 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:34,914 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:34,924 - root - INFO - command executed successfully 2013-01-04 16:31:34,925 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:34,936 - root - INFO - command executed successfully 2013-01-04 16:31:34,937 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:34,938 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:35,015 - root - INFO - command executed successfully 2013-01-04 16:31:35,016 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:35,018 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:35,025 - root - INFO - command executed successfully 2013-01-04 16:31:35,026 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,027 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,053 - root - INFO - command executed successfully 2013-01-04 16:31:35,054 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,055 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,130 - root - INFO - command executed successfully 2013-01-04 16:31:35,131 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:35,132 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:35,140 - root - INFO - command executed successfully 2013-01-04 16:31:35,142 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,169 - root - INFO - command executed successfully 2013-01-04 16:31:35,170 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,219 - root - INFO - command executed successfully 2013-01-04 16:31:35,221 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:35,251 - root - INFO - command executed successfully 2013-01-04 16:31:35,252 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,281 - root - INFO - command executed successfully 2013-01-04 16:31:35,282 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,337 - root - INFO - command executed successfully 2013-01-04 16:31:35,338 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:35,362 - root - INFO - command executed successfully 2013-01-04 16:31:35,364 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,394 - root - INFO - command executed successfully 2013-01-04 16:31:35,395 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,454 - root - INFO - command executed successfully 2013-01-04 16:31:35,456 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:35,475 - root - INFO - command executed successfully 2013-01-04 16:31:35,476 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,511 - root - INFO - command executed successfully 2013-01-04 16:31:35,512 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,570 - root - INFO - command executed successfully 2013-01-04 16:31:35,571 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:35,587 - root - INFO - command executed successfully 2013-01-04 16:31:35,588 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,602 - root - INFO - command executed successfully 2013-01-04 16:31:35,603 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,686 - root - INFO - command executed successfully 2013-01-04 16:31:35,688 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:35,698 - root - INFO - command executed successfully 2013-01-04 16:31:35,718 - root - INFO - command executed successfully 2013-01-04 16:31:35,815 - root - INFO - command executed successfully 2013-01-04 16:31:36,006 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,074 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,089 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:36,095 - root - INFO - command executed successfully 2013-01-04 16:31:36,097 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,180 - root - INFO - command executed successfully 2013-01-04 16:31:36,181 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,189 - root - INFO - command executed successfully 2013-01-04 16:31:36,189 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:36,207 - root - INFO - command executed successfully 2013-01-04 16:31:36,208 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,271 - root - INFO - command executed successfully 2013-01-04 16:31:36,272 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,281 - root - INFO - command executed successfully 2013-01-04 16:31:36,283 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:36,318 - root - INFO - command executed successfully 2013-01-04 16:31:36,319 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,385 - root - INFO - command executed successfully 2013-01-04 16:31:36,388 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,395 - root - INFO - command executed successfully 2013-01-04 16:31:36,396 - root - INFO - running command.raw hostname 2013-01-04 16:31:36,430 - root - INFO - command executed successfully 2013-01-04 16:31:36,434 - root - INFO - CentOS 2013-01-04 16:31:36,435 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,435 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:36,504 - root - INFO - command executed successfully 2013-01-04 16:31:36,509 - root - INFO - CentOS 2013-01-04 16:31:36,509 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,509 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:36,512 - root - INFO - command executed successfully 2013-01-04 16:31:36,515 - root - INFO - CentOS 2013-01-04 16:31:36,516 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:36,517 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:39,326 - root - INFO - command executed successfully 2013-01-04 16:31:39,327 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,328 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,329 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,330 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,331 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,332 - root - INFO - 2013-01-04 16:31:39,332 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,341 - root - INFO - command executed successfully 2013-01-04 16:31:39,342 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,342 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,343 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,343 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,344 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,345 - root - INFO - 2013-01-04 16:31:39,346 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,381 - root - INFO - command executed successfully 2013-01-04 16:31:39,382 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-01-04 16:31:39,382 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-01-04 16:31:39,384 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-01-04 16:31:39,386 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-01-04 16:31:39,386 - root - INFO - {badrpc,nodedown} 2013-01-04 16:31:39,387 - root - INFO - 2013-01-04 16:31:39,387 - root - INFO - running command.raw killall -9 beam.smp 2013-01-04 16:31:39,447 - root - INFO - command executed successfully 2013-01-04 16:31:39,449 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,458 - root - INFO - command executed successfully 2013-01-04 16:31:39,460 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,502 - root - INFO - command executed successfully 2013-01-04 16:31:39,504 - root - INFO - running command.raw killall -9 memcached 2013-01-04 16:31:39,559 - root - INFO - command executed successfully 2013-01-04 16:31:39,561 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,571 - root - INFO - command executed successfully 2013-01-04 16:31:39,573 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,613 - root - INFO - command executed successfully 2013-01-04 16:31:39,614 - root - INFO - running command.raw killall -9 moxi 2013-01-04 16:31:39,674 - root - INFO - command executed successfully 2013-01-04 16:31:39,676 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,677 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,683 - root - INFO - command executed successfully 2013-01-04 16:31:39,685 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,686 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,727 - root - INFO - command executed successfully 2013-01-04 16:31:39,729 - root - ERROR - moxi: no process killed 2013-01-04 16:31:39,730 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-01-04 16:31:39,787 - root - INFO - command executed successfully 2013-01-04 16:31:39,788 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,790 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,796 - root - INFO - command executed successfully 2013-01-04 16:31:39,798 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,799 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,842 - root - INFO - command executed successfully 2013-01-04 16:31:39,843 - root - ERROR - vbucketmigrator: no process killed 2013-01-04 16:31:39,844 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:39,905 - root - INFO - command executed successfully 2013-01-04 16:31:39,907 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,908 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:39,909 - root - INFO - command executed successfully 2013-01-04 16:31:39,912 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,912 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:39,962 - root - INFO - command executed successfully 2013-01-04 16:31:39,963 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:39,964 - root - INFO - running command.raw killall -9 epmd 2013-01-04 16:31:40,024 - root - INFO - command executed successfully 2013-01-04 16:31:40,026 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,029 - root - INFO - command executed successfully 2013-01-04 16:31:40,031 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,082 - root - INFO - command executed successfully 2013-01-04 16:31:40,083 - root - INFO - running command.raw killall -9 memsup 2013-01-04 16:31:40,137 - root - INFO - command executed successfully 2013-01-04 16:31:40,138 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,140 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,143 - root - INFO - command executed successfully 2013-01-04 16:31:40,144 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,145 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,195 - root - INFO - command executed successfully 2013-01-04 16:31:40,196 - root - ERROR - memsup: no process killed 2013-01-04 16:31:40,196 - root - INFO - running command.raw killall -9 cpu_sup 2013-01-04 16:31:40,253 - root - INFO - command executed successfully 2013-01-04 16:31:40,254 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,255 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,260 - root - INFO - command executed successfully 2013-01-04 16:31:40,261 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,263 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,307 - root - INFO - command executed successfully 2013-01-04 16:31:40,308 - root - ERROR - cpu_sup: no process killed 2013-01-04 16:31:40,309 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-01-04 16:31:40,367 - root - INFO - command executed successfully 2013-01-04 16:31:40,368 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,377 - root - INFO - command executed successfully 2013-01-04 16:31:40,379 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,424 - root - INFO - command executed successfully 2013-01-04 16:31:40,425 - root - INFO - running command.raw rm -rf /opt/membase 2013-01-04 16:31:40,481 - root - INFO - command executed successfully 2013-01-04 16:31:40,483 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,496 - root - INFO - command executed successfully 2013-01-04 16:31:40,497 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,543 - root - INFO - command executed successfully 2013-01-04 16:31:40,544 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-01-04 16:31:40,580 - root - INFO - command executed successfully 2013-01-04 16:31:40,581 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,593 - root - INFO - command executed successfully 2013-01-04 16:31:40,594 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,659 - root - INFO - command executed successfully 2013-01-04 16:31:40,660 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-01-04 16:31:40,697 - root - INFO - command executed successfully 2013-01-04 16:31:40,699 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,707 - root - INFO - command executed successfully 2013-01-04 16:31:40,708 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,777 - root - INFO - command executed successfully 2013-01-04 16:31:40,778 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-01-04 16:31:40,815 - root - INFO - command executed successfully 2013-01-04 16:31:40,817 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,821 - root - INFO - command executed successfully 2013-01-04 16:31:40,823 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,892 - root - INFO - command executed successfully 2013-01-04 16:31:40,894 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:40,934 - root - INFO - command executed successfully 2013-01-04 16:31:40,937 - root - INFO - command executed successfully 2013-01-04 16:31:41,008 - root - INFO - command executed successfully 2013-01-04 16:31:41,317 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,321 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,345 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:41,402 - root - INFO - command executed successfully 2013-01-04 16:31:41,404 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,437 - root - INFO - command executed successfully 2013-01-04 16:31:41,439 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,459 - root - INFO - command executed successfully 2013-01-04 16:31:41,461 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:41,515 - root - INFO - command executed successfully 2013-01-04 16:31:41,517 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,553 - root - INFO - command executed successfully 2013-01-04 16:31:41,555 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,575 - root - INFO - command executed successfully 2013-01-04 16:31:41,576 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:41,628 - root - INFO - command executed successfully 2013-01-04 16:31:41,629 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,669 - root - INFO - command executed successfully 2013-01-04 16:31:41,672 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,693 - root - INFO - command executed successfully 2013-01-04 16:31:41,694 - root - INFO - running command.raw hostname 2013-01-04 16:31:41,741 - root - INFO - command executed successfully 2013-01-04 16:31:41,745 - root - INFO - CentOS 2013-01-04 16:31:41,746 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,746 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,786 - root - INFO - command executed successfully 2013-01-04 16:31:41,790 - root - INFO - CentOS 2013-01-04 16:31:41,790 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,791 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,809 - root - INFO - command executed successfully 2013-01-04 16:31:41,811 - root - INFO - CentOS 2013-01-04 16:31:41,812 - root - INFO - running rpm -e to remove couchbase-server 2013-01-04 16:31:41,813 - root - INFO - running command.raw rpm -e couchbase-server 2013-01-04 16:31:41,869 - root - INFO - command executed successfully 2013-01-04 16:31:41,871 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,872 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,909 - root - INFO - command executed successfully 2013-01-04 16:31:41,911 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,912 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,929 - root - INFO - command executed successfully 2013-01-04 16:31:41,931 - root - ERROR - error: package couchbase-server is not installed 2013-01-04 16:31:41,932 - root - INFO - running command.raw killall -9 beam 2013-01-04 16:31:41,980 - root - INFO - command executed successfully 2013-01-04 16:31:41,982 - root - ERROR - beam: no process killed 2013-01-04 16:31:41,983 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,027 - root - INFO - command executed successfully 2013-01-04 16:31:42,029 - root - ERROR - beam: no process killed 2013-01-04 16:31:42,030 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,046 - root - INFO - command executed successfully 2013-01-04 16:31:42,047 - root - ERROR - beam: no process killed 2013-01-04 16:31:42,049 - root - INFO - running command.raw killall -9 couchdb 2013-01-04 16:31:42,094 - root - INFO - command executed successfully 2013-01-04 16:31:42,096 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,097 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,147 - root - INFO - command executed successfully 2013-01-04 16:31:42,149 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,149 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,159 - root - INFO - command executed successfully 2013-01-04 16:31:42,160 - root - ERROR - couchdb: no process killed 2013-01-04 16:31:42,161 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-01-04 16:31:42,208 - root - INFO - command executed successfully 2013-01-04 16:31:42,263 - root - INFO - command executed successfully 2013-01-04 16:31:42,276 - root - INFO - command executed successfully thread uninstaller-thread-172.27.33.3 finished thread uninstaller-thread-172.27.33.6 finished thread uninstaller-thread-172.27.33.5 finished 2013-01-04 16:31:42,278 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,279 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,279 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:31:42,852 - root - INFO - Connected 2013-01-04 16:31:42,853 - root - INFO - Connected 2013-01-04 16:31:43,045 - root - INFO - Connected 2013-01-04 16:31:43,351 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,371 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,418 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:31:43,453 - root - INFO - command executed successfully 2013-01-04 16:31:43,455 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,462 - root - INFO - command executed successfully 2013-01-04 16:31:43,463 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,505 - root - INFO - command executed successfully 2013-01-04 16:31:43,507 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:31:43,568 - root - INFO - command executed successfully 2013-01-04 16:31:43,569 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,580 - root - INFO - command executed successfully 2013-01-04 16:31:43,581 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,622 - root - INFO - command executed successfully 2013-01-04 16:31:43,623 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:31:43,680 - root - INFO - command executed successfully 2013-01-04 16:31:43,681 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,699 - root - INFO - command executed successfully 2013-01-04 16:31:43,700 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,738 - root - INFO - command executed successfully 2013-01-04 16:31:43,739 - root - INFO - running command.raw hostname 2013-01-04 16:31:43,787 - root - INFO - command executed successfully 2013-01-04 16:31:43,793 - root - INFO - command executed successfully 2013-01-04 16:31:43,828 - root - INFO - command executed successfully 2013-01-04 16:32:46,827 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:32:47,158 - root - INFO - Connected 2013-01-04 16:32:47,469 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:48,298 - root - INFO - command executed successfully 2013-01-04 16:32:48,325 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:48,804 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:32:48,819 - root - INFO - command executed successfully 2013-01-04 16:32:48,845 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:49,056 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:32:49,172 - root - INFO - command executed successfully 2013-01-04 16:32:49,173 - root - INFO - running command.raw hostname 2013-01-04 16:32:49,380 - root - INFO - Connected 2013-01-04 16:32:49,567 - root - INFO - command executed successfully 2013-01-04 16:32:49,581 - root - INFO - Connected 2013-01-04 16:32:49,923 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:49,930 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:49,937 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,013 - root - INFO - command executed successfully 2013-01-04 16:32:50,014 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,039 - root - INFO - command executed successfully 2013-01-04 16:32:50,040 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,053 - root - INFO - command executed successfully 2013-01-04 16:32:50,054 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,125 - root - INFO - command executed successfully 2013-01-04 16:32:50,126 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,153 - root - INFO - command executed successfully 2013-01-04 16:32:50,155 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,166 - root - INFO - command executed successfully 2013-01-04 16:32:50,167 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,213 - root - INFO - command executed successfully 2013-01-04 16:32:50,215 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,270 - root - INFO - command executed successfully 2013-01-04 16:32:50,271 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,283 - root - INFO - command executed successfully 2013-01-04 16:32:50,285 - root - INFO - running command.raw hostname 2013-01-04 16:32:50,325 - root - INFO - command executed successfully 2013-01-04 16:32:50,330 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:50,384 - root - INFO - command executed successfully 2013-01-04 16:32:50,402 - root - INFO - command executed successfully 2013-01-04 16:32:50,450 - root - INFO - command executed successfully 2013-01-04 16:32:50,452 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:50,453 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:50,454 - root - ERROR - rm: cannot remove `cb_8077': Is a directory 2013-01-04 16:32:50,455 - root - ERROR - mv: cannot stat `cb_8077/*': No such file or directory 2013-01-04 16:32:50,456 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:50,457 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:50,702 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,716 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:32:50,804 - root - INFO - command executed successfully 2013-01-04 16:32:50,805 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,814 - root - INFO - command executed successfully 2013-01-04 16:32:50,816 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:32:50,920 - root - INFO - command executed successfully 2013-01-04 16:32:50,922 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:50,928 - root - INFO - command executed successfully 2013-01-04 16:32:50,930 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:32:51,038 - root - INFO - command executed successfully 2013-01-04 16:32:51,040 - root - INFO - running command.raw hostname 2013-01-04 16:32:51,043 - root - INFO - command executed successfully 2013-01-04 16:32:51,045 - root - INFO - running command.raw hostname 2013-01-04 16:32:51,059 - root - INFO - command executed successfully 2013-01-04 16:32:51,060 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,062 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,063 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:32:51,157 - root - INFO - command executed successfully 2013-01-04 16:32:51,161 - root - INFO - command executed successfully 2013-01-04 16:32:51,165 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:51,168 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-01-04 16:32:51,254 - root - INFO - command executed successfully 2013-01-04 16:32:51,256 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:51,257 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:51,258 - root - ERROR - rm: cannot remove `cb_1951': Is a directory 2013-01-04 16:32:51,258 - root - ERROR - mv: cannot stat `cb_1951/*': No such file or directory 2013-01-04 16:32:51,259 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:51,259 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:51,287 - root - INFO - command executed successfully 2013-01-04 16:32:51,289 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm': No such file or directory 2013-01-04 16:32:51,290 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-01-04 16:32:51,291 - root - ERROR - rm: cannot remove `cb_1410': Is a directory 2013-01-04 16:32:51,291 - root - ERROR - mv: cannot stat `cb_1410/*': No such file or directory 2013-01-04 16:32:51,292 - root - INFO - get md5 sum for local and remote 2013-01-04 16:32:51,292 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm > couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l 2013-01-04 16:32:51,842 - root - INFO - command executed successfully 2013-01-04 16:32:51,843 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,845 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,846 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:32:51,869 - root - INFO - command executed successfully 2013-01-04 16:32:51,870 - root - ERROR - md5sum: couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm: No such file or directory 2013-01-04 16:32:51,872 - root - INFO - comparing md5 sum and downloading if needed 2013-01-04 16:32:51,872 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5 couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm;rm -f *.md5 *.md5l 2013-01-04 16:36:32,684 - root - INFO - command executed successfully 2013-01-04 16:36:32,686 - root - INFO - 1d0 2013-01-04 16:36:32,687 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:32,951 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:33,068 - root - INFO - command executed successfully 2013-01-04 16:36:33,069 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:33,184 - root - INFO - command executed successfully 2013-01-04 16:36:33,186 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:33,300 - root - INFO - command executed successfully 2013-01-04 16:36:33,302 - root - INFO - running command.raw hostname 2013-01-04 16:36:33,417 - root - INFO - command executed successfully 2013-01-04 16:36:33,421 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:33,528 - root - INFO - command executed successfully 2013-01-04 16:36:33,779 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:33,893 - root - INFO - command executed successfully 2013-01-04 16:36:33,894 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:34,008 - root - INFO - command executed successfully 2013-01-04 16:36:34,010 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:34,125 - root - INFO - command executed successfully 2013-01-04 16:36:34,126 - root - INFO - running command.raw hostname 2013-01-04 16:36:34,236 - root - INFO - command executed successfully 2013-01-04 16:36:34,240 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:34,347 - root - INFO - command executed successfully 2013-01-04 16:36:34,645 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:34,736 - root - INFO - command executed successfully 2013-01-04 16:36:34,737 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:34,853 - root - INFO - command executed successfully 2013-01-04 16:36:34,854 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:34,967 - root - INFO - command executed successfully 2013-01-04 16:36:34,969 - root - INFO - running command.raw hostname 2013-01-04 16:36:35,082 - root - INFO - command executed successfully 2013-01-04 16:36:35,086 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:35,087 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:35,088 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:44,640 - root - INFO - command executed successfully 2013-01-04 16:36:44,641 - root - INFO - 1d0 2013-01-04 16:36:44,642 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:44,920 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:45,034 - root - INFO - command executed successfully 2013-01-04 16:36:45,035 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:45,152 - root - INFO - command executed successfully 2013-01-04 16:36:45,155 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:45,273 - root - INFO - command executed successfully 2013-01-04 16:36:45,274 - root - INFO - running command.raw hostname 2013-01-04 16:36:45,387 - root - INFO - command executed successfully 2013-01-04 16:36:45,391 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:45,499 - root - INFO - command executed successfully 2013-01-04 16:36:45,748 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:45,864 - root - INFO - command executed successfully 2013-01-04 16:36:45,865 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:45,980 - root - INFO - command executed successfully 2013-01-04 16:36:45,981 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:45,985 - root - INFO - command executed successfully 2013-01-04 16:36:45,986 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:36:45,988 - root - INFO - 2013-01-04 16:36:45,989 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:36:45,989 - root - INFO - Please browse to http://kiwi-r102:8091/ to configure your server. 2013-01-04 16:36:45,991 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:36:45,992 - root - INFO - 2013-01-04 16:36:45,992 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:36:45,994 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:36:45,995 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:36:45,995 - root - INFO - 2013-01-04 16:36:45,996 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:36:45,996 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:36:45,996 - root - INFO - 2013-01-04 16:36:45,997 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:36:46,074 - root - INFO - command executed successfully 2013-01-04 16:36:46,075 - root - INFO - running command.raw hostname 2013-01-04 16:36:46,080 - root - INFO - command executed successfully 2013-01-04 16:36:46,081 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:36:46,187 - root - INFO - command executed successfully 2013-01-04 16:36:46,190 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:46,197 - root - INFO - command executed successfully 2013-01-04 16:36:46,197 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:36:46,305 - root - INFO - command executed successfully 2013-01-04 16:36:46,616 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:46,704 - root - INFO - command executed successfully 2013-01-04 16:36:46,705 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:46,815 - root - INFO - command executed successfully 2013-01-04 16:36:46,817 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:46,930 - root - INFO - command executed successfully 2013-01-04 16:36:46,931 - root - INFO - running command.raw hostname 2013-01-04 16:36:47,046 - root - INFO - command executed successfully 2013-01-04 16:36:47,050 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:47,050 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:47,052 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:47,500 - root - INFO - command executed successfully 2013-01-04 16:36:47,501 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:36:47,503 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.3 finished 2013-01-04 16:36:56,036 - root - INFO - command executed successfully 2013-01-04 16:36:56,037 - root - INFO - 1d0 2013-01-04 16:36:56,039 - root - INFO - < 13ffaf24b0b1da7455cf1be1c6c6e4ad couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:56,319 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:56,432 - root - INFO - command executed successfully 2013-01-04 16:36:56,433 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:56,544 - root - INFO - command executed successfully 2013-01-04 16:36:56,545 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:56,632 - root - INFO - command executed successfully 2013-01-04 16:36:56,633 - root - INFO - running command.raw hostname 2013-01-04 16:36:56,719 - root - INFO - command executed successfully 2013-01-04 16:36:56,723 - root - INFO - running command.raw /sbin/iptables -F 2013-01-04 16:36:56,830 - root - INFO - command executed successfully 2013-01-04 16:36:57,103 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:57,188 - root - INFO - command executed successfully 2013-01-04 16:36:57,189 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:57,299 - root - INFO - command executed successfully 2013-01-04 16:36:57,301 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:57,412 - root - INFO - command executed successfully 2013-01-04 16:36:57,414 - root - INFO - running command.raw hostname 2013-01-04 16:36:57,524 - root - INFO - command executed successfully 2013-01-04 16:36:57,528 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-01-04 16:36:57,636 - root - INFO - command executed successfully 2013-01-04 16:36:57,963 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:36:58,048 - root - INFO - command executed successfully 2013-01-04 16:36:58,049 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:36:58,160 - root - INFO - command executed successfully 2013-01-04 16:36:58,161 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:36:58,271 - root - INFO - command executed successfully 2013-01-04 16:36:58,272 - root - INFO - running command.raw hostname 2013-01-04 16:36:58,383 - root - INFO - command executed successfully 2013-01-04 16:36:58,386 - root - INFO - deliverable_type : rpm 2013-01-04 16:36:58,386 - root - INFO - /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm or /tmp/couchbase-server-enterprise 2013-01-04 16:36:58,387 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_64_2.0.1-112-rel.rpm 2013-01-04 16:36:58,479 - root - INFO - command executed successfully 2013-01-04 16:36:58,480 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:36:58,481 - root - INFO - 2013-01-04 16:36:58,482 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:36:58,482 - root - INFO - Please browse to http://kiwi-r105:8091/ to configure your server. 2013-01-04 16:36:58,483 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:36:58,484 - root - INFO - 2013-01-04 16:36:58,485 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:36:58,486 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:36:58,486 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:36:58,486 - root - INFO - 2013-01-04 16:36:58,487 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:36:58,488 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:36:58,488 - root - INFO - 2013-01-04 16:36:58,489 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:36:58,569 - root - INFO - command executed successfully 2013-01-04 16:36:58,571 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:36:58,685 - root - INFO - command executed successfully 2013-01-04 16:36:58,687 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:37:00,004 - root - INFO - command executed successfully 2013-01-04 16:37:00,005 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:37:00,006 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.6 finished 2013-01-04 16:37:09,052 - root - INFO - command executed successfully 2013-01-04 16:37:09,054 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-01-04 16:37:09,055 - root - INFO - 2013-01-04 16:37:09,056 - root - INFO - You have successfully installed Couchbase Server. 2013-01-04 16:37:09,057 - root - INFO - Please browse to http://kiwi-r104:8091/ to configure your server. 2013-01-04 16:37:09,058 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-01-04 16:37:09,058 - root - INFO - 2013-01-04 16:37:09,059 - root - INFO - Please note that you have to update your firewall configuration to 2013-01-04 16:37:09,059 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-01-04 16:37:09,059 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-01-04 16:37:09,060 - root - INFO - 2013-01-04 16:37:09,060 - root - INFO - By using this software you agree to the End User License Agreement. 2013-01-04 16:37:09,060 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-01-04 16:37:09,061 - root - INFO - 2013-01-04 16:37:09,061 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ 2013-01-04 16:37:09,177 - root - INFO - command executed successfully 2013-01-04 16:37:09,179 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=64/' /opt/couchbase/etc/couchbase_init.d 2013-01-04 16:37:09,289 - root - INFO - command executed successfully 2013-01-04 16:37:09,291 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-01-04 16:37:10,642 - root - INFO - command executed successfully 2013-01-04 16:37:10,644 - root - INFO - Starting couchbase-server[ OK ] 2013-01-04 16:37:10,645 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-172.27.33.5 finished 2013-01-04 16:37:15,657 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:37:15,658 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:37:15,658 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:37:16,228 - root - INFO - Connected 2013-01-04 16:37:16,235 - root - INFO - Connected 2013-01-04 16:37:16,430 - root - INFO - Connected 2013-01-04 16:37:19,746 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,749 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,759 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:19,838 - root - INFO - command executed successfully 2013-01-04 16:37:19,840 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,845 - root - INFO - command executed successfully 2013-01-04 16:37:19,846 - root - INFO - command executed successfully 2013-01-04 16:37:19,849 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,848 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:19,935 - root - INFO - command executed successfully 2013-01-04 16:37:19,936 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:19,962 - root - INFO - command executed successfully 2013-01-04 16:37:19,964 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:19,968 - root - INFO - command executed successfully 2013-01-04 16:37:19,969 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:20,047 - root - INFO - command executed successfully 2013-01-04 16:37:20,048 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,078 - root - INFO - command executed successfully 2013-01-04 16:37:20,080 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,081 - root - INFO - command executed successfully 2013-01-04 16:37:20,086 - root - INFO - running command.raw hostname 2013-01-04 16:37:20,163 - root - INFO - command executed successfully 2013-01-04 16:37:20,167 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,198 - root - INFO - command executed successfully 2013-01-04 16:37:20,200 - root - INFO - command executed successfully 2013-01-04 16:37:20,206 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,208 - root - INFO - running command.raw rm -rf /tmp//* 2013-01-04 16:37:20,288 - root - INFO - command executed successfully 2013-01-04 16:37:20,289 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,299 - root - INFO - command executed successfully 2013-01-04 16:37:20,301 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,303 - root - INFO - command executed successfully 2013-01-04 16:37:20,304 - root - INFO - running command.raw chown -R couchbase:couchbase /tmp/ 2013-01-04 16:37:20,377 - root - INFO - command executed successfully 2013-01-04 16:37:20,378 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:20,393 - root - INFO - command executed successfully 2013-01-04 16:37:20,394 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:20,416 - root - INFO - command executed successfully 2013-01-04 16:37:20,417 - root - INFO - /nodes/self/controller/settings params : path=%2Ftmp%2F 2013-01-04 16:37:22,371 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:22,375 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:22,510 - root - INFO - Setting data_path: /tmp/: status True 2013-01-04 16:37:25,374 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,378 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,390 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password 2013-01-04 16:37:25,396 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.3 finished thread initializer-thread-172.27.33.6 finished 2013-01-04 16:37:25,514 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-01-04 16:37:25,528 - root - INFO - pools/default params : memoryQuota=3276&username=Administrator&password=password thread initializer-thread-172.27.33.5 finished 2013-01-04 16:37:25,607 - root - INFO - connecting to 172.27.33.3 with username : root password : couchbase ssh_key: 2013-01-04 16:37:25,932 - root - INFO - Connected 2013-01-04 16:37:26,243 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:26,359 - root - INFO - command executed successfully 2013-01-04 16:37:26,360 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:26,474 - root - INFO - command executed successfully 2013-01-04 16:37:26,476 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:26,586 - root - INFO - command executed successfully 2013-01-04 16:37:26,589 - root - INFO - running command.raw hostname 2013-01-04 16:37:26,698 - root - INFO - command executed successfully 2013-01-04 16:37:26,818 - root - INFO - connecting to 172.27.33.6 with username : root password : couchbase ssh_key: 2013-01-04 16:37:27,142 - root - INFO - Connected 2013-01-04 16:37:27,416 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:27,504 - root - INFO - command executed successfully 2013-01-04 16:37:27,505 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:27,619 - root - INFO - command executed successfully 2013-01-04 16:37:27,620 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:27,734 - root - INFO - command executed successfully 2013-01-04 16:37:27,736 - root - INFO - running command.raw hostname 2013-01-04 16:37:27,849 - root - INFO - command executed successfully 2013-01-04 16:37:27,972 - root - INFO - connecting to 172.27.33.5 with username : root password : couchbase ssh_key: 2013-01-04 16:37:28,296 - root - INFO - Connected 2013-01-04 16:37:28,614 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-01-04 16:37:28,698 - root - INFO - command executed successfully 2013-01-04 16:37:28,700 - root - INFO - running command.raw sudo df -Th 2013-01-04 16:37:28,809 - root - INFO - command executed successfully 2013-01-04 16:37:28,811 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-01-04 16:37:28,922 - root - INFO - command executed successfully 2013-01-04 16:37:28,924 - root - INFO - running command.raw hostname 2013-01-04 16:37:29,033 - root - INFO - command executed successfully 2013-01-04 16:37:29,156 - root - INFO - Installation of new version is done. Wait for 10 sec for rebalance 2013-01-04 16:51:41,243 - root - INFO - fail_over successful 2013-01-04 16:52:36,258 - root - INFO - adding node 172.27.33.3:8091 to cluster 2013-01-04 16:52:36,258 - root - INFO - adding remote node @172.27.33.3:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:37,575 - root - INFO - adding node 172.27.33.6:8091 to cluster 2013-01-04 16:52:37,576 - root - INFO - adding remote node @172.27.33.6:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:38,548 - root - INFO - adding node 172.27.33.5:8091 to cluster 2013-01-04 16:52:38,549 - root - INFO - adding remote node @172.27.33.5:8091 to this cluster @172.27.33.2:8091 2013-01-04 16:52:39,763 - root - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5%2Cns_1%40172.27.33.2%2Cns_1%40172.27.33.4 2013-01-04 16:52:39,773 - root - INFO - rebalance operation started 2013-01-04 16:52:39,779 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:52:49,796 - root - INFO - rebalancing was completed with progress: 100% in 10.02257514 sec 2013-01-04 16:53:36,118 - root - INFO - Rebalance in all 2.0 Nodes 2013-01-04 16:53:47,639 - root - INFO - /diag/eval status: True content: 'ns_1@172.27.33.2' command: node(global:whereis_name(ns_orchestrator)) ERROR 2013-01-04 16:55:40,065 - root - INFO - ============== basetestcase cleanup was started for test #1 online_upgrade_rebalance_in_out ============== 2013-01-04 16:55:40,101 - root - INFO - deleting existing buckets on [ip:172.27.33.2 port:8091 ssh_username:root, ip:172.27.33.3 port:8091 ssh_username:root, ip:172.27.33.6 port:8091 ssh_username:root, ip:172.27.33.5 port:8091 ssh_username:root] 2013-01-04 16:55:40,402 - root - INFO - deleted bucket : default from 172.27.33.2 2013-01-04 16:55:40,402 - root - INFO - waiting for bucket deletion to complete.... 2013-01-04 16:55:40,407 - root - INFO - existing buckets : [] 2013-01-04 16:55:40,480 - root - INFO - rebalancing all nodes in order to remove nodes 2013-01-04 16:55:40,485 - root - INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5&user=Administrator&knownNodes=ns_1%40172.27.33.6%2Cns_1%40172.27.33.3%2Cns_1%40172.27.33.5%2Cns_1%40172.27.33.2 2013-01-04 16:55:40,491 - root - INFO - rebalance operation started 2013-01-04 16:55:40,499 - root - INFO - rebalance percentage : 0 % 2013-01-04 16:55:44,511 - root - INFO - rebalance progress took 4.02038812637 seconds 2013-01-04 16:55:44,512 - root - INFO - sleep for 4.02038812637 seconds after rebalance... 2013-01-04 16:55:48,574 - root - INFO - removed all the nodes from cluster associated with ip:172.27.33.2 port:8091 ssh_username:root ? [(u'ns_1@172.27.33.6', 8091), (u'ns_1@172.27.33.3', 8091), (u'ns_1@172.27.33.5', 8091)] 2013-01-04 16:55:58,593 - root - INFO - waiting for ns_server @ 172.27.33.2:8091 2013-01-04 16:55:58,602 - root - INFO - ns_server @ 172.27.33.2:8091 is running 2013-01-04 16:55:58,609 - root - INFO - waiting for ns_server @ 172.27.33.3:8091 2013-01-04 16:55:58,616 - root - INFO - ns_server @ 172.27.33.3:8091 is running 2013-01-04 16:55:58,623 - root - INFO - waiting for ns_server @ 172.27.33.6:8091 2013-01-04 16:55:58,629 - root - INFO - ns_server @ 172.27.33.6:8091 is running 2013-01-04 16:55:58,637 - root - INFO - waiting for ns_server @ 172.27.33.5:8091 2013-01-04 16:55:58,644 - root - INFO - ns_server @ 172.27.33.5:8091 is running 2013-01-04 16:55:58,645 - root - INFO - ============== basetestcase cleanup was finished for test #1 online_upgrade_rebalance_in_out ============== ====================================================================== ERROR: online_upgrade_rebalance_in_out (newupgradetests.MultiNodesUpgradeTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/newupgradetests.py", line 195, in online_upgrade_rebalance_in_out self.online_upgrade() File "pytests/newupgradetests.py", line 240, in online_upgrade raise Exception("After rebalance in 2.0 Nodes, 2.0 doesn't become the master") Exception: After rebalance in 2.0 Nodes, 2.0 doesn't become the master ---------------------------------------------------------------------- Ran 1 test in 1788.471s FAILED (errors=1) summary so far suite newupgradetests.MultiNodesUpgradeTests , pass 0 , fail 1 failures so far... newupgradetests.MultiNodesUpgradeTests.online_upgrade_rebalance_in_out testrunner logs, diags and results are available under logs/testrunner-13-Jan-04_16-26-10