pydev debugger: starting Global Test input params: {'cluster_name': 'centos-32-2.0-upgrade', 'ini': 'centos-32-2.0-upgrade.ini', 'num_nodes': 4} ./testrunner -i centos-32-2.0-upgrade.ini -t newupgradetests.MultiNodesUpgradeTests.offline_cluster_upgrade,initial_version=2.0.0-1976-rel,ddocs-num=2,upgrade_version=2.0.1-161-rel,items=10 Test Input params: {'case_number': 1, 'cluster_name': 'centos-32-2.0-upgrade', 'ddocs-num': '2', 'ini': 'centos-32-2.0-upgrade.ini', 'initial_version': '2.0.0-1976-rel', 'items': '10', 'num_nodes': 4, 'upgrade_version': '2.0.1-161-rel'} offline_cluster_upgrade (newupgradetests.MultiNodesUpgradeTests) ... 2013-02-19 12:38:45,683 - root - INFO - ============== basetestcase setup was started for test #1 offline_cluster_upgrade============== 2013-02-19 12:38:45,684 - root - INFO - any cluster operation in setup will be skipped 2013-02-19 12:38:45,684 - root - INFO - ============== basetestcase setup was finished for test #1 offline_cluster_upgrade ============== 2013-02-19 12:38:45,685 - root - INFO - connecting to 10.1.4.24 with username : root password : couchbase ssh_key: 2013-02-19 12:38:45,687 - root - INFO - connecting to 10.1.4.14 with username : root password : couchbase ssh_key: 2013-02-19 12:38:49,691 - root - INFO - Connected 2013-02-19 12:38:49,692 - root - INFO - Connected 2013-02-19 12:38:58,318 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:38:58,319 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:38:59,626 - root - INFO - command executed successfully 2013-02-19 12:38:59,630 - root - INFO - command executed successfully 2013-02-19 12:38:59,632 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:38:59,632 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:39:00,999 - root - INFO - command executed successfully 2013-02-19 12:39:01,000 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:39:01,009 - root - INFO - command executed successfully 2013-02-19 12:39:01,010 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:39:02,367 - root - INFO - command executed successfully 2013-02-19 12:39:02,369 - root - INFO - running command.raw hostname 2013-02-19 12:39:02,555 - root - INFO - command executed successfully 2013-02-19 12:39:02,556 - root - INFO - running command.raw hostname 2013-02-19 12:39:03,692 - root - INFO - command executed successfully 2013-02-19 12:39:03,697 - root - INFO - CentOS 2013-02-19 12:39:03,698 - root - INFO - running rpm -e to remove membase-server 2013-02-19 12:39:04,036 - root - INFO - command executed successfully 2013-02-19 12:39:04,041 - root - INFO - CentOS 2013-02-19 12:39:04,042 - root - INFO - running rpm -e to remove membase-server 2013-02-19 12:39:12,201 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:39:12,983 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:39:13,547 - root - INFO - command executed successfully 2013-02-19 12:39:13,549 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:39:14,334 - root - INFO - command executed successfully 2013-02-19 12:39:14,334 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:39:14,901 - root - INFO - command executed successfully 2013-02-19 12:39:14,903 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:39:15,651 - root - INFO - command executed successfully 2013-02-19 12:39:15,652 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:39:16,257 - root - INFO - command executed successfully 2013-02-19 12:39:16,259 - root - INFO - running command.raw hostname 2013-02-19 12:39:17,042 - root - INFO - command executed successfully 2013-02-19 12:39:17,043 - root - INFO - running command.raw hostname 2013-02-19 12:39:17,608 - root - INFO - command executed successfully 2013-02-19 12:39:17,614 - root - INFO - running command.raw rpm -e membase-server 2013-02-19 12:39:18,377 - root - INFO - command executed successfully 2013-02-19 12:39:18,382 - root - INFO - running command.raw rpm -e membase-server 2013-02-19 12:39:18,949 - root - INFO - command executed successfully 2013-02-19 12:39:18,949 - root - ERROR - error: package membase-server is not installed 2013-02-19 12:39:18,950 - root - INFO - running command.raw killall -9 beam 2013-02-19 12:39:19,877 - root - INFO - command executed successfully 2013-02-19 12:39:19,878 - root - ERROR - error: package membase-server is not installed 2013-02-19 12:39:19,880 - root - INFO - running command.raw killall -9 beam 2013-02-19 12:39:20,439 - root - INFO - command executed successfully 2013-02-19 12:39:20,441 - root - ERROR - beam: no process killed 2013-02-19 12:39:20,442 - root - INFO - running command.raw killall -9 memcached 2013-02-19 12:39:21,231 - root - INFO - command executed successfully 2013-02-19 12:39:21,232 - root - ERROR - beam: no process killed 2013-02-19 12:39:21,233 - root - INFO - running command.raw killall -9 memcached 2013-02-19 12:39:22,011 - root - INFO - command executed successfully 2013-02-19 12:39:22,014 - root - INFO - running command.raw killall -9 moxi 2013-02-19 12:39:22,615 - root - INFO - command executed successfully 2013-02-19 12:39:22,617 - root - INFO - running command.raw killall -9 moxi 2013-02-19 12:39:23,285 - root - INFO - command executed successfully 2013-02-19 12:39:23,287 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-02-19 12:39:23,925 - root - INFO - command executed successfully 2013-02-19 12:39:23,927 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-02-19 12:39:24,673 - root - INFO - command executed successfully 2013-02-19 12:39:24,673 - root - ERROR - vbucketmigrator: no process killed 2013-02-19 12:39:24,674 - root - INFO - running command.raw killall -9 couchdb 2013-02-19 12:39:25,320 - root - INFO - command executed successfully 2013-02-19 12:39:25,321 - root - ERROR - vbucketmigrator: no process killed 2013-02-19 12:39:25,322 - root - INFO - running command.raw killall -9 couchdb 2013-02-19 12:39:26,028 - root - INFO - command executed successfully 2013-02-19 12:39:26,030 - root - ERROR - couchdb: no process killed 2013-02-19 12:39:26,031 - root - INFO - running command.raw killall -9 epmd 2013-02-19 12:39:26,663 - root - INFO - command executed successfully 2013-02-19 12:39:26,665 - root - ERROR - couchdb: no process killed 2013-02-19 12:39:26,667 - root - INFO - running command.raw killall -9 epmd 2013-02-19 12:39:27,375 - root - INFO - command executed successfully 2013-02-19 12:39:27,376 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-02-19 12:39:28,040 - root - INFO - command executed successfully 2013-02-19 12:39:28,042 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-02-19 12:39:28,557 - root - INFO - command executed successfully 2013-02-19 12:39:28,559 - root - INFO - running command.raw rm -rf /opt/membase 2013-02-19 12:39:29,378 - root - INFO - command executed successfully 2013-02-19 12:39:29,379 - root - INFO - running command.raw rm -rf /opt/membase 2013-02-19 12:39:30,018 - root - INFO - command executed successfully 2013-02-19 12:39:30,021 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-02-19 12:39:30,687 - root - INFO - command executed successfully 2013-02-19 12:39:30,689 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-02-19 12:39:31,507 - root - INFO - command executed successfully 2013-02-19 12:39:31,509 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-02-19 12:39:32,293 - root - INFO - command executed successfully 2013-02-19 12:39:32,295 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-02-19 12:39:33,042 - root - INFO - command executed successfully 2013-02-19 12:39:33,043 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-02-19 12:39:33,829 - root - INFO - command executed successfully 2013-02-19 12:39:33,831 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-02-19 12:39:34,390 - root - INFO - command executed successfully 2013-02-19 12:39:35,326 - root - INFO - command executed successfully 2013-02-19 12:39:43,122 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:39:43,683 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:39:44,251 - root - INFO - command executed successfully 2013-02-19 12:39:44,253 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:39:45,187 - root - INFO - command executed successfully 2013-02-19 12:39:45,189 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:39:45,559 - root - INFO - command executed successfully 2013-02-19 12:39:45,561 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:39:46,495 - root - INFO - command executed successfully 2013-02-19 12:39:46,497 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:39:46,908 - root - INFO - command executed successfully 2013-02-19 12:39:46,910 - root - INFO - running command.raw hostname 2013-02-19 12:39:48,031 - root - INFO - command executed successfully 2013-02-19 12:39:48,033 - root - INFO - running command.raw hostname 2013-02-19 12:39:48,408 - root - INFO - command executed successfully 2013-02-19 12:39:48,412 - root - INFO - CentOS 2013-02-19 12:39:48,413 - root - INFO - running rpm -e to remove couchbase-server 2013-02-19 12:39:48,414 - root - INFO - running command.raw rpm -e couchbase-server 2013-02-19 12:39:49,532 - root - INFO - command executed successfully 2013-02-19 12:39:49,535 - root - INFO - CentOS 2013-02-19 12:39:49,536 - root - INFO - running rpm -e to remove couchbase-server 2013-02-19 12:39:49,537 - root - INFO - running command.raw rpm -e couchbase-server 2013-02-19 12:39:52,547 - root - INFO - command executed successfully 2013-02-19 12:39:52,548 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-02-19 12:39:52,548 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-02-19 12:39:52,549 - root - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave 2013-02-19 12:39:52,549 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-02-19 12:39:52,550 - root - INFO - {badrpc,nodedown} 2013-02-19 12:39:52,550 - root - INFO - 2013-02-19 12:39:52,551 - root - INFO - running command.raw killall -9 beam.smp 2013-02-19 12:39:53,433 - root - INFO - command executed successfully 2013-02-19 12:39:53,434 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave 2013-02-19 12:39:53,434 - root - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave 2013-02-19 12:39:53,435 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-02-19 12:39:53,435 - root - INFO - {badrpc,nodedown} 2013-02-19 12:39:53,436 - root - INFO - 2013-02-19 12:39:53,436 - root - INFO - running command.raw killall -9 beam.smp 2013-02-19 12:39:53,902 - root - INFO - command executed successfully 2013-02-19 12:39:53,905 - root - INFO - running command.raw killall -9 memcached 2013-02-19 12:39:54,878 - root - INFO - command executed successfully 2013-02-19 12:39:54,880 - root - INFO - running command.raw killall -9 memcached 2013-02-19 12:39:55,253 - root - INFO - command executed successfully 2013-02-19 12:39:55,254 - root - ERROR - memcached: no process killed 2013-02-19 12:39:55,256 - root - INFO - running command.raw killall -9 moxi 2013-02-19 12:39:56,198 - root - INFO - command executed successfully 2013-02-19 12:39:56,201 - root - ERROR - memcached: no process killed 2013-02-19 12:39:56,202 - root - INFO - running command.raw killall -9 moxi 2013-02-19 12:39:56,571 - root - INFO - command executed successfully 2013-02-19 12:39:56,573 - root - ERROR - moxi: no process killed 2013-02-19 12:39:56,574 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-02-19 12:39:57,510 - root - INFO - command executed successfully 2013-02-19 12:39:57,511 - root - ERROR - moxi: no process killed 2013-02-19 12:39:57,513 - root - INFO - running command.raw killall -9 vbucketmigrator 2013-02-19 12:39:58,074 - root - INFO - command executed successfully 2013-02-19 12:39:58,077 - root - ERROR - vbucketmigrator: no process killed 2013-02-19 12:39:58,078 - root - INFO - running command.raw killall -9 couchdb 2013-02-19 12:39:58,861 - root - INFO - command executed successfully 2013-02-19 12:39:58,862 - root - ERROR - vbucketmigrator: no process killed 2013-02-19 12:39:58,863 - root - INFO - running command.raw killall -9 couchdb 2013-02-19 12:39:59,470 - root - INFO - command executed successfully 2013-02-19 12:39:59,471 - root - ERROR - couchdb: no process killed 2013-02-19 12:39:59,472 - root - INFO - running command.raw killall -9 epmd 2013-02-19 12:40:00,174 - root - INFO - command executed successfully 2013-02-19 12:40:00,175 - root - ERROR - couchdb: no process killed 2013-02-19 12:40:00,176 - root - INFO - running command.raw killall -9 epmd 2013-02-19 12:40:00,810 - root - INFO - command executed successfully 2013-02-19 12:40:00,811 - root - INFO - running command.raw killall -9 memsup 2013-02-19 12:40:01,589 - root - INFO - command executed successfully 2013-02-19 12:40:01,590 - root - INFO - running command.raw killall -9 memsup 2013-02-19 12:40:02,371 - root - INFO - command executed successfully 2013-02-19 12:40:02,373 - root - ERROR - memsup: no process killed 2013-02-19 12:40:02,374 - root - INFO - running command.raw killall -9 cpu_sup 2013-02-19 12:40:03,120 - root - INFO - command executed successfully 2013-02-19 12:40:03,122 - root - ERROR - memsup: no process killed 2013-02-19 12:40:03,123 - root - INFO - running command.raw killall -9 cpu_sup 2013-02-19 12:40:03,871 - root - INFO - command executed successfully 2013-02-19 12:40:03,873 - root - ERROR - cpu_sup: no process killed 2013-02-19 12:40:03,874 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-02-19 12:40:04,621 - root - INFO - command executed successfully 2013-02-19 12:40:04,623 - root - ERROR - cpu_sup: no process killed 2013-02-19 12:40:04,624 - root - INFO - running command.raw rm -rf /var/opt/membase 2013-02-19 12:40:05,426 - root - INFO - command executed successfully 2013-02-19 12:40:05,428 - root - INFO - running command.raw rm -rf /opt/membase 2013-02-19 12:40:06,213 - root - INFO - command executed successfully 2013-02-19 12:40:06,214 - root - INFO - running command.raw rm -rf /opt/membase 2013-02-19 12:40:07,037 - root - INFO - command executed successfully 2013-02-19 12:40:07,038 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-02-19 12:40:07,599 - root - INFO - command executed successfully 2013-02-19 12:40:07,601 - root - INFO - running command.raw rm -rf /etc/opt/membase 2013-02-19 12:40:08,347 - root - INFO - command executed successfully 2013-02-19 12:40:08,349 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-02-19 12:40:08,907 - root - INFO - command executed successfully 2013-02-19 12:40:08,909 - root - INFO - running command.raw rm -rf /var/membase/data/* 2013-02-19 12:40:09,705 - root - INFO - command executed successfully 2013-02-19 12:40:09,706 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-02-19 12:40:10,298 - root - INFO - command executed successfully 2013-02-19 12:40:10,300 - root - INFO - running command.raw rm -rf /opt/membase/var/lib/membase/* 2013-02-19 12:40:11,053 - root - INFO - command executed successfully 2013-02-19 12:40:11,054 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-02-19 12:40:11,614 - root - INFO - command executed successfully 2013-02-19 12:40:11,616 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-02-19 12:40:12,449 - root - INFO - command executed successfully 2013-02-19 12:40:13,203 - root - INFO - command executed successfully 2013-02-19 12:40:21,537 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:40:22,094 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:40:22,882 - root - INFO - command executed successfully 2013-02-19 12:40:22,883 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:40:23,819 - root - INFO - command executed successfully 2013-02-19 12:40:23,821 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:40:24,198 - root - INFO - command executed successfully 2013-02-19 12:40:24,199 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:40:25,323 - root - INFO - command executed successfully 2013-02-19 12:40:25,324 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:40:25,512 - root - INFO - command executed successfully 2013-02-19 12:40:25,513 - root - INFO - running command.raw hostname 2013-02-19 12:40:26,868 - root - INFO - command executed successfully 2013-02-19 12:40:26,869 - root - INFO - command executed successfully 2013-02-19 12:40:26,870 - root - INFO - running command.raw hostname 2013-02-19 12:40:26,879 - root - INFO - CentOS 2013-02-19 12:40:26,880 - root - INFO - running rpm -e to remove couchbase-server 2013-02-19 12:40:26,881 - root - INFO - running command.raw rpm -e couchbase-server 2013-02-19 12:40:28,217 - root - INFO - command executed successfully 2013-02-19 12:40:28,225 - root - INFO - CentOS 2013-02-19 12:40:28,226 - root - INFO - running rpm -e to remove couchbase-server 2013-02-19 12:40:28,226 - root - INFO - running command.raw rpm -e couchbase-server 2013-02-19 12:40:28,250 - root - INFO - command executed successfully 2013-02-19 12:40:28,251 - root - ERROR - error: package couchbase-server is not installed 2013-02-19 12:40:28,251 - root - INFO - running command.raw killall -9 beam 2013-02-19 12:40:29,643 - root - INFO - command executed successfully 2013-02-19 12:40:29,646 - root - ERROR - beam: no process killed 2013-02-19 12:40:29,648 - root - INFO - running command.raw killall -9 couchdb 2013-02-19 12:40:29,650 - root - INFO - command executed successfully 2013-02-19 12:40:29,652 - root - ERROR - error: package couchbase-server is not installed 2013-02-19 12:40:29,653 - root - INFO - running command.raw killall -9 beam 2013-02-19 12:40:31,198 - root - INFO - command executed successfully 2013-02-19 12:40:31,199 - root - ERROR - couchdb: no process killed 2013-02-19 12:40:31,201 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-02-19 12:40:31,201 - root - INFO - command executed successfully 2013-02-19 12:40:31,205 - root - ERROR - beam: no process killed 2013-02-19 12:40:31,206 - root - INFO - running command.raw killall -9 couchdb 2013-02-19 12:40:32,559 - root - INFO - command executed successfully 2013-02-19 12:40:32,562 - root - INFO - command executed successfully 2013-02-19 12:40:32,563 - root - ERROR - couchdb: no process killed 2013-02-19 12:40:32,564 - root - INFO - running command.raw rm -rf /opt/couchbase 2013-02-19 12:40:33,998 - root - INFO - command executed successfully thread uninstaller-thread-10.1.4.24 finished thread uninstaller-thread-10.1.4.14 finished 2013-02-19 12:40:34,001 - root - INFO - connecting to 10.1.4.24 with username : root password : couchbase ssh_key: 2013-02-19 12:40:34,002 - root - INFO - connecting to 10.1.4.14 with username : root password : couchbase ssh_key: 2013-02-19 12:40:37,605 - root - INFO - Connected 2013-02-19 12:40:37,790 - root - INFO - Connected 2013-02-19 12:40:46,167 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:40:46,169 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:40:47,400 - root - INFO - command executed successfully 2013-02-19 12:40:47,402 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:40:47,407 - root - INFO - command executed successfully 2013-02-19 12:40:47,408 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:40:48,688 - root - INFO - command executed successfully 2013-02-19 12:40:48,690 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:40:48,835 - root - INFO - command executed successfully 2013-02-19 12:40:48,836 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:40:50,129 - root - INFO - command executed successfully 2013-02-19 12:40:50,130 - root - INFO - running command.raw hostname 2013-02-19 12:40:50,341 - root - INFO - command executed successfully 2013-02-19 12:40:50,343 - root - INFO - running command.raw hostname 2013-02-19 12:40:51,382 - root - INFO - command executed successfully 2013-02-19 12:40:51,701 - root - INFO - command executed successfully 2013-02-19 12:42:37,811 - root - INFO - connecting to 10.1.4.24 with username : root password : couchbase ssh_key: 2013-02-19 12:42:38,575 - root - INFO - connecting to 10.1.4.14 with username : root password : couchbase ssh_key: 2013-02-19 12:42:41,609 - root - INFO - Connected 2013-02-19 12:42:42,087 - root - INFO - Connected 2013-02-19 12:42:49,597 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:42:50,751 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:42:51,125 - root - INFO - command executed successfully 2013-02-19 12:42:51,126 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:42:52,059 - root - INFO - command executed successfully 2013-02-19 12:42:52,061 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:42:52,470 - root - INFO - command executed successfully 2013-02-19 12:42:52,472 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:42:53,671 - root - INFO - command executed successfully 2013-02-19 12:42:53,672 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:42:54,041 - root - INFO - command executed successfully 2013-02-19 12:42:54,042 - root - INFO - running command.raw hostname 2013-02-19 12:42:55,057 - root - INFO - command executed successfully 2013-02-19 12:42:55,059 - root - INFO - running command.raw hostname 2013-02-19 12:42:55,374 - root - INFO - command executed successfully 2013-02-19 12:42:56,693 - root - INFO - command executed successfully 2013-02-19 12:43:03,994 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:43:05,343 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:43:05,344 - root - INFO - command executed successfully 2013-02-19 12:43:05,346 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:43:06,659 - root - INFO - command executed successfully 2013-02-19 12:43:06,661 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:43:06,664 - root - INFO - command executed successfully 2013-02-19 12:43:06,665 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:43:07,977 - root - INFO - command executed successfully 2013-02-19 12:43:07,978 - root - INFO - running command.raw hostname 2013-02-19 12:43:07,979 - root - INFO - command executed successfully 2013-02-19 12:43:07,980 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:43:09,508 - root - INFO - command executed successfully 2013-02-19 12:43:09,513 - root - INFO - command executed successfully 2013-02-19 12:43:09,514 - root - INFO - running command.raw hostname 2013-02-19 12:43:11,049 - root - INFO - command executed successfully 2013-02-19 12:43:18,262 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:43:19,610 - root - INFO - command executed successfully 2013-02-19 12:43:19,611 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:43:20,009 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:43:20,946 - root - INFO - command executed successfully 2013-02-19 12:43:20,948 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:43:21,545 - root - INFO - command executed successfully 2013-02-19 12:43:21,547 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:43:22,303 - root - INFO - command executed successfully 2013-02-19 12:43:22,304 - root - INFO - running command.raw hostname 2013-02-19 12:43:22,863 - root - INFO - command executed successfully 2013-02-19 12:43:22,864 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:43:23,611 - root - INFO - command executed successfully 2013-02-19 12:43:23,616 - root - INFO - running command.raw /sbin/iptables -F 2013-02-19 12:43:24,247 - root - INFO - command executed successfully 2013-02-19 12:43:24,249 - root - INFO - running command.raw hostname 2013-02-19 12:43:25,149 - root - INFO - command executed successfully 2013-02-19 12:43:25,583 - root - INFO - command executed successfully 2013-02-19 12:43:25,588 - root - INFO - running command.raw /sbin/iptables -F 2013-02-19 12:43:26,994 - root - INFO - command executed successfully 2013-02-19 12:43:33,383 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:43:34,879 - root - INFO - command executed successfully 2013-02-19 12:43:34,880 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:43:35,441 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:43:36,192 - root - INFO - command executed successfully 2013-02-19 12:43:36,195 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:43:36,806 - root - INFO - command executed successfully 2013-02-19 12:43:36,807 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:43:37,595 - root - INFO - command executed successfully 2013-02-19 12:43:37,596 - root - INFO - running command.raw hostname 2013-02-19 12:43:38,156 - root - INFO - command executed successfully 2013-02-19 12:43:38,158 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:43:39,092 - root - INFO - command executed successfully 2013-02-19 12:43:39,097 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-02-19 12:43:39,471 - root - INFO - command executed successfully 2013-02-19 12:43:39,473 - root - INFO - running command.raw hostname 2013-02-19 12:43:40,483 - root - INFO - command executed successfully 2013-02-19 12:43:40,485 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-02-19 12:43:40,821 - root - INFO - command executed successfully 2013-02-19 12:43:40,826 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-02-19 12:43:41,993 - root - INFO - command executed successfully 2013-02-19 12:43:41,995 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-02-19 12:43:41,995 - root - ERROR - rm: cannot remove `cb_0732': Is a directory 2013-02-19 12:43:41,996 - root - ERROR - rm: cannot remove `data': Is a directory 2013-02-19 12:43:41,998 - root - ERROR - rm: cannot remove `index': Is a directory 2013-02-19 12:43:41,998 - root - INFO - get md5 sum for local and remote 2013-02-19 12:43:41,999 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm > couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm.md5l 2013-02-19 12:43:42,364 - root - INFO - command executed successfully 2013-02-19 12:43:42,365 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-02-19 12:43:43,615 - root - INFO - command executed successfully 2013-02-19 12:43:43,616 - root - INFO - comparing md5 sum and downloading if needed 2013-02-19 12:43:43,617 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm.md5 couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm;rm -f *.md5 *.md5l 2013-02-19 12:43:44,003 - root - INFO - command executed successfully 2013-02-19 12:43:44,003 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm': No such file or directory 2013-02-19 12:43:44,004 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-02-19 12:43:44,005 - root - ERROR - rm: cannot remove `cb_2869': Is a directory 2013-02-19 12:43:44,005 - root - ERROR - mv: cannot stat `cb_2869/*': No such file or directory 2013-02-19 12:43:44,006 - root - INFO - get md5 sum for local and remote 2013-02-19 12:43:44,007 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm > couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm.md5l 2013-02-19 12:43:45,880 - root - INFO - command executed successfully 2013-02-19 12:43:45,884 - root - INFO - command executed successfully 2013-02-19 12:43:45,884 - root - ERROR - md5sum: couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm: No such file or directory 2013-02-19 12:43:45,885 - root - INFO - comparing md5 sum and downloading if needed 2013-02-19 12:43:45,885 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm.md5 couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm;rm -f *.md5 *.md5l 2013-02-19 12:43:58,023 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:43:59,270 - root - INFO - command executed successfully 2013-02-19 12:43:59,272 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:44:00,616 - root - INFO - command executed successfully 2013-02-19 12:44:00,617 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:44:02,000 - root - INFO - command executed successfully 2013-02-19 12:44:02,002 - root - INFO - running command.raw hostname 2013-02-19 12:44:03,559 - root - INFO - command executed successfully 2013-02-19 12:44:03,561 - root - INFO - deliverable_type : rpm 2013-02-19 12:44:03,562 - root - INFO - /tmp/couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm or /tmp/couchbase-server-enterprise 2013-02-19 12:44:03,562 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm 2013-02-19 12:44:08,071 - root - INFO - command executed successfully 2013-02-19 12:44:08,073 - root - INFO - 1d0 2013-02-19 12:44:08,074 - root - INFO - < 1d4192c24e21e04f1c4512729375e828 couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm 2013-02-19 12:44:16,054 - root - INFO - command executed successfully 2013-02-19 12:44:16,055 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-02-19 12:44:16,056 - root - INFO - 2013-02-19 12:44:16,057 - root - INFO - You have successfully installed Couchbase Server. 2013-02-19 12:44:16,058 - root - INFO - Please browse to http://localhost.localdomain:8091/ to configure your server. 2013-02-19 12:44:16,059 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-02-19 12:44:16,060 - root - INFO - 2013-02-19 12:44:16,061 - root - INFO - Please note that you have to update your firewall configuration to 2013-02-19 12:44:16,062 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-02-19 12:44:16,062 - root - INFO - 8091 and from 21100 to 21299. 2013-02-19 12:44:16,063 - root - INFO - 2013-02-19 12:44:16,063 - root - INFO - By using this software you agree to the End User License Agreement. 2013-02-19 12:44:16,064 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-02-19 12:44:16,064 - root - INFO - 2013-02-19 12:44:17,633 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2013-02-19 12:44:17,634 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp 2013-02-19 12:44:18,674 - root - INFO - command executed successfully 2013-02-19 12:44:18,675 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=1024/' /opt/couchbase/etc/couchbase_init.d 2013-02-19 12:44:19,809 - root - INFO - command executed successfully 2013-02-19 12:44:19,811 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-02-19 12:44:20,003 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:44:21,124 - root - INFO - command executed successfully 2013-02-19 12:44:21,126 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:44:22,287 - root - INFO - command executed successfully 2013-02-19 12:44:22,288 - root - INFO - Starting couchbase-server[ OK ] 2013-02-19 12:44:22,288 - root - INFO - wait 5 seconds for membase server to start 2013-02-19 12:44:22,660 - root - INFO - command executed successfully 2013-02-19 12:44:22,661 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:44:24,047 - root - INFO - command executed successfully 2013-02-19 12:44:24,048 - root - INFO - running command.raw hostname 2013-02-19 12:44:25,377 - root - INFO - command executed successfully 2013-02-19 12:44:25,382 - root - INFO - deliverable_type : rpm 2013-02-19 12:44:25,383 - root - INFO - /tmp/couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm or /tmp/couchbase-server-enterprise 2013-02-19 12:44:25,383 - root - INFO - running command.raw INSTALL_DONT_START_SERVER=1 rpm -i /tmp/couchbase-server-enterprise_x86_2.0.0-1976-rel.rpm thread installer-thread-10.1.4.24 finished 2013-02-19 12:44:38,236 - root - INFO - command executed successfully 2013-02-19 12:44:38,237 - root - INFO - Skipping server start due to INSTALL_DONT_START_SERVER ... 2013-02-19 12:44:38,239 - root - INFO - 2013-02-19 12:44:38,240 - root - INFO - You have successfully installed Couchbase Server. 2013-02-19 12:44:38,241 - root - INFO - Please browse to http://localhost.localdomain:8091/ to configure your server. 2013-02-19 12:44:38,242 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-02-19 12:44:38,243 - root - INFO - 2013-02-19 12:44:38,243 - root - INFO - Please note that you have to update your firewall configuration to 2013-02-19 12:44:38,244 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-02-19 12:44:38,244 - root - INFO - 8091 and from 21100 to 21299. 2013-02-19 12:44:38,245 - root - INFO - 2013-02-19 12:44:38,246 - root - INFO - By using this software you agree to the End User License Agreement. 2013-02-19 12:44:38,246 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-02-19 12:44:38,247 - root - INFO - 2013-02-19 12:44:40,007 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2013-02-19 12:44:40,007 - root - INFO - running command.raw /opt/couchbase/bin/cbenable_core_dumps.sh /tmp 2013-02-19 12:44:41,136 - root - INFO - command executed successfully 2013-02-19 12:44:41,138 - root - INFO - running command.raw sed -i 's/ulimit -c unlimited/ulimit -c unlimited\n export COUCHBASE_NUM_VBUCKETS=1024/' /opt/couchbase/etc/couchbase_init.d 2013-02-19 12:44:42,275 - root - INFO - command executed successfully 2013-02-19 12:44:42,277 - root - INFO - running command.raw /etc/init.d/couchbase-server start 2013-02-19 12:44:45,044 - root - INFO - command executed successfully 2013-02-19 12:44:45,045 - root - INFO - Starting couchbase-server[ OK ] 2013-02-19 12:44:45,046 - root - INFO - wait 5 seconds for membase server to start thread installer-thread-10.1.4.14 finished 2013-02-19 12:44:50,059 - root - INFO - connecting to 10.1.4.24 with username : root password : couchbase ssh_key: 2013-02-19 12:44:50,060 - root - INFO - connecting to 10.1.4.14 with username : root password : couchbase ssh_key: 2013-02-19 12:44:53,653 - root - INFO - Connected 2013-02-19 12:44:54,010 - root - INFO - Connected 2013-02-19 12:44:57,775 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-02-19 12:44:58,012 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-02-19 12:45:00,019 - root - INFO - pools/default params : memoryQuota=1920&username=Administrator&password=password 2013-02-19 12:45:00,020 - root - INFO - pools/default params : memoryQuota=1920&username=Administrator&password=password thread initializer-thread-10.1.4.24 finished thread initializer-thread-10.1.4.14 finished 2013-02-19 12:45:00,564 - root - INFO - connecting to 10.1.4.24 with username : root password : couchbase ssh_key: 2013-02-19 12:45:05,188 - root - INFO - Connected 2013-02-19 12:45:13,648 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:45:14,968 - root - INFO - command executed successfully 2013-02-19 12:45:14,969 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:45:16,349 - root - INFO - command executed successfully 2013-02-19 12:45:16,352 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:45:17,696 - root - INFO - command executed successfully 2013-02-19 12:45:17,698 - root - INFO - running command.raw hostname 2013-02-19 12:45:19,010 - root - INFO - command executed successfully 2013-02-19 12:45:22,189 - root - INFO - connecting to 10.1.4.14 with username : root password : couchbase ssh_key: 2013-02-19 12:45:25,856 - root - INFO - Connected 2013-02-19 12:45:34,497 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 12:45:36,013 - root - INFO - command executed successfully 2013-02-19 12:45:36,014 - root - INFO - running command.raw sudo df -Th 2013-02-19 12:45:37,319 - root - INFO - command executed successfully 2013-02-19 12:45:37,320 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 12:45:39,004 - root - INFO - command executed successfully 2013-02-19 12:45:39,006 - root - INFO - running command.raw hostname 2013-02-19 12:45:40,392 - root - INFO - command executed successfully 2013-02-19 12:45:45,620 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-02-19 12:45:48,660 - root - INFO - pools/default params : memoryQuota=1280&username=Administrator&password=password 2013-02-19 12:45:50,387 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-02-19 12:45:53,049 - root - INFO - pools/default params : memoryQuota=1280&username=Administrator&password=password 2013-02-19 12:45:55,662 - root - INFO - adding node 10.1.4.14:8091 to cluster 2013-02-19 12:45:55,663 - root - INFO - adding remote node @10.1.4.14:8091 to this cluster @10.1.4.24:8091 2013-02-19 12:45:58,382 - root - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.1.4.24%2Cns_1%4010.1.4.14 2013-02-19 12:45:58,993 - root - INFO - rebalance operation started 2013-02-19 12:45:59,880 - root - INFO - rebalancing was completed with progress: 100% in 0.886430025101 sec 2013-02-19 12:46:01,799 - root - INFO - http://10.1.4.24:8091/pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=None&flushEnabled=1&replicaNumber=1&ramQuotaMB=213 2013-02-19 12:46:02,421 - root - INFO - 0.621421098709 seconds to create bucket default 2013-02-19 12:46:02,422 - root - INFO - waiting for memcached bucket : default in 10.1.4.24 to accept set ops 2013-02-19 12:46:12,759 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 12:46:20,508 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 12:46:30,877 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 12:46:39,617 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 12:52:37,114 - root - INFO - bucket 'default' was created with per node RAM quota: 213 2013-02-19 12:52:47,353 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 12:52:55,393 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 12:53:00,140 - root - INFO - Batch create documents done #: 0 with exp:0 2013-02-19 12:53:01,631 - root - INFO - Batch create documents done #: 10 with exp:0 2013-02-19 12:53:04,355 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 12:53:10,334 - root - INFO - Saw ep_queue_size 0 == 0 expected on '10.1.4.24:8091' 2013-02-19 12:53:12,310 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 12:53:18,598 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '10.1.4.24:8091' 2013-02-19 12:53:20,501 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 12:53:26,649 - root - INFO - Saw ep_queue_size 0 == 0 expected on '10.1.4.14:8091' 2013-02-19 12:53:28,709 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 12:53:34,595 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '10.1.4.14:8091' 2013-02-19 12:53:43,524 - root - ERROR - http://10.1.4.24:8092//default/_design/upgrade-test-view0 error 404 reason: not_found {"error":"not_found","reason":"missing"} 2013-02-19 12:53:44,548 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?stale=ok 2013-02-19 12:53:45,620 - root - INFO - view : upgrade-test-view0 was created successfully in ddoc: upgrade-test-view0 2013-02-19 12:53:50,017 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?stale=ok 2013-02-19 12:53:51,018 - root - INFO - view : upgrade-test-view1 was created successfully in ddoc: upgrade-test-view0 2013-02-19 12:53:54,014 - root - ERROR - http://10.1.4.24:8092//default/_design/upgrade-test-view1 error 404 reason: not_found {"error":"not_found","reason":"missing"} 2013-02-19 12:53:54,675 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view1/_view/upgrade-test-view0?stale=ok 2013-02-19 12:53:55,671 - root - INFO - view : upgrade-test-view0 was created successfully in ddoc: upgrade-test-view1 2013-02-19 12:54:00,599 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view1/_view/upgrade-test-view1?stale=ok 2013-02-19 12:54:01,727 - root - INFO - view : upgrade-test-view1 was created successfully in ddoc: upgrade-test-view1 2013-02-19 12:54:02,705 - root - INFO - waiting for memcached bucket : default in 10.1.4.24 to accept set ops 2013-02-19 12:54:13,330 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 12:54:20,854 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 12:54:31,659 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 12:54:40,005 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 13:00:40,542 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 13:00:48,463 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 13:00:56,384 - root - INFO - sleep for 10 secs. Pre-setup of old version is done. Wait for upgrade ... 2013-02-19 13:01:06,395 - root - INFO - connecting to 10.1.4.24 with username : root password : couchbase ssh_key: 2013-02-19 13:01:10,070 - root - INFO - Connected 2013-02-19 13:01:18,008 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:01:19,549 - root - INFO - command executed successfully 2013-02-19 13:01:19,550 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:01:20,909 - root - INFO - command executed successfully 2013-02-19 13:01:20,910 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:01:22,228 - root - INFO - command executed successfully 2013-02-19 13:01:22,229 - root - INFO - running command.raw hostname 2013-02-19 13:01:23,617 - root - INFO - command executed successfully 2013-02-19 13:01:35,097 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:01:36,264 - root - INFO - command executed successfully 2013-02-19 13:01:36,266 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:01:37,540 - root - INFO - command executed successfully 2013-02-19 13:01:37,542 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:01:38,887 - root - INFO - command executed successfully 2013-02-19 13:01:38,889 - root - INFO - running command.raw hostname 2013-02-19 13:01:40,221 - root - INFO - command executed successfully 2013-02-19 13:01:40,227 - root - INFO - running command.raw /etc/init.d/couchbase-server stop 2013-02-19 13:01:51,295 - root - INFO - command executed successfully 2013-02-19 13:01:51,297 - root - INFO - Stopping couchbase-server 2013-02-19 13:01:51,299 - root - INFO - =INFO REPORT==== 19-Feb-2013::02:01:41 === 2013-02-19 13:01:51,300 - root - INFO - Initiated server shutdown** at node ns_1@10.1.4.24 ** 2013-02-19 13:01:51,300 - root - INFO - 2013-02-19 13:01:51,301 - root - INFO - =INFO REPORT==== 19-Feb-2013::02:01:50 === 2013-02-19 13:01:51,301 - root - INFO - Stopped ns_server application** at node ns_1@10.1.4.24 ** 2013-02-19 13:01:51,302 - root - INFO - 2013-02-19 13:01:51,303 - root - INFO - sleep for 10 secs. ... 2013-02-19 13:02:01,321 - root - INFO - connecting to 10.1.4.14 with username : root password : couchbase ssh_key: 2013-02-19 13:02:05,081 - root - INFO - Connected 2013-02-19 13:02:13,990 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:02:15,307 - root - INFO - command executed successfully 2013-02-19 13:02:15,309 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:02:16,645 - root - INFO - command executed successfully 2013-02-19 13:02:16,646 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:02:17,995 - root - INFO - command executed successfully 2013-02-19 13:02:17,996 - root - INFO - running command.raw hostname 2013-02-19 13:02:19,341 - root - INFO - command executed successfully 2013-02-19 13:02:31,242 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:02:32,581 - root - INFO - command executed successfully 2013-02-19 13:02:32,582 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:02:33,935 - root - INFO - command executed successfully 2013-02-19 13:02:33,936 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:02:35,248 - root - INFO - command executed successfully 2013-02-19 13:02:35,249 - root - INFO - running command.raw hostname 2013-02-19 13:02:36,596 - root - INFO - command executed successfully 2013-02-19 13:02:36,599 - root - INFO - running command.raw /etc/init.d/couchbase-server stop 2013-02-19 13:02:45,586 - root - INFO - command executed successfully 2013-02-19 13:02:45,588 - root - INFO - Stopping couchbase-server 2013-02-19 13:02:45,589 - root - INFO - =INFO REPORT==== 19-Feb-2013::02:02:37 === 2013-02-19 13:02:45,591 - root - INFO - Initiated server shutdown** at node ns_1@10.1.4.14 ** 2013-02-19 13:02:45,592 - root - INFO - 2013-02-19 13:02:45,592 - root - INFO - =INFO REPORT==== 19-Feb-2013::02:02:44 === 2013-02-19 13:02:45,593 - root - INFO - Stopped ns_server application** at node ns_1@10.1.4.14 ** 2013-02-19 13:02:45,593 - root - INFO - 2013-02-19 13:02:45,594 - root - INFO - sleep for 10 secs. ... 2013-02-19 13:02:55,613 - root - INFO - connecting to 10.1.4.24 with username : root password : couchbase ssh_key: 2013-02-19 13:02:55,614 - root - INFO - connecting to 10.1.4.14 with username : root password : couchbase ssh_key: 2013-02-19 13:02:59,481 - root - INFO - Connected 2013-02-19 13:02:59,553 - root - INFO - Connected 2013-02-19 13:03:07,816 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:03:07,819 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:03:09,211 - root - INFO - command executed successfully 2013-02-19 13:03:09,212 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:03:09,214 - root - INFO - command executed successfully 2013-02-19 13:03:09,215 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:03:10,522 - root - INFO - command executed successfully 2013-02-19 13:03:10,524 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:03:10,711 - root - INFO - command executed successfully 2013-02-19 13:03:10,713 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:03:12,030 - root - INFO - command executed successfully 2013-02-19 13:03:12,031 - root - INFO - running command.raw hostname 2013-02-19 13:03:12,259 - root - INFO - command executed successfully 2013-02-19 13:03:12,260 - root - INFO - running command.raw hostname 2013-02-19 13:03:13,197 - root - INFO - command executed successfully 2013-02-19 13:03:13,758 - root - INFO - command executed successfully 2013-02-19 13:04:55,608 - root - INFO - finding build 2.0.1-161-rel for machine ip:10.1.4.14 port:8091 ssh_username:root 2013-02-19 13:04:57,288 - root - INFO - finding build 2.0.1-161-rel for machine ip:10.1.4.24 port:8091 ssh_username:root 2013-02-19 13:05:04,033 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:05:05,131 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:05:05,509 - root - INFO - command executed successfully 2013-02-19 13:05:05,511 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:05:06,442 - root - INFO - command executed successfully 2013-02-19 13:05:06,443 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:05:06,816 - root - INFO - command executed successfully 2013-02-19 13:05:06,818 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:05:08,129 - root - INFO - command executed successfully 2013-02-19 13:05:08,130 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:05:08,315 - root - INFO - command executed successfully 2013-02-19 13:05:08,316 - root - INFO - running command.raw hostname 2013-02-19 13:05:09,475 - root - INFO - command executed successfully 2013-02-19 13:05:09,477 - root - INFO - running command.raw hostname 2013-02-19 13:05:09,970 - root - INFO - command executed successfully 2013-02-19 13:05:10,907 - root - INFO - command executed successfully 2013-02-19 13:05:19,479 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:05:20,412 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:05:20,973 - root - INFO - command executed successfully 2013-02-19 13:05:20,974 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:05:21,730 - root - INFO - command executed successfully 2013-02-19 13:05:21,731 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:05:22,294 - root - INFO - command executed successfully 2013-02-19 13:05:22,295 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:05:23,067 - root - INFO - command executed successfully 2013-02-19 13:05:23,069 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:05:23,814 - root - INFO - command executed successfully 2013-02-19 13:05:23,815 - root - INFO - running command.raw hostname 2013-02-19 13:05:24,604 - root - INFO - command executed successfully 2013-02-19 13:05:24,605 - root - INFO - running command.raw hostname 2013-02-19 13:05:24,976 - root - INFO - command executed successfully 2013-02-19 13:05:24,981 - root - INFO - running command.raw /sbin/iptables -F 2013-02-19 13:05:25,995 - root - INFO - command executed successfully 2013-02-19 13:05:26,000 - root - INFO - running command.raw /sbin/iptables -F 2013-02-19 13:05:26,314 - root - INFO - command executed successfully 2013-02-19 13:05:27,268 - root - INFO - command executed successfully 2013-02-19 13:05:35,123 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:05:35,421 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:05:36,396 - root - INFO - command executed successfully 2013-02-19 13:05:36,398 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:05:36,993 - root - INFO - command executed successfully 2013-02-19 13:05:36,995 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:05:38,001 - root - INFO - command executed successfully 2013-02-19 13:05:38,003 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:05:38,525 - root - INFO - command executed successfully 2013-02-19 13:05:38,526 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:05:39,462 - root - INFO - command executed successfully 2013-02-19 13:05:39,464 - root - INFO - running command.raw hostname 2013-02-19 13:05:40,091 - root - INFO - command executed successfully 2013-02-19 13:05:40,092 - root - INFO - running command.raw hostname 2013-02-19 13:05:41,046 - root - INFO - command executed successfully 2013-02-19 13:05:41,051 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-02-19 13:05:41,421 - root - INFO - command executed successfully 2013-02-19 13:05:41,426 - root - INFO - running command.raw /sbin/iptables -t nat -F 2013-02-19 13:05:42,552 - root - INFO - command executed successfully 2013-02-19 13:05:42,552 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_2.0.1-161-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-02-19 13:05:43,059 - root - INFO - command executed successfully 2013-02-19 13:05:43,059 - root - INFO - running command.raw cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_2.0.1-161-rel.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D 2013-02-19 13:05:44,089 - root - INFO - command executed successfully 2013-02-19 13:05:44,090 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_2.0.1-161-rel.rpm': No such file or directory 2013-02-19 13:05:44,091 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-02-19 13:05:44,092 - root - ERROR - rm: cannot remove `cb_4727': Is a directory 2013-02-19 13:05:44,093 - root - ERROR - mv: cannot stat `cb_4727/*': No such file or directory 2013-02-19 13:05:44,094 - root - INFO - get md5 sum for local and remote 2013-02-19 13:05:44,095 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.1-161-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_2.0.1-161-rel.rpm > couchbase-server-enterprise_x86_2.0.1-161-rel.rpm.md5l 2013-02-19 13:05:44,498 - root - INFO - command executed successfully 2013-02-19 13:05:44,499 - root - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_2.0.1-161-rel.rpm': No such file or directory 2013-02-19 13:05:44,500 - root - ERROR - mv: cannot stat `core.*': No such file or directory 2013-02-19 13:05:44,501 - root - ERROR - rm: cannot remove `cb_2573': Is a directory 2013-02-19 13:05:44,502 - root - ERROR - rm: cannot remove `data': Is a directory 2013-02-19 13:05:44,504 - root - ERROR - rm: cannot remove `index': Is a directory 2013-02-19 13:05:44,504 - root - ERROR - mv: cannot stat `cb_2573/*': No such file or directory 2013-02-19 13:05:44,505 - root - INFO - get md5 sum for local and remote 2013-02-19 13:05:44,505 - root - INFO - running command.raw cd /tmp ; rm -f *.md5 *.md5l ; wget -q http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.1-161-rel.rpm.md5 ; md5sum couchbase-server-enterprise_x86_2.0.1-161-rel.rpm > couchbase-server-enterprise_x86_2.0.1-161-rel.rpm.md5l 2013-02-19 13:05:45,698 - root - INFO - command executed successfully 2013-02-19 13:05:45,700 - root - ERROR - md5sum: couchbase-server-enterprise_x86_2.0.1-161-rel.rpm: No such file or directory 2013-02-19 13:05:45,701 - root - INFO - comparing md5 sum and downloading if needed 2013-02-19 13:05:45,703 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_2.0.1-161-rel.rpm.md5 couchbase-server-enterprise_x86_2.0.1-161-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_2.0.1-161-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.1-161-rel.rpm;rm -f *.md5 *.md5l 2013-02-19 13:05:46,053 - root - INFO - command executed successfully 2013-02-19 13:05:46,054 - root - ERROR - md5sum: couchbase-server-enterprise_x86_2.0.1-161-rel.rpm: No such file or directory 2013-02-19 13:05:46,055 - root - INFO - comparing md5 sum and downloading if needed 2013-02-19 13:05:46,056 - root - INFO - running command.raw cd /tmp;diff couchbase-server-enterprise_x86_2.0.1-161-rel.rpm.md5 couchbase-server-enterprise_x86_2.0.1-161-rel.rpm.md5l || wget -q -O couchbase-server-enterprise_x86_2.0.1-161-rel.rpm http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_2.0.1-161-rel.rpm;rm -f *.md5 *.md5l 2013-02-19 13:07:13,155 - root - INFO - command executed successfully 2013-02-19 13:07:13,157 - root - INFO - 1d0 2013-02-19 13:07:13,158 - root - INFO - < 3f718b8dd3bd5dfa6a1f0204d926cf59 couchbase-server-enterprise_x86_2.0.1-161-rel.rpm 2013-02-19 13:07:14,335 - root - INFO - command executed successfully 2013-02-19 13:07:14,336 - root - INFO - 1d0 2013-02-19 13:07:14,336 - root - INFO - < 3f718b8dd3bd5dfa6a1f0204d926cf59 couchbase-server-enterprise_x86_2.0.1-161-rel.rpm 2013-02-19 13:07:23,608 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:07:24,793 - root - INFO - command executed successfully 2013-02-19 13:07:24,794 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:07:25,194 - root - INFO - running command.raw sudo cat /proc/cpuinfo 2013-02-19 13:07:26,144 - root - INFO - command executed successfully 2013-02-19 13:07:26,145 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:07:26,409 - root - INFO - command executed successfully 2013-02-19 13:07:26,410 - root - INFO - running command.raw sudo df -Th 2013-02-19 13:07:27,481 - root - INFO - command executed successfully 2013-02-19 13:07:27,482 - root - INFO - running command.raw hostname 2013-02-19 13:07:27,485 - root - INFO - command executed successfully 2013-02-19 13:07:27,486 - root - INFO - running command.raw sudo cat /proc/meminfo 2013-02-19 13:07:28,603 - root - INFO - command executed successfully 2013-02-19 13:07:28,607 - root - INFO - command executed successfully 2013-02-19 13:07:28,609 - root - INFO - running command.raw hostname 2013-02-19 13:07:28,613 - root - INFO - deliverable_type : rpm 2013-02-19 13:07:28,614 - root - INFO - /tmp/couchbase-server-enterprise_x86_2.0.1-161-rel.rpm or /tmp/couchbase-server-enterprise 2013-02-19 13:07:28,615 - root - INFO - running command.raw rpm -U /tmp/couchbase-server-enterprise_x86_2.0.1-161-rel.rpm 2013-02-19 13:07:30,174 - root - INFO - command executed successfully 2013-02-19 13:07:30,179 - root - INFO - deliverable_type : rpm 2013-02-19 13:07:30,179 - root - INFO - /tmp/couchbase-server-enterprise_x86_2.0.1-161-rel.rpm or /tmp/couchbase-server-enterprise 2013-02-19 13:07:30,180 - root - INFO - running command.raw rpm -U /tmp/couchbase-server-enterprise_x86_2.0.1-161-rel.rpm 2013-02-19 13:07:47,137 - root - INFO - command executed successfully 2013-02-19 13:07:47,138 - root - INFO - Stopping couchbase-server ... 2013-02-19 13:07:47,138 - root - INFO - Stopping couchbase-server 2013-02-19 13:07:47,139 - root - INFO - Minimum RAM required : 4 GB 2013-02-19 13:07:47,139 - root - INFO - System RAM configured : 2457784 kB 2013-02-19 13:07:47,140 - root - INFO - 2013-02-19 13:07:47,140 - root - INFO - Minimum number of processors required : 4 cores 2013-02-19 13:07:47,141 - root - INFO - Number of processors on the system : 4 cores 2013-02-19 13:07:47,141 - root - INFO - 2013-02-19 13:07:47,142 - root - INFO - Upgrading couchbase-server ... 2013-02-19 13:07:47,142 - root - INFO - /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes 2013-02-19 13:07:47,143 - root - INFO - Automatic mode: running without interactive questions or confirmations. 2013-02-19 13:07:47,143 - root - INFO - Analysing... 2013-02-19 13:07:47,144 - root - INFO - Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat 2013-02-19 13:07:47,144 - root - INFO - Target node: ns_1@10.1.4.24 2013-02-19 13:07:47,145 - root - INFO - Upgrading from 2.0 2013-02-19 13:07:47,145 - root - INFO - 2013-02-19 13:07:47,146 - root - INFO - Couchbase should not be running. 2013-02-19 13:07:47,146 - root - INFO - Please use: /etc/init.d/couchbase-server stop 2013-02-19 13:07:47,147 - root - INFO - 2013-02-19 13:07:47,147 - root - INFO - Database dir: /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:47,148 - root - INFO - 2013-02-19 13:07:47,148 - root - INFO - Buckets to upgrade: default 2013-02-19 13:07:47,149 - root - INFO - 2013-02-19 13:07:47,149 - root - INFO - Checking disk space available for buckets in directory: 2013-02-19 13:07:47,149 - root - INFO - /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:47,149 - root - INFO - Free disk bucket space wanted: 0.0 2013-02-19 13:07:47,150 - root - INFO - Free disk bucket space available: 30138163200 2013-02-19 13:07:47,150 - root - INFO - Free disk space factor: 2.0 2013-02-19 13:07:47,150 - root - INFO - Ok. 2013-02-19 13:07:47,150 - root - INFO - 2013-02-19 13:07:47,151 - root - INFO - Analysis complete. 2013-02-19 13:07:47,151 - root - INFO - 2013-02-19 13:07:47,151 - root - INFO - Copying /opt/couchbase/var/lib/couchbase/config/config.dat 2013-02-19 13:07:47,151 - root - INFO - cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat 2013-02-19 13:07:47,152 - root - INFO - Copying /opt/couchbase/var/lib/couchbase/ip 2013-02-19 13:07:47,152 - root - INFO - cp /opt/couchbase/var/lib/couchbase/ip /opt/couchbase/bin/install/../../var/lib/couchbase/ip 2013-02-19 13:07:47,152 - root - INFO - Copying /opt/couchbase/etc/couchdb/local.ini 2013-02-19 13:07:47,152 - root - INFO - cp /opt/couchbase/etc/couchdb/local.ini /opt/couchbase/bin/install/../../etc/couchdb/local.ini 2013-02-19 13:07:47,152 - root - INFO - Ensuring bucket data directories. 2013-02-19 13:07:47,153 - root - INFO - Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/default 2013-02-19 13:07:47,153 - root - INFO - mkdir -p /opt/couchbase/var/lib/couchbase/data/default 2013-02-19 13:07:47,153 - root - INFO - Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:47,153 - root - INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:47,154 - root - INFO - Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:47,154 - root - INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:47,154 - root - INFO - Upgrading buckets. 2013-02-19 13:07:47,154 - root - INFO - Skipping already converted bucket: /opt/couchbase/var/lib/couchbase/data/default 2013-02-19 13:07:47,154 - root - INFO - 2013-02-19 13:07:47,155 - root - INFO - Done. 2013-02-19 13:07:47,155 - root - INFO - Starting couchbase-server[ OK ] 2013-02-19 13:07:47,155 - root - INFO - 2013-02-19 13:07:47,155 - root - INFO - 2013-02-19 13:07:47,156 - root - INFO - You have successfully installed Couchbase Server. 2013-02-19 13:07:47,156 - root - INFO - Please browse to http://localhost.localdomain:8091/ to configure your server. 2013-02-19 13:07:47,156 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-02-19 13:07:47,156 - root - INFO - 2013-02-19 13:07:47,157 - root - INFO - Please note that you have to update your firewall configuration to 2013-02-19 13:07:47,157 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-02-19 13:07:47,157 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-02-19 13:07:47,157 - root - INFO - 2013-02-19 13:07:47,157 - root - INFO - By using this software you agree to the End User License Agreement. 2013-02-19 13:07:47,158 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-02-19 13:07:47,158 - root - INFO - 2013-02-19 13:07:47,158 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-02-19 13:07:47,158 - root - INFO - {badrpc,nodedown} 2013-02-19 13:07:47,159 - root - INFO - 2013-02-19 13:07:47,475 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:48,132 - root - INFO - command executed successfully 2013-02-19 13:07:48,132 - root - INFO - Stopping couchbase-server ... 2013-02-19 13:07:48,133 - root - INFO - Stopping couchbase-server 2013-02-19 13:07:48,133 - root - INFO - Minimum RAM required : 4 GB 2013-02-19 13:07:48,133 - root - INFO - System RAM configured : 2457784 kB 2013-02-19 13:07:48,134 - root - INFO - 2013-02-19 13:07:48,134 - root - INFO - Minimum number of processors required : 4 cores 2013-02-19 13:07:48,135 - root - INFO - Number of processors on the system : 4 cores 2013-02-19 13:07:48,135 - root - INFO - 2013-02-19 13:07:48,136 - root - INFO - Upgrading couchbase-server ... 2013-02-19 13:07:48,136 - root - INFO - /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes 2013-02-19 13:07:48,137 - root - INFO - Automatic mode: running without interactive questions or confirmations. 2013-02-19 13:07:48,137 - root - INFO - Analysing... 2013-02-19 13:07:48,138 - root - INFO - Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat 2013-02-19 13:07:48,138 - root - INFO - Target node: ns_1@10.1.4.14 2013-02-19 13:07:48,139 - root - INFO - Upgrading from 2.0 2013-02-19 13:07:48,139 - root - INFO - 2013-02-19 13:07:48,140 - root - INFO - Couchbase should not be running. 2013-02-19 13:07:48,140 - root - INFO - Please use: /etc/init.d/couchbase-server stop 2013-02-19 13:07:48,141 - root - INFO - 2013-02-19 13:07:48,141 - root - INFO - Database dir: /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:48,142 - root - INFO - 2013-02-19 13:07:48,142 - root - INFO - Buckets to upgrade: default 2013-02-19 13:07:48,143 - root - INFO - 2013-02-19 13:07:48,143 - root - INFO - Checking disk space available for buckets in directory: 2013-02-19 13:07:48,144 - root - INFO - /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:48,144 - root - INFO - Free disk bucket space wanted: 0.0 2013-02-19 13:07:48,145 - root - INFO - Free disk bucket space available: 30486421504 2013-02-19 13:07:48,145 - root - INFO - Free disk space factor: 2.0 2013-02-19 13:07:48,145 - root - INFO - Ok. 2013-02-19 13:07:48,145 - root - INFO - 2013-02-19 13:07:48,146 - root - INFO - Analysis complete. 2013-02-19 13:07:48,146 - root - INFO - 2013-02-19 13:07:48,146 - root - INFO - Copying /opt/couchbase/var/lib/couchbase/config/config.dat 2013-02-19 13:07:48,146 - root - INFO - cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat 2013-02-19 13:07:48,146 - root - INFO - Copying /opt/couchbase/var/lib/couchbase/ip 2013-02-19 13:07:48,147 - root - INFO - cp /opt/couchbase/var/lib/couchbase/ip /opt/couchbase/bin/install/../../var/lib/couchbase/ip 2013-02-19 13:07:48,147 - root - INFO - Copying /opt/couchbase/etc/couchdb/local.ini 2013-02-19 13:07:48,147 - root - INFO - cp /opt/couchbase/etc/couchdb/local.ini /opt/couchbase/bin/install/../../etc/couchdb/local.ini 2013-02-19 13:07:48,147 - root - INFO - Ensuring bucket data directories. 2013-02-19 13:07:48,148 - root - INFO - Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/default 2013-02-19 13:07:48,148 - root - INFO - mkdir -p /opt/couchbase/var/lib/couchbase/data/default 2013-02-19 13:07:48,148 - root - INFO - Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:48,148 - root - INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:48,149 - root - INFO - Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:48,149 - root - INFO - chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data 2013-02-19 13:07:48,149 - root - INFO - Upgrading buckets. 2013-02-19 13:07:48,149 - root - INFO - Skipping already converted bucket: /opt/couchbase/var/lib/couchbase/data/default 2013-02-19 13:07:48,149 - root - INFO - 2013-02-19 13:07:48,150 - root - INFO - Done. 2013-02-19 13:07:48,150 - root - INFO - Starting couchbase-server[ OK ] 2013-02-19 13:07:48,150 - root - INFO - 2013-02-19 13:07:48,150 - root - INFO - 2013-02-19 13:07:48,151 - root - INFO - You have successfully installed Couchbase Server. 2013-02-19 13:07:48,151 - root - INFO - Please browse to http://localhost.localdomain:8091/ to configure your server. 2013-02-19 13:07:48,151 - root - INFO - Please refer to http://couchbase.com for additional resources. 2013-02-19 13:07:48,151 - root - INFO - 2013-02-19 13:07:48,152 - root - INFO - Please note that you have to update your firewall configuration to 2013-02-19 13:07:48,152 - root - INFO - allow connections to the following ports: 11211, 11210, 11209, 4369, 2013-02-19 13:07:48,152 - root - INFO - 8091, 8092 and from 21100 to 21299. 2013-02-19 13:07:48,152 - root - INFO - 2013-02-19 13:07:48,153 - root - INFO - By using this software you agree to the End User License Agreement. 2013-02-19 13:07:48,153 - root - INFO - See /opt/couchbase/LICENSE.txt. 2013-02-19 13:07:48,153 - root - INFO - 2013-02-19 13:07:48,153 - root - INFO - Stopping couchbase-serverNOTE: shutdown failed 2013-02-19 13:07:48,153 - root - INFO - {badrpc,nodedown} 2013-02-19 13:07:48,154 - root - INFO - 2013-02-19 13:07:48,416 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:48,675 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:49,773 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:49,961 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:51,135 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:51,321 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:52,534 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:52,535 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:54,047 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:54,046 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:55,270 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:55,270 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:56,579 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:56,579 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:57,889 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:57,890 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:59,100 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:07:59,101 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:08:00,452 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:08:00,453 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:08:01,806 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:08:01,807 - root - ERROR - socket error while connecting to http://10.1.4.24:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:08:03,052 - root - ERROR - socket error while connecting to http://10.1.4.14:8091/nodes/self error [Errno 111] Connection refused 2013-02-19 13:08:04,805 - root - WARNING - server 10.1.4.24:8091 status is warmup 2013-02-19 13:08:06,122 - root - WARNING - server 10.1.4.14:8091 status is warmup 2013-02-19 13:08:06,686 - root - WARNING - server 10.1.4.24:8091 status is warmup 2013-02-19 13:08:08,045 - root - WARNING - server 10.1.4.14:8091 status is warmup 2013-02-19 13:08:08,798 - root - WARNING - server 10.1.4.24:8091 status is warmup 2013-02-19 13:08:10,158 - root - WARNING - server 10.1.4.14:8091 status is warmup 2013-02-19 13:08:10,630 - root - WARNING - server 10.1.4.24:8091 status is warmup 2013-02-19 13:08:12,061 - root - WARNING - server 10.1.4.14:8091 status is warmup 2013-02-19 13:08:12,513 - root - WARNING - server 10.1.4.24:8091 status is warmup 2013-02-19 13:08:14,118 - root - WARNING - server 10.1.4.14:8091 status is warmup 2013-02-19 13:08:14,493 - root - WARNING - server 10.1.4.24:8091 status is warmup 2013-02-19 13:08:15,884 - root - WARNING - server 10.1.4.14:8091 status is warmup 2013-02-19 13:08:16,449 - root - WARNING - server 10.1.4.24:8091 status is warmup 2013-02-19 13:08:17,684 - root - WARNING - server 10.1.4.14:8091 status is warmup 2013-02-19 13:08:18,433 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-02-19 13:08:19,554 - root - INFO - sleep for 10 secs. ... 2013-02-19 13:08:19,742 - root - INFO - settings/web params : username=Administrator&password=password&port=8091 2013-02-19 13:08:20,688 - root - INFO - sleep for 10 secs. ... 2013-02-19 13:08:30,690 - root - INFO - sleep for 0 secs. ... 2013-02-19 13:10:05,367 - root - INFO - existing buckets : [u'default'] 2013-02-19 13:10:05,367 - root - INFO - found bucket default 2013-02-19 13:10:10,623 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 13:10:16,597 - root - INFO - Saw ep_queue_size 0 == 0 expected on '10.1.4.24:8091' 2013-02-19 13:10:18,548 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 13:10:24,559 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '10.1.4.24:8091' 2013-02-19 13:10:26,421 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 13:10:31,632 - root - INFO - Saw ep_queue_size 0 == 0 expected on '10.1.4.14:8091' 2013-02-19 13:10:33,128 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 13:10:38,656 - root - INFO - Saw ep_flusher_todo 0 == 0 expected on '10.1.4.14:8091' 2013-02-19 13:10:43,912 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 13:10:50,784 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 13:10:55,434 - root - INFO - 10 items will be verified on default bucket 2013-02-19 13:10:55,675 - root - INFO - 0 items were verified 2013-02-19 13:10:56,459 - root - INFO - 10 items were verified in 1.02416706085 sec.the average number of ops - 9.76401385952 per second 2013-02-19 13:10:58,531 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 13:11:07,493 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 13:11:13,369 - root - INFO - Saw curr_items 10 == 10 expected on '10.1.4.24:8091''10.1.4.14:8091' 2013-02-19 13:11:15,417 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 13:11:23,243 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 13:11:28,982 - root - INFO - Saw vb_active_curr_items 10 == 10 expected on '10.1.4.24:8091''10.1.4.14:8091' 2013-02-19 13:11:30,750 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 13:11:40,053 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 13:11:45,764 - root - INFO - Saw vb_replica_curr_items 10 == 10 expected on '10.1.4.24:8091''10.1.4.14:8091' 2013-02-19 13:11:47,675 - root - INFO - creating direct client 10.1.4.24:11210 default 2013-02-19 13:11:56,052 - root - INFO - creating direct client 10.1.4.14:11210 default 2013-02-19 13:12:01,907 - root - INFO - Saw curr_items_tot 20 == 20 expected on '10.1.4.24:8091''10.1.4.14:8091' 2013-02-19 13:12:01,910 - root - INFO - waiting for sum_of_curr_items == total_items.... 2013-02-19 13:12:10,027 - root - INFO - curr_items from 10.1.4.24:8091 : 5 2013-02-19 13:12:10,028 - root - INFO - vb_pending_num from 10.1.4.24:8091 : 0 2013-02-19 13:12:10,029 - root - INFO - vb_active_num from 10.1.4.24:8091 : 512 2013-02-19 13:12:10,029 - root - INFO - vb_replica_num from 10.1.4.24:8091 : 512 2013-02-19 13:12:10,030 - root - INFO - curr_items from 10.1.4.14:8091 : 5 2013-02-19 13:12:10,030 - root - INFO - vb_pending_num from 10.1.4.14:8091 : 0 2013-02-19 13:12:10,031 - root - INFO - vb_active_num from 10.1.4.14:8091 : 512 2013-02-19 13:12:10,031 - root - INFO - vb_replica_num from 10.1.4.14:8091 : 512 2013-02-19 13:12:10,032 - root - INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 2013-02-19 13:12:10,032 - root - INFO - sum : 10 and sum * (replica_factor + 1) (2) : 20 2013-02-19 13:12:12,093 - root - INFO - curr_items_tot from master: 20 2013-02-19 13:12:12,094 - root - INFO - delta : 0 missing_percentage : 1 replica_factor : 1 2013-02-19 13:13:43,795 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:13:45,528 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:13:47,512 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:13:48,654 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:13:49,446 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:13:50,609 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:14:00,494 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:14:01,460 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:14:02,469 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:14:03,528 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:14:12,577 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:14:13,514 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:14:15,472 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:14:16,506 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:14:24,475 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:14:25,413 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:14:28,480 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:14:29,415 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:14:36,548 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:14:37,627 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:14:40,614 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:14:41,694 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:14:48,710 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:14:49,644 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:14:53,678 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:14:55,013 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:15:01,089 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:15:02,068 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:15:06,042 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:15:06,977 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:15:14,047 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:15:14,898 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:15:18,756 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:15:19,731 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:15:26,599 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:15:27,607 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:15:31,669 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:15:32,793 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:15:38,592 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:15:39,561 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:15:44,237 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:15:45,195 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:15:51,167 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:15:52,145 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:15:56,967 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:15:57,975 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:16:03,967 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:16:05,089 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:16:08,836 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:16:09,832 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:16:16,778 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:16:17,723 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:16:21,627 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:16:22,513 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:16:29,460 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:16:30,542 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:16:33,359 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:16:34,105 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:16:42,127 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:16:43,062 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:16:46,133 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:16:47,261 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:16:54,164 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:16:55,287 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:16:58,354 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:16:59,290 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:17:05,991 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:17:06,971 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:17:10,899 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:17:11,841 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:17:18,801 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:17:19,735 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:17:23,514 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:17:24,496 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:17:31,616 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:17:32,684 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:17:35,442 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:17:36,407 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:17:44,262 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:17:45,234 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:17:48,254 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:17:49,301 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:17:56,332 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:17:57,245 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:18:01,188 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:18:02,200 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:18:09,048 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:18:09,997 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:18:14,061 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:18:15,257 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:18:20,956 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:18:21,834 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:18:26,733 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:18:27,557 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:18:33,711 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000 2013-02-19 13:18:34,728 - root - INFO - (10 rows) expected, (0 rows) returned 2013-02-19 13:18:38,506 - root - INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000 2013-02-19 13:18:39,656 - root - INFO - (10 rows) expected, (0 rows) returned [('/home/andrey/Desktop/Aptana_Studio_3/plugins/org.python.pydev_2.7.0.2012110722/pysrc/pydevd.py', 1397, '', "debugger.run(setup['file'], None, None)"), ('/home/andrey/Desktop/Aptana_Studio_3/plugins/org.python.pydev_2.7.0.2012110722/pysrc/pydevd.py', 1090, 'run', 'pydev_imports.execfile(file, globals, locals) #execute the script'), ('/home/andrey/testrunner_aptana/testrunner/testrunner.py', 321, '', 'result = unittest.TextTestRunner(verbosity=2).run(suite)'), ('/usr/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/suite.py', 70, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/suite.py', 108, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/case.py', 391, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/case.py', 327, 'run', 'testMethod()'), ('pytests/newupgradetests.py', 91, 'offline_cluster_upgrade', 'self.verification(self.servers[:self.nodes_init])'), ('pytests/newupgradebasetest.py', 192, 'verification', 'retry_time=10)'), ('pytests/basetestcase.py', 483, 'perform_verify_queries', 'task.result(wait_time)'), ('lib/tasks/future.py', 161, 'result', 'self.set_exception(TimeoutError())'), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')] Tue Feb 19 13:18:42 2013 ERROR 2013-02-19 13:23:00,395 - root - WARNING - CLEANUP WAS SKIPPED DUE TO FAILURES IN UPGRADE TEST ====================================================================== ERROR: offline_cluster_upgrade (newupgradetests.MultiNodesUpgradeTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/newupgradetests.py", line 91, in offline_cluster_upgrade self.verification(self.servers[:self.nodes_init]) File "pytests/newupgradebasetest.py", line 192, in verification retry_time=10) File "pytests/basetestcase.py", line 488, in perform_verify_queries raise Exception("unable to get expected results for view queries during {0} sec".format(wait_time)) Exception: unable to get expected results for view queries during 300 sec ---------------------------------------------------------------------- Ran 1 test in 2654.713s FAILED (errors=1) summary so far suite newupgradetests.MultiNodesUpgradeTests , pass 0 , fail 1 failures so far... newupgradetests.MultiNodesUpgradeTests.offline_cluster_upgrade testrunner logs, diags and results are available under logs/testrunner-13-Feb-19_12-38-45