erics-mbp:testrunner eric$ ./testrunner -i /Users/eric/servers/tunable-vms.ini -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=1,replicas=1,items=500,doc_ops=create,value_size=1024,active_resident_threshold=90,dgm_run=true,eviction_policy=fullEviction,max_verify=1000 Global Test input params: {'cluster_name': 'tunable-vms', 'ini': '/Users/eric/servers/tunable-vms.ini', 'num_nodes': 6} Logs will be stored at /Users/eric/source/Jenkins-cleanup/testrunner/logs/testrunner-15-Aug-24_14-40-26/test_1 ./testrunner -i /Users/eric/servers/tunable-vms.ini -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=1,replicas=1,items=500,doc_ops=create,value_size=1024,active_resident_threshold=90,dgm_run=true,eviction_policy=fullEviction,max_verify=1000 Test Input params: {'max_verify': '1000', 'value_size': '1024', 'replicas': '1', 'doc_ops': 'create', 'items': '500', 'nodes_in': '1', 'num_nodes': 6, 'cluster_name': 'tunable-vms', 'ini': '/Users/eric/servers/tunable-vms.ini', 'case_number': 1, 'active_resident_threshold': '90', 'eviction_policy': 'fullEviction', 'logs_folder': '/Users/eric/source/Jenkins-cleanup/testrunner/logs/testrunner-15-Aug-24_14-40-26/test_1', 'dgm_run': 'true'} Run before suite setup for rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops 2015-08-24 14:40:26 | WARNING | MainProcess | MainThread | [data_helper.] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... 2015-08-24 14:40:26 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'4.0.0-4047-enterprise'] 2015-08-24 14:40:26 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 rebalance_in_with_ops============== 2015-08-24 14:40:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.24 2015-08-24 14:40:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.25 2015-08-24 14:40:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.26 2015-08-24 14:40:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.27 2015-08-24 14:40:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.28 2015-08-24 14:40:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.29 2015-08-24 14:40:27 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2015-08-24 14:40:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.24:8091 2015-08-24 14:40:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.24:8091 is running 2015-08-24 14:40:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.25:8091 2015-08-24 14:40:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.25:8091 is running 2015-08-24 14:40:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.26:8091 2015-08-24 14:40:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.26:8091 is running 2015-08-24 14:40:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.27:8091 2015-08-24 14:40:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.27:8091 is running 2015-08-24 14:40:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.28:8091 2015-08-24 14:40:38 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.28:8091 is running 2015-08-24 14:40:38 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.29:8091 2015-08-24 14:40:38 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.29:8091 is running 2015-08-24 14:40:38 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 rebalance_in_with_ops ============== 2015-08-24 14:40:38 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.24 2015-08-24 14:40:38 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.25 2015-08-24 14:40:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.26 2015-08-24 14:40:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.27 2015-08-24 14:40:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.28 2015-08-24 14:40:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.29 2015-08-24 14:40:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.24:8091 2015-08-24 14:40:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.24:8091 is running 2015-08-24 14:40:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.25:8091 2015-08-24 14:40:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.25:8091 is running 2015-08-24 14:40:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.26:8091 2015-08-24 14:40:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.26:8091 is running 2015-08-24 14:40:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.27:8091 2015-08-24 14:40:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.27:8091 is running 2015-08-24 14:40:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.28:8091 2015-08-24 14:40:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.28:8091 is running 2015-08-24 14:40:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.29:8091 2015-08-24 14:40:41 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.29:8091 is running 2015-08-24 14:40:41 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 rebalance_in_with_ops ============== Cluster instance shutdown with force 2015-08-24 14:40:41 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2015-08-24 14:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.24:8091:username=Administrator&password=password&port=8091 2015-08-24 14:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-24 14:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.25:8091:username=Administrator&password=password&port=8091 2015-08-24 14:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-24 14:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.26:8091:username=Administrator&password=password&port=8091 2015-08-24 14:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-24 14:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.27:8091:username=Administrator&password=password&port=8091 2015-08-24 14:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-24 14:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.28:8091:username=Administrator&password=password&port=8091 2015-08-24 14:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-24 14:40:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.29:8091:username=Administrator&password=password&port=8091 2015-08-24 14:40:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-24 14:40:43 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2015-08-24 14:40:44 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.24:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=fullEviction&threadsNumber=3&ramQuotaMB=256&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None 2015-08-24 14:40:44 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.02 seconds to create bucket default 2015-08-24 14:40:44 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.106.24 to accept set ops 2015-08-24 14:40:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:40:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:40:56 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 256 2015-08-24 14:40:56 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 rebalance_in_with_ops ============== 2015-08-24 14:40:56 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:40:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:40:58 | INFO | MainProcess | test_thread | [basetestcase._load_all_buckets] resident ratio is 100 greater than 90 for 172.23.106.24 in bucket default. Continue loading to the cluster 2015-08-24 14:40:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:42:07 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:42:19 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:42:31 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:42:35 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:42:55 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:43:08 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:43:21 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:43:33 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:43:45 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:43:57 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:44:09 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:44:22 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:44:34 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:45:04 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:45:17 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:45:29 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:45:41 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:46:04 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:46:16 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:46:29 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:46:43 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:46:57 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:47:09 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:47:23 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:47:36 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:47:49 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:48:17 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:48:30 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:48:44 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:48:58 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:49:11 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:49:24 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:49:37 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:49:59 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:50:12 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:50:27 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:50:43 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:53:09 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:53:18 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:53:18 | INFO | MainProcess | test_thread | [basetestcase._load_all_buckets] DGM state achieved for 172.23.106.24 in bucket default! 2015-08-24 14:53:18 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:53:18 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.25:8091 to cluster 2015-08-24 14:53:18 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.25:8091 to this cluster @172.23.106.24:8091 2015-08-24 14:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.106.25%2Cns_1%40172.23.106.24 2015-08-24 14:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2015-08-24 14:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 % 2015-08-24 14:53:33 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 10.48 % 2015-08-24 14:53:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 22.14 % 2015-08-24 14:53:53 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 26.63 % 2015-08-24 14:54:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 35.55 % 2015-08-24 14:54:13 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 41.93 % 2015-08-24 14:54:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 43.42 % 2015-08-24 14:54:33 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 54.82 % 2015-08-24 14:54:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 61.91 % 2015-08-24 14:54:53 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 67.97 % 2015-08-24 14:55:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 70.70 % 2015-08-24 14:55:13 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 74.35 % 2015-08-24 14:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 78.84 % 2015-08-24 14:55:33 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 83.27 % 2015-08-24 14:55:43 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 86.52 % 2015-08-24 14:55:53 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 91.28 % 2015-08-24 14:56:03 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 95.64 % 2015-08-24 14:56:13 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 170.677115202 sec about to verify cluster stats 2015-08-24 14:56:13 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'4.0.0-4047-enterprise', u'4.0.0-4047-enterprise'] 2015-08-24 14:56:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:172.23.106.25 port:8091 ssh_username:root, ip:172.23.106.24 port:8091 ssh_username:root] verify_cluster_stats about to wait for stats 2015-08-24 14:56:14 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'4.0.0-4047-enterprise', u'4.0.0-4047-enterprise'] 2015-08-24 14:56:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:172.23.106.25 port:8091 ssh_username:root, ip:172.23.106.24 port:8091 ssh_username:root] 2015-08-24 14:56:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default for ep_queue_size on ip:172.23.106.24 port:8091 ssh_username:root got 0 2015-08-24 14:56:15 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.106.24:8091',default bucket 2015-08-24 14:56:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default for ep_queue_size on ip:172.23.106.25 port:8091 ssh_username:root got 0 2015-08-24 14:56:15 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.106.25:8091',default bucket verify_cluster_stats about to verify all buckets 2015-08-24 14:56:15 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 14:56:16 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default 2015-08-24 14:56:18 | INFO | MainProcess | test_thread | [task.__init__] 1000 items will be verified on default bucket 2015-08-24 14:56:18 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified 2015-08-24 14:56:19 | INFO | MainProcess | load_gen_task | [task.has_next] 1000 items were verified in 1.44817399979 sec.the average number of ops - 690.524302751 per second verify_cluster_stats about to verify stats all buckets 2015-08-24 14:56:19 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'4.0.0-4047-enterprise', u'4.0.0-4047-enterprise'] 2015-08-24 14:56:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:172.23.106.25 port:8091 ssh_username:root, ip:172.23.106.24 port:8091 ssh_username:root] _verify_stats_all_buckets items 1000749 2015-08-24 14:56:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500374 2015-08-24 14:56:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-24 14:56:21 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items 1000749 == 1000749 expected on '172.23.106.24:8091''172.23.106.25:8091',default bucket 2015-08-24 14:56:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500374 2015-08-24 14:56:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-24 14:56:21 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 1000749 == 1000749 expected on '172.23.106.24:8091''172.23.106.25:8091',default bucket 2015-08-24 14:56:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 2015-08-24 14:56:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:56:22 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket 2015-08-24 14:56:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 2015-08-24 14:56:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:56:23 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:56:28 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:56:28 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:56:33 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:56:33 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:56:38 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:56:38 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:56:43 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:56:43 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:56:48 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:56:48 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:56:53 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:56:53 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:56:58 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:56:58 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:57:03 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:57:03 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:57:09 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:57:09 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:57:14 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:57:14 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:57:19 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:57:19 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:57:24 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:57:24 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:57:29 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:57:29 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:57:34 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:57:34 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:57:39 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:57:39 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:57:44 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:57:44 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:57:49 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:57:49 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:57:54 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:57:54 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:58:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:58:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:58:05 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:58:05 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:58:10 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:58:10 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:58:15 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:58:15 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500377 for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-24 14:58:20 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_replica_curr_items 1000751 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000751 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-24 14:58:20 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001500 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket [('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 781, '__bootstrap', 'self.__bootstrap_inner()'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 808, '__bootstrap_inner', 'self.run()'), ('./testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py', 395, '__call__', 'return self.run(*args, **kwds)'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py', 331, 'run', 'testMethod()'), ('pytests/rebalance/rebalancein.py', 171, 'rebalance_in_with_ops', 'self.verify_cluster_stats(self.servers[:self.nodes_in + self.nodes_init])'), ('pytests/basetestcase.py', 770, 'verify_cluster_stats', 'self._verify_stats_all_buckets(servers, timeout=(timeout or 120))'), ('pytests/basetestcase.py', 512, '_verify_stats_all_buckets', 'task.result(timeout)'), ('lib/tasks/future.py', 162, 'result', 'self.set_exception(TimeoutError())'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')] Mon Aug 24 14:58:21 2015 2015-08-24 14:58:21 | ERROR | MainProcess | test_thread | [basetestcase._verify_stats_all_buckets] unable to get expected stats for any node! Print taps for all nodes: > /Users/eric/source/Jenkins-cleanup/testrunner/pytests/basetestcase.py(519)_verify_stats_all_buckets() -> rest = RestConnection(self.master) (Pdb) c 2015-08-24 15:00:34 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-24 15:00:35 | INFO | MainProcess | test_thread | [rebalance_helper.print_taps_from_all_nodes] getting tap stats... for 172.23.106.24 2015-08-24 15:00:35 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default 2015-08-24 15:00:35 | INFO | MainProcess | test_thread | [rebalance_helper.print_taps_from_all_nodes] getting tap stats... for 172.23.106.25 ERROR 2015-08-24 15:00:35 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 rebalance_in_with_ops ============== 2015-08-24 15:00:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 172.23.106.24 2015-08-24 15:00:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ... 2015-08-24 15:00:37 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.106.24 2015-08-24 15:00:37 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2015-08-24 15:00:37 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.24 existing buckets : [] 2015-08-24 15:00:37 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.25 2015-08-24 15:00:37 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.26 2015-08-24 15:00:38 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.27 2015-08-24 15:00:38 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.28 2015-08-24 15:00:38 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.29 2015-08-24 15:00:38 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2015-08-24 15:00:38 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%40172.23.106.25&user=Administrator&knownNodes=ns_1%40172.23.106.25%2Cns_1%40172.23.106.24 2015-08-24 15:00:39 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2015-08-24 15:00:39 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 % 2015-08-24 15:00:49 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 10.0961909294 seconds 2015-08-24 15:00:49 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.106.24 port:8091 ssh_username:root ? [(u'ns_1@172.23.106.25', 8091)] 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.24:8091 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.24:8091 is running 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.25:8091 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.25:8091 is running 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.26:8091 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.26:8091 is running 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.27:8091 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.27:8091 is running 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.28:8091 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.28:8091 is running 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.29:8091 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.29:8091 is running 2015-08-24 15:00:59 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 rebalance_in_with_ops ============== Cluster instance shutdown with force ====================================================================== ERROR: rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/rebalance/rebalancein.py", line 171, in rebalance_in_with_ops self.verify_cluster_stats(self.servers[:self.nodes_in + self.nodes_init]) File "pytests/basetestcase.py", line 770, in verify_cluster_stats self._verify_stats_all_buckets(servers, timeout=(timeout or 120)) File "pytests/basetestcase.py", line 519, in _verify_stats_all_buckets rest = RestConnection(self.master) Exception: unable to get expected stats during 120 sec ---------------------------------------------------------------------- Ran 1 test in 1232.634s FAILED (errors=1)