./testrunner -i /Users/eric/servers/tunable-vms.ini -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=1,replicas=1,items=500,doc_ops=create,value_size=1024,active_resident_threshold=90,dgm_run=true,eviction_policy=fullEviction,max_verify=1000 Global Test input params: {'cluster_name': 'tunable-vms', 'ini': '/Users/eric/servers/tunable-vms.ini', 'num_nodes': 6} Logs will be stored at /Users/eric/source/Jenkins-cleanup/testrunner/logs/testrunner-15-Aug-25_09-15-56/test_1 ./testrunner -i /Users/eric/servers/tunable-vms.ini -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=1,replicas=1,items=500,doc_ops=create,value_size=1024,active_resident_threshold=90,dgm_run=true,eviction_policy=fullEviction,max_verify=1000 Test Input params: {'max_verify': '1000', 'value_size': '1024', 'replicas': '1', 'doc_ops': 'create', 'items': '500', 'nodes_in': '1', 'num_nodes': 6, 'cluster_name': 'tunable-vms', 'ini': '/Users/eric/servers/tunable-vms.ini', 'case_number': 1, 'active_resident_threshold': '90', 'eviction_policy': 'fullEviction', 'logs_folder': '/Users/eric/source/Jenkins-cleanup/testrunner/logs/testrunner-15-Aug-25_09-15-56/test_1', 'dgm_run': 'true'} Run before suite setup for rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops 2015-08-25 09:15:56 | WARNING | MainProcess | MainThread | [data_helper.] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... 2015-08-25 09:15:56 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'4.0.0-4047-enterprise'] 2015-08-25 09:15:56 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 rebalance_in_with_ops============== 2015-08-25 09:15:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.24 2015-08-25 09:15:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.25 2015-08-25 09:15:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.26 2015-08-25 09:15:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.27 2015-08-25 09:15:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.28 2015-08-25 09:15:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.29 2015-08-25 09:15:57 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.24:8091 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.24:8091 is running 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.25:8091 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.25:8091 is running 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.26:8091 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.26:8091 is running 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.27:8091 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.27:8091 is running 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.28:8091 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.28:8091 is running 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.29:8091 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.29:8091 is running 2015-08-25 09:16:07 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 rebalance_in_with_ops ============== 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.24 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.25 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.26 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.27 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.28 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.29 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.24:8091 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.24:8091 is running 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.25:8091 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.25:8091 is running 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.26:8091 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.26:8091 is running 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.27:8091 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.27:8091 is running 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.28:8091 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.28:8091 is running 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.29:8091 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.29:8091 is running 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 rebalance_in_with_ops ============== Cluster instance shutdown with force 2015-08-25 09:16:08 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2015-08-25 09:16:09 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.24:8091:username=Administrator&password=password&port=8091 2015-08-25 09:16:09 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-25 09:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.25:8091:username=Administrator&password=password&port=8091 2015-08-25 09:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-25 09:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.26:8091:username=Administrator&password=password&port=8091 2015-08-25 09:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-25 09:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.27:8091:username=Administrator&password=password&port=8091 2015-08-25 09:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-25 09:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.28:8091:username=Administrator&password=password&port=8091 2015-08-25 09:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-25 09:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.29:8091:username=Administrator&password=password&port=8091 2015-08-25 09:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069 2015-08-25 09:16:10 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2015-08-25 09:16:11 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.24:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=fullEviction&threadsNumber=3&ramQuotaMB=256&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None 2015-08-25 09:16:11 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket default 2015-08-25 09:16:11 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.106.24 to accept set ops 2015-08-25 09:16:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:16:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:16:22 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 256 2015-08-25 09:16:22 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 rebalance_in_with_ops ============== 2015-08-25 09:16:23 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:16:24 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:16:24 | INFO | MainProcess | test_thread | [basetestcase._load_all_buckets] resident ratio is 100 greater than 90 for 172.23.106.24 in bucket default. Continue loading to the cluster 2015-08-25 09:16:24 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:17:43 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:17:56 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:18:09 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:18:22 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:18:33 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:18:45 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:18:57 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:19:10 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:19:22 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:19:35 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:19:46 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:20:00 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:20:13 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:20:25 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:20:38 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:20:53 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:21:07 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:21:22 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:21:35 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:22:11 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:22:25 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:22:38 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:23:08 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:23:20 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:23:33 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:24:03 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:24:15 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:24:28 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:24:41 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:24:54 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:25:06 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:25:18 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:25:30 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:25:43 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:25:56 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:26:10 | INFO | MainProcess | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:26:32 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:26:32 | INFO | MainProcess | test_thread | [basetestcase._load_all_buckets] DGM state achieved for 172.23.106.24 in bucket default! num items is 500 create range is 501 to 750 2015-08-25 09:26:32 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:26:33 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.25:8091 to cluster 2015-08-25 09:26:33 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.25:8091 to this cluster @172.23.106.24:8091 2015-08-25 09:26:39 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.106.25%2Cns_1%40172.23.106.24 2015-08-25 09:26:39 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2015-08-25 09:26:39 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 % 2015-08-25 09:26:49 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 10.94 % 2015-08-25 09:26:59 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 23.50 % 2015-08-25 09:27:09 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 26.56 % 2015-08-25 09:27:19 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 27.86 % 2015-08-25 09:27:29 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 34.83 % 2015-08-25 09:27:39 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 43.88 % 2015-08-25 09:27:49 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 51.76 % 2015-08-25 09:27:59 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 61.46 % 2015-08-25 09:28:09 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 67.97 % 2015-08-25 09:28:19 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 71.16 % 2015-08-25 09:28:29 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 75.07 % 2015-08-25 09:28:39 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 79.43 % 2015-08-25 09:28:49 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 82.94 % 2015-08-25 09:28:59 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 87.30 % 2015-08-25 09:29:09 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 91.54 % 2015-08-25 09:29:19 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 95.51 % 2015-08-25 09:29:29 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 170.699526072 sec about to verify cluster stats 2015-08-25 09:29:30 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'4.0.0-4047-enterprise', u'4.0.0-4047-enterprise'] 2015-08-25 09:29:30 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:172.23.106.25 port:8091 ssh_username:root, ip:172.23.106.24 port:8091 ssh_username:root] verify_cluster_stats about to wait for stats 2015-08-25 09:29:30 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'4.0.0-4047-enterprise', u'4.0.0-4047-enterprise'] 2015-08-25 09:29:30 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:172.23.106.25 port:8091 ssh_username:root, ip:172.23.106.24 port:8091 ssh_username:root] 2015-08-25 09:29:31 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default for ep_queue_size on ip:172.23.106.24 port:8091 ssh_username:root got 0 2015-08-25 09:29:31 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.106.24:8091',default bucket 2015-08-25 09:29:31 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default for ep_queue_size on ip:172.23.106.25 port:8091 ssh_username:root got 0 2015-08-25 09:29:31 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.106.25:8091',default bucket verify_cluster_stats about to verify all buckets 2015-08-25 09:29:32 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:29:33 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default 2015-08-25 09:29:35 | INFO | MainProcess | test_thread | [task.__init__] 1000 items will be verified on default bucket 2015-08-25 09:29:35 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified 2015-08-25 09:29:37 | INFO | MainProcess | load_gen_task | [task.has_next] 1000 items were verified in 2.33657121658 sec.the average number of ops - 427.977190081 per second verify_cluster_stats about to verify stats all buckets 2015-08-25 09:29:37 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'4.0.0-4047-enterprise', u'4.0.0-4047-enterprise'] 2015-08-25 09:29:38 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:172.23.106.25 port:8091 ssh_username:root, ip:172.23.106.24 port:8091 ssh_username:root] _verify_stats_all_buckets items 1000749 2015-08-25 09:29:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 2015-08-25 09:29:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:29:39 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket 2015-08-25 09:29:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 2015-08-25 09:29:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:29:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket 2015-08-25 09:29:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default for vb_replica_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500375 2015-08-25 09:29:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default for vb_replica_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500374 2015-08-25 09:29:41 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_replica_curr_items 1000749 == 1000749 expected on '172.23.106.24:8091''172.23.106.25:8091',default bucket 2015-08-25 09:29:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 2015-08-25 09:29:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:29:42 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:29:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:29:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:29:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:29:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:29:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:29:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:01 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:01 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:06 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:06 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:06 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:11 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:11 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:11 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:16 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:16 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:16 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:21 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:21 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:21 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:26 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:26 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:26 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:31 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:31 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:31 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:36 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:37 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:37 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:42 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:42 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:42 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:47 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:47 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:47 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:52 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:52 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:52 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:57 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:30:57 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:30:57 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:02 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:02 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:31:02 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:07 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:08 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:31:08 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:13 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:13 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:31:13 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:18 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:18 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:31:18 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:24 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:24 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:31:24 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:29 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:29 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:31:29 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:34 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for vb_active_curr_items on ip:172.23.106.24 port:8091 ssh_username:root got 500379 for vb_active_curr_items on ip:172.23.106.25 port:8091 ssh_username:root got 500375 2015-08-25 09:31:34 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: vb_active_curr_items 1000754 == 1000749 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket for curr_items_tot on ip:172.23.106.24 port:8091 ssh_username:root got 1000754 for curr_items_tot on ip:172.23.106.25 port:8091 ssh_username:root got 1000749 2015-08-25 09:31:34 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready saw: curr_items_tot 2001503 == 2001498 and expected on '172.23.106.24:8091''172.23.106.25:8091', default bucket [('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 781, '__bootstrap', 'self.__bootstrap_inner()'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 808, '__bootstrap_inner', 'self.run()'), ('./testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py', 395, '__call__', 'return self.run(*args, **kwds)'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py', 331, 'run', 'testMethod()'), ('pytests/rebalance/rebalancein.py', 174, 'rebalance_in_with_ops', 'self.verify_cluster_stats(self.servers[:self.nodes_in + self.nodes_init])'), ('pytests/basetestcase.py', 770, 'verify_cluster_stats', 'self._verify_stats_all_buckets(servers, timeout=(timeout or 120))'), ('pytests/basetestcase.py', 512, '_verify_stats_all_buckets', 'task.result(timeout)'), ('lib/tasks/future.py', 162, 'result', 'self.set_exception(TimeoutError())'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')] Tue Aug 25 09:31:38 2015 2015-08-25 09:31:38 | ERROR | MainProcess | test_thread | [basetestcase._verify_stats_all_buckets] unable to get expected stats for any node! Print taps for all nodes: > /Users/eric/source/Jenkins-cleanup/testrunner/pytests/basetestcase.py(519)_verify_stats_all_buckets() -> rest = RestConnection(self.master) (Pdb) c 2015-08-25 09:43:38 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.24:11210 default 2015-08-25 09:43:38 | INFO | MainProcess | test_thread | [rebalance_helper.print_taps_from_all_nodes] getting tap stats... for 172.23.106.24 2015-08-25 09:43:38 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.25:11210 default 2015-08-25 09:43:38 | INFO | MainProcess | test_thread | [rebalance_helper.print_taps_from_all_nodes] getting tap stats... for 172.23.106.25 ERROR 2015-08-25 09:43:38 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 rebalance_in_with_ops ============== 2015-08-25 09:43:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 172.23.106.24 2015-08-25 09:43:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ... 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.106.24 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.24 existing buckets : [] 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.25 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.26 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.27 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.28 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.29 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%40172.23.106.25&user=Administrator&knownNodes=ns_1%40172.23.106.25%2Cns_1%40172.23.106.24 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2015-08-25 09:43:41 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 % 2015-08-25 09:43:51 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 10.077147007 seconds 2015-08-25 09:43:51 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2015-08-25 09:44:01 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.106.24 port:8091 ssh_username:root ? [(u'ns_1@172.23.106.25', 8091)] 2015-08-25 09:44:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.24:8091 2015-08-25 09:44:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.24:8091 is running 2015-08-25 09:44:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.25:8091 2015-08-25 09:44:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.25:8091 is running 2015-08-25 09:44:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.26:8091 2015-08-25 09:44:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.26:8091 is running 2015-08-25 09:44:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.27:8091 2015-08-25 09:44:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.27:8091 is running 2015-08-25 09:44:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.28:8091 2015-08-25 09:44:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.28:8091 is running 2015-08-25 09:44:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.29:8091 2015-08-25 09:44:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.29:8091 is running 2015-08-25 09:44:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 rebalance_in_with_ops ============== Cluster instance shutdown with force ====================================================================== ERROR: rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/rebalance/rebalancein.py", line 174, in rebalance_in_with_ops self.verify_cluster_stats(self.servers[:self.nodes_in + self.nodes_init]) File "pytests/basetestcase.py", line 770, in verify_cluster_stats self._verify_stats_all_buckets(servers, timeout=(timeout or 120)) File "pytests/basetestcase.py", line 519, in _verify_stats_all_buckets rest = RestConnection(self.master) Exception: unable to get expected stats during 120 sec ---------------------------------------------------------------------- Ran 1 test in 1685.230s