Test 11 : MB-13185
+ ./testrunner -i INI_FILE.ini -p '{test_params}' -t xdcr.rebalanceXDCR.Rebalance.async_rebalance_in,items=50000,rdirection=unidirection,ctopology=chain,update=C1,delete=C1,rebalance=C2,num_rebalance=1,replication_type=xmem
async_rebalance_in (xdcr.rebalanceXDCR.Rebalance) ... Global Test input params:
{'cluster_name': 'INI_FILE', 'ini': 'INI_FILE.ini', 'num_nodes': 6}
Logs will be stored at /jenkins/workspace/cen006-p0-xxdcr-vset01-00-dev-sanity/logs/testrunner-15-Mar-03_14-48-40/test_1

./testrunner -i INI_FILE.ini {test_params} -t xdcr.rebalanceXDCR.Rebalance.async_rebalance_in,items=50000,rdirection=unidirection,ctopology=chain,update=C1,delete=C1,rebalance=C2,num_rebalance=1,replication_type=xmem

Test Input params:
{'case_number': 1, 'items': '50000', 'replication_type': 'xmem', 'update': 'C1', 'num_nodes': 6, 'cluster_name': 'INI_FILE', 'ctopology': 'chain', 'rdirection': 'unidirection', 'ini': 'INI_FILE.ini', 'num_rebalance': '1', 'logs_folder': '/jenkins/workspace/cen006-p0-xxdcr-vset01-00-dev-sanity/logs/testrunner-15-Mar-03_14-48-40/test_1', 'rebalance': 'C2', 'delete': 'C1'}
Run before suite setup for xdcr.rebalanceXDCR.Rebalance.async_rebalance_in
2015-03-03 14:48:40 | INFO | MainProcess | test_thread | [xdcrnewbasetests.setUp] ==== XDCRNewbasetests setup is started for test #1 async_rebalance_in ====
2015-03-03 14:48:40 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] removing xdcr/nodes settings
2015-03-03 14:48:40 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] cleanup [ip:172.23.107.170 port:8091 ssh_username:root, ip:172.23.107.171 port:8091 ssh_username:root]
2015-03-03 14:48:40 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.107.170
2015-03-03 14:48:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.170:8091
2015-03-03 14:48:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.170:8091 is running
2015-03-03 14:48:40 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.107.171
2015-03-03 14:48:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.171:8091
2015-03-03 14:48:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.171:8091 is running
2015-03-03 14:48:40 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] removing xdcr/nodes settings
2015-03-03 14:48:41 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] cleanup [ip:172.23.107.120 port:8091 ssh_username:root, ip:172.23.107.169 port:8091 ssh_username:root]
2015-03-03 14:48:41 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.107.120
2015-03-03 14:48:41 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes
2015-03-03 14:48:41 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : password=welcome&ejectedNodes=ns_1%40172.23.107.169&user=Administrator&knownNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.107.120
2015-03-03 14:48:41 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started
2015-03-03 14:48:41 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-03-03 14:48:51 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 10.0540139675 seconds 
2015-03-03 14:48:51 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance...
2015-03-03 14:49:01 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.107.120 port:8091 ssh_username:root ? [(u'ns_1@172.23.107.169', 8091)]
2015-03-03 14:49:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.120:8091
2015-03-03 14:49:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.120:8091 is running
2015-03-03 14:49:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.107.169
2015-03-03 14:49:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.169:8091
2015-03-03 14:49:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.169:8091 is running
2015-03-03 14:49:01 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__init_clusters] Initializing all clusters...
2015-03-03 14:49:01 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.170:8091: True content: [3,2] command: cluster_compat_mode:get_compat_version().
2015-03-03 14:49:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.170:8091:username=Administrator&password=welcome&port=8091
2015-03-03 14:49:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069&username=Administrator&password=welcome
2015-03-03 14:49:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.171:8091:username=Administrator&password=welcome&port=8091
2015-03-03 14:49:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069&username=Administrator&password=welcome
2015-03-03 14:49:02 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.171:8091 to cluster
2015-03-03 14:49:02 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.171:8091 to this cluster @172.23.107.170:8091
2015-03-03 14:49:07 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=welcome&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.171%2Cns_1%40172.23.107.170
2015-03-03 14:49:07 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2015-03-03 14:49:07 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-03-03 14:49:17 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.0367119312 sec
2015-03-03 14:49:17 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.170:8091: True content: ok command: ale:set_loglevel(xdcr_trace, debug).
2015-03-03 14:49:17 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.171:8091: True content: ok command: ale:set_loglevel(xdcr_trace, debug).
2015-03-03 14:49:17 | INFO | MainProcess | test_thread | [xdcrnewbasetests.init_cluster] Audit status on C1 is True
2015-03-03 14:49:17 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.120:8091: True content: [3,2] command: cluster_compat_mode:get_compat_version().
2015-03-03 14:49:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.120:8091:username=Administrator&password=welcome&port=8091
2015-03-03 14:49:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069&username=Administrator&password=welcome
2015-03-03 14:49:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.169:8091:username=Administrator&password=welcome&port=8091
2015-03-03 14:49:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2069&username=Administrator&password=welcome
2015-03-03 14:49:19 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.169:8091 to cluster
2015-03-03 14:49:19 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.169:8091 to this cluster @172.23.107.120:8091
2015-03-03 14:49:23 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=welcome&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.107.120
2015-03-03 14:49:23 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2015-03-03 14:49:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-03-03 14:49:33 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.0328581333 sec
2015-03-03 14:49:33 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.120:8091: True content: ok command: ale:set_loglevel(xdcr_trace, debug).
2015-03-03 14:49:33 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.169:8091: True content: ok command: ale:set_loglevel(xdcr_trace, debug).
2015-03-03 14:49:33 | INFO | MainProcess | test_thread | [xdcrnewbasetests.init_cluster] Audit status on C2 is True
2015-03-03 14:49:33 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.107.170:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=2069&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None
2015-03-03 14:49:33 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket default
2015-03-03 14:49:33 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.170 to accept set ops
2015-03-03 14:49:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:49:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:49:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:49:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:49:37 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 2069
2015-03-03 14:49:37 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.107.120:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=2069&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None
2015-03-03 14:49:37 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.05 seconds to create bucket default
2015-03-03 14:49:37 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.120 to accept set ops
2015-03-03 14:49:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.120:11210 default
2015-03-03 14:49:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.169:11210 default
2015-03-03 14:49:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.120:11210 default
2015-03-03 14:49:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.169:11210 default
2015-03-03 14:49:43 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 2069
2015-03-03 14:49:43 | INFO | MainProcess | test_thread | [rest_client.set_internalSetting] Update internal setting xdcrCheckpointInterval=60
2015-03-03 14:49:43 | INFO | MainProcess | test_thread | [rest_client.set_internalSetting] Update internal setting xdcrCheckpointInterval=60
2015-03-03 14:49:43 | INFO | MainProcess | test_thread | [xdcrnewbasetests.setUp] ==== XDCRNewbasetests setup is finished for test #1 async_rebalance_in ====
2015-03-03 14:49:43 | INFO | MainProcess | test_thread | [rest_client.add_remote_cluster] adding remote cluster hostname:172.23.107.120:8091 with username:password Administrator:welcome name:remote_cluster_C1-C2 to source node: 172.23.107.170:8091
2015-03-03 14:49:43 | INFO | MainProcess | test_thread | [rest_client.start_replication] starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {}
2015-03-03 14:49:43 | INFO | MainProcess | test_thread | [rest_client.start_replication] Replication created with id: replicationSpec/acbb10af4ac017768cfb4bffb6a7a245/default/default
2015-03-03 14:49:43 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:49:43 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:49:44 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 0 with exp:0
2015-03-03 14:50:02 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 50000 with exp:0
2015-03-03 14:50:02 | INFO | MainProcess | test_thread | [xdcrnewbasetests.sleep] sleep for 10 secs.  ...
2015-03-03 14:50:12 | INFO | MainProcess | test_thread | [xdcrnewbasetests.async_rebalance_in] Starting rebalance-in nodes:[ip:172.23.105.45 port:8091 ssh_username:root] at C2 cluster 172.23.107.120
2015-03-03 14:50:13 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.45:8091 to cluster
2015-03-03 14:50:13 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.45:8091 to this cluster @172.23.107.120:8091
2015-03-03 14:50:18 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=welcome&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.45%2Cns_1%40172.23.107.169%2Cns_1%40172.23.107.120
2015-03-03 14:50:18 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2015-03-03 14:50:18 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-03-03 14:50:28 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 12.05 %
2015-03-03 14:50:38 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 27.10 %
2015-03-03 14:50:48 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 43.66 %
2015-03-03 14:50:58 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 59.55 %
2015-03-03 14:51:08 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 76.22 %
2015-03-03 14:51:18 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 92.07 %
2015-03-03 14:51:28 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 70.1818230152 sec
2015-03-03 14:51:28 | INFO | MainProcess | test_thread | [xdcrnewbasetests.perform_update_delete] Updating keys @ C1
2015-03-03 14:51:28 | INFO | MainProcess | test_thread | [xdcrnewbasetests.async_update_delete] At bucket 'default' @ C1: operation: update, key range 0 - 15000
2015-03-03 14:51:28 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:51:28 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:51:56 | INFO | MainProcess | test_thread | [xdcrnewbasetests.perform_update_delete] Deleting keys @ C1
2015-03-03 14:51:56 | INFO | MainProcess | test_thread | [xdcrnewbasetests.async_update_delete] At bucket 'default' @ C1: operation: delete, key range 35000 - 50000
2015-03-03 14:51:56 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:51:56 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:52:09 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__merge_all_buckets] Merging keys for replication Replication C1:default -> C2:default
2015-03-03 14:52:10 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__merge_keys] Destination bucket's kv_store now has 35000 valid keys and 0 deleted keys
2015-03-03 14:52:10 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'172.23.107.170', 'password': 'welcome', 'port': u'8091'}
2015-03-03 14:52:10 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:52:10 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.170', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2015-03-03 14:52:10 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.170', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (3030287981, 0, '')
2015-03-03 14:52:10 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'172.23.107.171', 'password': 'welcome', 'port': u'8091'}
2015-03-03 14:52:10 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:52:11 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.171', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2015-03-03 14:52:11 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.171', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (2579630210, 0, '')
2015-03-03 14:52:11 | INFO | MainProcess | test_thread | [xdcrnewbasetests.run_expiry_pager] wait for expiry pager to run on all these nodes
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'172.23.105.45', 'password': 'welcome', 'port': u'8091'}
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.45:11210 default
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.105.45', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.105.45', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (463904792, 0, '')
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'172.23.107.120', 'password': 'welcome', 'port': u'8091'}
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.120:11210 default
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.120', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.120', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (754888544, 0, '')
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'172.23.107.169', 'password': 'welcome', 'port': u'8091'}
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.169:11210 default
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.169', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.169', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (2443004396, 0, '')
2015-03-03 14:52:21 | INFO | MainProcess | test_thread | [xdcrnewbasetests.run_expiry_pager] wait for expiry pager to run on all these nodes
2015-03-03 14:52:32 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:52:32 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.107.170:8091',default bucket
2015-03-03 14:52:32 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:52:32 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.107.171:8091',default bucket
2015-03-03 14:52:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.120:11210 default
2015-03-03 14:52:33 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.107.120:8091',default bucket
2015-03-03 14:52:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.169:11210 default
2015-03-03 14:52:33 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.107.169:8091',default bucket
2015-03-03 14:52:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.45:11210 default
2015-03-03 14:52:34 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.105.45:8091',default bucket
2015-03-03 14:52:34 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Waiting for Outbound mutation to be zero on cluster node: 172.23.107.170
2015-03-03 14:52:34 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Current Outbound mutations on cluster node: 172.23.107.170 for bucket default is 0
2015-03-03 14:52:34 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Waiting for Outbound mutation to be zero on cluster node: 172.23.107.120
2015-03-03 14:52:34 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Current Outbound mutations on cluster node: 172.23.107.120 for bucket default is 0
2015-03-03 14:52:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:52:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:52:35 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items 35000 == 35000 expected on '172.23.107.170:8091''172.23.107.171:8091',default bucket
2015-03-03 14:52:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:52:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:52:35 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 35000 == 35000 expected on '172.23.107.170:8091''172.23.107.171:8091',default bucket
2015-03-03 14:52:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:52:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:52:36 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_replica_curr_items 35000 == 35000 expected on '172.23.107.170:8091''172.23.107.171:8091',default bucket
2015-03-03 14:52:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.120:11210 default
2015-03-03 14:52:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.169:11210 default
2015-03-03 14:52:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.45:11210 default
2015-03-03 14:52:37 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:52:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.120:11210 default
2015-03-03 14:52:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.169:11210 default
2015-03-03 14:52:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.45:11210 default
2015-03-03 14:52:38 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:52:43 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:52:43 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:52:48 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:52:48 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:52:53 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:52:53 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:52:58 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:52:58 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:03 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:03 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:08 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:08 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:13 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:13 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:18 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:18 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:23 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:23 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:28 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:29 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:34 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:34 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:39 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:39 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:44 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:44 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:49 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:49 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:54 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:54 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:59 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:53:59 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:04 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:04 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:09 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:09 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:14 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:14 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:19 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:19 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:24 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:24 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:29 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:29 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:34 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:45 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:45 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:54:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:05 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:05 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:10 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:10 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:15 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:15 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:20 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:20 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:25 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:25 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
[('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('./testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib64/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib64/python2.7/unittest/case.py', 433, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.7/unittest/case.py', 369, 'run', 'testMethod()'), ('pytests/xdcr/rebalanceXDCR.py', 37, 'async_rebalance_in', 'self.verify_results()'), ('pytests/xdcr/xdcrnewbasetests.py', 2874, 'verify_results', 'dest_key_count_ok = dest_cluster.verify_items_count()'), ('pytests/xdcr/xdcrnewbasetests.py', 2055, 'verify_items_count', 'task.result(timeout)'), ('lib/tasks/future.py', 162, 'result', 'self.set_exception(TimeoutError())'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
Tue Mar  3 14:55:37 2015
2015-03-03 14:55:37 | ERROR | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] ERROR: Timed-out waiting for active item count to match
2015-03-03 14:55:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.120:11210 default
2015-03-03 14:55:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.169:11210 default
2015-03-03 14:55:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.45:11210 default
2015-03-03 14:55:38 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:41 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:43 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:46 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:48 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:51 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:53 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:56 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:55:58 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:01 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:03 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:06 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:08 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:12 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:14 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:17 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:19 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:22 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:24 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:27 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:29 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:32 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:34 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:37 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:39 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:42 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:44 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:47 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:49 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:52 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:54 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:57 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:56:59 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:02 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:04 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:07 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:09 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:12 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:14 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:17 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:19 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:22 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:25 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:28 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:33 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:38 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:43 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:45 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:48 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:53 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:57:58 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:03 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:05 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:08 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:10 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:13 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:15 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:18 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:20 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:23 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:25 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:28 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:33 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:36 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
[('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('./testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib64/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib64/python2.7/unittest/case.py', 433, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.7/unittest/case.py', 369, 'run', 'testMethod()'), ('pytests/xdcr/rebalanceXDCR.py', 37, 'async_rebalance_in', 'self.verify_results()'), ('pytests/xdcr/xdcrnewbasetests.py', 2874, 'verify_results', 'dest_key_count_ok = dest_cluster.verify_items_count()'), ('pytests/xdcr/xdcrnewbasetests.py', 2071, 'verify_items_count', 'task.result(timeout)'), ('lib/tasks/future.py', 162, 'result', 'self.set_exception(TimeoutError())'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
Tue Mar  3 14:58:37 2015
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.120 with username : root password : couchbase ssh_key: 
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.120
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [xdcrnewbasetests.run_cbvdiff] Executing cbvdiff for bucket default
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.120: sudo cat /proc/cpuinfo
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.120: df -Th
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.120: sudo cat /proc/meminfo
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.120: hostname
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.120: hostname -d
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2015-03-03 14:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.120: /opt/couchbase/bin/cbvdiff -b default 172.23.107.120:11210,172.23.107.169:11210,172.23.105.45:11210 
2015-03-03 14:58:39 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2015-03-03 14:58:40 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Active item count = 35008
2015-03-03 14:58:40 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 
2015-03-03 14:58:40 | ERROR | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] ERROR: Timed-out waiting for replica item count to match
2015-03-03 14:58:41 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:58:41 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:58:41 | INFO | MainProcess | test_thread | [task.__init__] 50000 items will be verified on default bucket
2015-03-03 14:58:42 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified
2015-03-03 14:58:44 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:46 | INFO | MainProcess | load_gen_task | [task.has_next] 10000 items were verified
2015-03-03 14:58:49 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:50 | INFO | MainProcess | load_gen_task | [task.has_next] 20000 items were verified
2015-03-03 14:58:54 | INFO | MainProcess | load_gen_task | [task.has_next] 30000 items were verified
2015-03-03 14:58:54 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:58:59 | INFO | MainProcess | load_gen_task | [task.has_next] 40000 items were verified
2015-03-03 14:58:59 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:59:04 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:59:06 | INFO | MainProcess | load_gen_task | [task.has_next] 50000 items were verified
2015-03-03 14:59:06 | INFO | MainProcess | load_gen_task | [task.has_next] 50000 items were verified in 24.9660220146 sec.the average number of ops                - 2002.72168964 per second
2015-03-03 14:59:06 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.120:11210 default
2015-03-03 14:59:07 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.169:11210 default
2015-03-03 14:59:07 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.45:11210 default
2015-03-03 14:59:07 | INFO | MainProcess | test_thread | [task.__init__] 35000 items will be verified on default bucket
2015-03-03 14:59:07 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified
2015-03-03 14:59:09 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:59:12 | INFO | MainProcess | load_gen_task | [task.has_next] 10000 items were verified
2015-03-03 14:59:15 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:59:16 | INFO | MainProcess | load_gen_task | [task.has_next] 20000 items were verified
2015-03-03 14:59:20 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:59:20 | INFO | MainProcess | load_gen_task | [task.has_next] 30000 items were verified
2015-03-03 14:59:22 | INFO | MainProcess | load_gen_task | [task.has_next] 35000 items were verified in 14.9562737942 sec.the average number of ops                - 2340.15443932 per second
2015-03-03 14:59:22 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_rev_ids] Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.107.170 -> Couchbase Cluster: C2, Master Ip: 172.23.107.120, bucket <bound method XDCReplication.get_src_bucket of <xdcr.xdcrnewbasetests.XDCReplication instance at 0x7ff1d0037d88>>
2015-03-03 14:59:22 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:59:23 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:59:23 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.170:11210 default
2015-03-03 14:59:23 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.171:11210 default
2015-03-03 14:59:24 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.120:11210 default
2015-03-03 14:59:24 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.169:11210 default
2015-03-03 14:59:24 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.45:11210 default
2015-03-03 14:59:25 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:59:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:59:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:59:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:59:45 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:59:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 14:59:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:05 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:10 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:15 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:20 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:25 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: C1-key-38887 =====
2015-03-03 15:00:25 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:1
2015-03-03 15:00:25 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:2
2015-03-03 15:00:25 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:1425422998371172353, Destination cas:1425422998371172352, Error Count:3
2015-03-03 15:00:25 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data: {'deleted': 1, 'seqno': 2, 'cas': 1425422998371172353, 'flags': 0, 'expiration': 1425423121}
2015-03-03 15:00:25 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1425422998371172352, 'flags': 0, 'expiration': 0}
2015-03-03 15:00:25 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:36 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: C1-key-39865 =====
2015-03-03 15:00:36 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:4
2015-03-03 15:00:36 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:5
2015-03-03 15:00:36 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:1425422998694330369, Destination cas:1425422998694330368, Error Count:6
2015-03-03 15:00:36 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data: {'deleted': 1, 'seqno': 2, 'cas': 1425422998694330369, 'flags': 0, 'expiration': 1425423121}
2015-03-03 15:00:36 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1425422998694330368, 'flags': 0, 'expiration': 0}
2015-03-03 15:00:39 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: C1-key-39706 =====
2015-03-03 15:00:39 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:7
2015-03-03 15:00:39 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:8
2015-03-03 15:00:39 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:1425422998710517761, Destination cas:1425422998710517760, Error Count:9
2015-03-03 15:00:39 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data: {'deleted': 1, 'seqno': 2, 'cas': 1425422998710517761, 'flags': 0, 'expiration': 1425423121}
2015-03-03 15:00:39 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1425422998710517760, 'flags': 0, 'expiration': 0}
2015-03-03 15:00:41 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:42 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: C1-key-39696 =====
2015-03-03 15:00:42 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:10
2015-03-03 15:00:42 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:11
2015-03-03 15:00:42 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:1425422998681616385, Destination cas:1425422998681616384, Error Count:12
2015-03-03 15:00:42 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data: {'deleted': 1, 'seqno': 2, 'cas': 1425422998681616385, 'flags': 0, 'expiration': 1425423121}
2015-03-03 15:00:42 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1425422998681616384, 'flags': 0, 'expiration': 0}
2015-03-03 15:00:43 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: C1-key-39074 =====
2015-03-03 15:00:43 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:13
2015-03-03 15:00:43 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:14
2015-03-03 15:00:43 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:1425422998721658881, Destination cas:1425422998721658880, Error Count:15
2015-03-03 15:00:43 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data: {'deleted': 1, 'seqno': 2, 'cas': 1425422998721658881, 'flags': 0, 'expiration': 1425423121}
2015-03-03 15:00:43 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1425422998721658880, 'flags': 0, 'expiration': 0}
2015-03-03 15:00:45 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: C1-key-38970 =====
2015-03-03 15:00:45 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:16
2015-03-03 15:00:45 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:17
2015-03-03 15:00:45 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:1425422998366978049, Destination cas:1425422998366978048, Error Count:18
2015-03-03 15:00:45 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data: {'deleted': 1, 'seqno': 2, 'cas': 1425422998366978049, 'flags': 0, 'expiration': 1425423121}
2015-03-03 15:00:45 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1425422998366978048, 'flags': 0, 'expiration': 0}
2015-03-03 15:00:46 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:51 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:56 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: C1-key-38917 =====
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:19
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:20
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:1425422998375432193, Destination cas:1425422998375432192, Error Count:21
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data: {'deleted': 1, 'seqno': 2, 'cas': 1425422998375432193, 'flags': 0, 'expiration': 1425423121}
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1425422998375432192, 'flags': 0, 'expiration': 0}
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: C1-key-40474 =====
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:22
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:23
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:1425422999232774145, Destination cas:1425422999232774144, Error Count:24
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data: {'deleted': 1, 'seqno': 2, 'cas': 1425422999232774145, 'flags': 0, 'expiration': 1425423123}
2015-03-03 15:00:58 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 1425422999232774144, 'flags': 0, 'expiration': 0}
2015-03-03 15:00:59 | INFO | MainProcess | load_gen_task | [task.next] 50000 items have been verified
2015-03-03 15:00:59 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 35000 existing items have been verified
2015-03-03 15:00:59 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 15000 deleted items have been verified
FAIL
2015-03-03 15:00:59 | INFO | MainProcess | test_thread | [xdcrnewbasetests.tearDown] ====  XDCRNewbasetests cleanup is started for test #1 async_rebalance_in ====
2015-03-03 15:00:59 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] removing xdcr/nodes settings
2015-03-03 15:00:59 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.170:8091: True content: true command: ns_config:read_key_fast(goxdcr_enabled, false)
2015-03-03 15:00:59 | INFO | MainProcess | test_thread | [rest_client.stop_replication] Deleting replication controller/cancelXDCR/replicationSpec%2Facbb10af4ac017768cfb4bffb6a7a245%2Fdefault%2Fdefault
2015-03-03 15:00:59 | INFO | MainProcess | test_thread | [rest_client.remove_remote_cluster] removing remote cluster name:remote_cluster_C1-C2
2015-03-03 15:00:59 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] cleanup [ip:172.23.107.170 port:8091 ssh_username:root, ip:172.23.107.171 port:8091 ssh_username:root]
2015-03-03 15:01:00 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 172.23.107.170
2015-03-03 15:01:00 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ...
2015-03-03 15:01:01 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:01:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.107.170
2015-03-03 15:01:02 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2015-03-03 15:01:02 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.107.170 existing buckets : []
2015-03-03 15:01:02 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes
2015-03-03 15:01:02 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : password=welcome&ejectedNodes=ns_1%40172.23.107.171&user=Administrator&knownNodes=ns_1%40172.23.107.171%2Cns_1%40172.23.107.170
2015-03-03 15:01:02 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started
2015-03-03 15:01:02 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-03-03 15:01:06 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:01:07 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-03-03 15:01:11 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:01:16 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:01:17 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 15.0951530933 seconds 
2015-03-03 15:01:17 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance...
2015-03-03 15:01:21 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:01:26 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 35008 == 35000 expected on '172.23.107.120:8091''172.23.107.169:8091''172.23.105.45:8091', default bucket
2015-03-03 15:01:27 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.107.170 port:8091 ssh_username:root ? [(u'ns_1@172.23.107.171', 8091)]
2015-03-03 15:01:27 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.170:8091
2015-03-03 15:01:28 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.170:8091 is running
2015-03-03 15:01:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.107.171
2015-03-03 15:01:28 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.171:8091
2015-03-03 15:01:28 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.171:8091 is running
Cluster instance shutdown with force
2015-03-03 15:01:28 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] removing xdcr/nodes settings
2015-03-03 15:01:28 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] cleanup [ip:172.23.107.120 port:8091 ssh_username:root, ip:172.23.107.169 port:8091 ssh_username:root, ip:172.23.105.45 port:8091 ssh_username:root]
2015-03-03 15:01:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 172.23.107.120
2015-03-03 15:01:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ...
2015-03-03 15:01:29 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.107.120
2015-03-03 15:01:29 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2015-03-03 15:01:29 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.107.120 existing buckets : []
2015-03-03 15:01:29 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes
2015-03-03 15:01:29 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : password=welcome&ejectedNodes=ns_1%40172.23.105.45%2Cns_1%40172.23.107.169&user=Administrator&knownNodes=ns_1%40172.23.105.45%2Cns_1%40172.23.107.169%2Cns_1%40172.23.107.120
2015-03-03 15:01:29 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started
2015-03-03 15:01:29 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
Exception in thread Cluster_Thread:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
  File "lib/tasks/taskmanager.py", line 31, in run
    task.step(self)
  File "lib/tasks/task.py", line 71, in step
    self.check(task_manager)
  File "lib/tasks/task.py", line 472, in check
    stats = client.stats(self.param)
  File "lib/mc_bin_client.py", line 408, in stats
    cmd, opaque, cas, klen, extralen, data = self._handleKeyedResponse(None)
  File "lib/mc_bin_client.py", line 117, in _handleKeyedResponse
    raise MemcachedError(errcode, rv)
MemcachedError: Memcached error #4 'Invalid':  Invalid arguments for vbucket :0 to mc 172.23.107.120:11210

2015-03-03 15:01:39 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 10.0475299358 seconds 
2015-03-03 15:01:39 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance...
2015-03-03 15:01:49 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.107.120 port:8091 ssh_username:root ? [(u'ns_1@172.23.105.45', 8091), (u'ns_1@172.23.107.169', 8091)]
2015-03-03 15:01:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.120:8091
2015-03-03 15:01:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.120:8091 is running
2015-03-03 15:01:49 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.107.169
2015-03-03 15:01:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.169:8091
2015-03-03 15:01:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.169:8091 is running
2015-03-03 15:01:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.105.45
2015-03-03 15:01:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.45:8091
2015-03-03 15:01:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.45:8091 is running
Cluster instance shutdown with force
2015-03-03 15:01:50 | INFO | MainProcess | test_thread | [xdcrnewbasetests.tearDown] ====  XDCRNewbasetests cleanup is finished for test #1 async_rebalance_in ===

======================================================================
FAIL: async_rebalance_in (xdcr.rebalanceXDCR.Rebalance)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/xdcr/rebalanceXDCR.py", line 37, in async_rebalance_in
    self.verify_results()
  File "pytests/xdcr/xdcrnewbasetests.py", line 2888, in verify_results
    self.fail("Active or replica key count is incorrect")
AssertionError: Active or replica key count is incorrect

----------------------------------------------------------------------