[root@cen-1413 testrunner]# ./testrunner -i ./test.ini -t xdcr.biXDCR.bidirectional.load_with_failover_then_add_back,items=10000,ctopology=chain,rdirection=bidirection,doc-ops=create-update-delete,doc-ops-dest=create-update-delete,failover=destination,GROUP=P1
Global Test input params:
{'cluster_name': 'test', 'ini': './test.ini', 'num_nodes': 7}

./testrunner -i ./test.ini  -t xdcr.biXDCR.bidirectional.load_with_failover_then_add_back,items=10000,ctopology=chain,rdirection=bidirection,doc-ops=create-update-delete,doc-ops-dest=create-update-delete,failover=destination,GROUP=P1

Test Input params:
{'doc-ops': 'create-update-delete', 'GROUP': 'P1', 'items': '10000', 'failover': 'destination', 'num_nodes': 7, 'cluster_name': 'test', 'ctopology': 'chain', 'rdirection': 'bidirection', 'ini': './test.ini', 'doc-ops-dest': 'create-update-delete', 'case_number': 1}
Run before suite setup for xdcr.biXDCR.bidirectional.load_with_failover_then_add_back
load_with_failover_then_add_back (xdcr.biXDCR.bidirectional) ... 2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [xdcrbasetests._init_parameters] Initializing input parameters started...
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [xdcrbasetests.set_xdcr_param] Setting xdcrFailureRestartInterval to 1 ..
lib/membase/api/rest_client.py:197: DeprecationWarning: object.__new__() takes no parameters
  obj = object.__new__(self, serverInfo)
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [rest_client.set_internalSetting] Update internal setting xdcrFailureRestartInterval=1
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [rest_client.set_internalSetting] Update internal setting xdcrFailureRestartInterval=1
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [xdcrbasetests._init_parameters] Initializing input parameters completed.
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [xdcrbasetests.setUp] ==============  XDCRbasetests setup was started for test #1 load_with_failover_then_add_back==============
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [xdcrbasetests._do_cleanup] cleanup cluster1: [ip:10.1.3.96 port:8091 ssh_username:root, ip:10.1.2.12 port:8091 ssh_username:root, ip:10.1.3.97 port:8091 ssh_username:root, ip:10.1.3.99 port:8091 ssh_username:root]
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.3.96
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.96:8091
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.96:8091 is running
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.2.12
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.2.12:8091
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.2.12:8091 is running
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.3.97
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.97:8091
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.97:8091 is running
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.3.99
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.99:8091
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.99:8091 is running
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [xdcrbasetests._do_cleanup] cleanup cluster2: [ip:10.1.3.93 port:8091 ssh_username:root, ip:10.1.3.95 port:8091 ssh_username:root, ip:10.1.3.94 port:8091 ssh_username:root]
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.3.93
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.93:8091
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.93:8091 is running
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.3.95
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.95:8091
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.95:8091 is running
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.3.94
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.94:8091
2013-12-26 22:55:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.94:8091 is running
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.set_reb_cons_view] Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.1.3.96:8091:username=Administrator&password=password&port=8091
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2106&username=Administrator&password=password
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.set_reb_cons_view] Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.1.2.12:8091:username=Administrator&password=password&port=8091
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2106&username=Administrator&password=password
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.set_reb_cons_view] Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.1.3.97:8091:username=Administrator&password=password&port=8091
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2106&username=Administrator&password=password
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.set_reb_cons_view] Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.1.3.99:8091:username=Administrator&password=password&port=8091
2013-12-26 22:55:34 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2106&username=Administrator&password=password
2013-12-26 22:55:35 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.1.2.12:8091 to cluster
2013-12-26 22:55:35 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.1.2.12:8091 to this cluster @10.1.3.96:8091
2013-12-26 22:55:35 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.1.3.97:8091 to cluster
2013-12-26 22:55:35 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.1.3.97:8091 to this cluster @10.1.3.96:8091
2013-12-26 22:55:35 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.1.3.99:8091 to cluster
2013-12-26 22:55:35 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.1.3.99:8091 to this cluster @10.1.3.96:8091
2013-12-26 22:55:35 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.1.3.99%2Cns_1%4010.1.3.97%2Cns_1%4010.1.3.96%2Cns_1%4010.1.2.12
2013-12-26 22:55:35 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2013-12-26 22:55:36 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0 %
2013-12-26 22:55:46 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.0287570953 sec
2013-12-26 22:55:47 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.1.3.96:8091/pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=None&threadsNumber=3&flushEnabled=1&replicaNumber=1&ramQuotaMB=1404
2013-12-26 22:55:47 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.0201470851898 seconds to create bucket default
2013-12-26 22:55:47 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 10.1.3.96 to accept set ops
2013-12-26 22:55:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.2.12:11210 default
2013-12-26 22:55:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.96:11210 default
2013-12-26 22:55:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.97:11210 default
2013-12-26 22:55:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.99:11210 default
2013-12-26 22:55:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.2.12:11210 default
2013-12-26 22:55:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.96:11210 default
2013-12-26 22:55:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.97:11210 default
2013-12-26 22:55:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.99:11210 default
2013-12-26 22:55:54 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 1404
2013-12-26 22:55:54 | INFO | MainProcess | MainThread | [xdcrbasetests.sleep] sleep for 30 secs.  ...
2013-12-26 22:56:25 | INFO | MainProcess | Cluster_Thread | [rest_client.set_reb_cons_view] Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
2013-12-26 22:56:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.1.3.93:8091:username=Administrator&password=password&port=8091
2013-12-26 22:56:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2106&username=Administrator&password=password
2013-12-26 22:56:25 | INFO | MainProcess | Cluster_Thread | [rest_client.set_reb_cons_view] Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
2013-12-26 22:56:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.1.3.95:8091:username=Administrator&password=password&port=8091
2013-12-26 22:56:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2106&username=Administrator&password=password
2013-12-26 22:56:25 | INFO | MainProcess | Cluster_Thread | [rest_client.set_reb_cons_view] Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
2013-12-26 22:56:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.1.3.94:8091:username=Administrator&password=password&port=8091
2013-12-26 22:56:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2106&username=Administrator&password=password
2013-12-26 22:56:26 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.1.3.95:8091 to cluster
2013-12-26 22:56:26 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.1.3.95:8091 to this cluster @10.1.3.93:8091
2013-12-26 22:56:26 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.1.3.94:8091 to cluster
2013-12-26 22:56:26 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.1.3.94:8091 to this cluster @10.1.3.93:8091
2013-12-26 22:56:26 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.1.3.94%2Cns_1%4010.1.3.93%2Cns_1%4010.1.3.95
2013-12-26 22:56:26 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2013-12-26 22:56:26 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0 %
2013-12-26 22:56:36 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.0252799988 sec
2013-12-26 22:56:37 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.1.3.93:8091/pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=None&threadsNumber=3&flushEnabled=1&replicaNumber=1&ramQuotaMB=1404
2013-12-26 22:56:37 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.0105519294739 seconds to create bucket default
2013-12-26 22:56:37 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 10.1.3.93 to accept set ops
2013-12-26 22:56:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.93:11210 default
2013-12-26 22:56:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.94:11210 default
2013-12-26 22:56:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.95:11210 default
2013-12-26 22:56:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.93:11210 default
2013-12-26 22:56:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.94:11210 default
2013-12-26 22:56:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.95:11210 default
2013-12-26 22:56:43 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 1404
2013-12-26 22:56:43 | INFO | MainProcess | MainThread | [xdcrbasetests.sleep] sleep for 30 secs.  ...
2013-12-26 22:57:13 | INFO | MainProcess | MainThread | [rest_client.add_remote_cluster] adding remote cluster hostname:10.1.3.93:8091 with username:password Administrator:password name:cluster1
2013-12-26 22:57:13 | INFO | MainProcess | MainThread | [rest_client.start_replication] starting replication type:continuous from default to default in the remote cluster cluster1
2013-12-26 22:57:13 | INFO | MainProcess | MainThread | [xdcrbasetests.sleep] sleep for 5 secs.  ...
2013-12-26 22:57:18 | INFO | MainProcess | MainThread | [rest_client.add_remote_cluster] adding remote cluster hostname:10.1.3.96:8091 with username:password Administrator:password name:cluster0
2013-12-26 22:57:18 | INFO | MainProcess | MainThread | [rest_client.start_replication] starting replication type:continuous from default to default in the remote cluster cluster0
2013-12-26 22:57:18 | INFO | MainProcess | MainThread | [xdcrbasetests.sleep] sleep for 5 secs.  ...
2013-12-26 22:57:23 | INFO | MainProcess | MainThread | [xdcrbasetests.setUp] ==============  XDCRbasetests setup was finished for test #1 load_with_failover_then_add_back ==============
2013-12-26 22:57:26 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.2.12:11210 default
2013-12-26 22:57:31 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.96:11210 default
2013-12-26 22:57:36 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.97:11210 default
2013-12-26 22:57:40 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.99:11210 default
2013-12-26 22:57:45 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 0 with exp:0
2013-12-26 23:01:30 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 10000 with exp:0
2013-12-26 23:01:34 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.93:11210 default
2013-12-26 23:01:39 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.94:11210 default
2013-12-26 23:01:44 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.95:11210 default
2013-12-26 23:01:48 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 0 with exp:0
2013-12-26 23:06:04 | INFO | MainProcess | load_gen_task | [task.has_next] Batch create documents done #: 10000 with exp:0
2013-12-26 23:06:04 | INFO | MainProcess | MainThread | [xdcrbasetests.sleep] sleep for 60 secs.  ...
2013-12-26 23:07:04 | INFO | MainProcess | MainThread | [biXDCR.load_with_failover_then_add_back]  Failing over Destination Non-Master Node 10.1.3.94:8091
2013-12-26 23:07:05 | INFO | MainProcess | Cluster_Thread | [task._failover_nodes] Failing over 10.1.3.94:8091
2013-12-26 23:07:05 | INFO | MainProcess | Cluster_Thread | [rest_client.fail_over] fail_over successful
2013-12-26 23:07:05 | INFO | MainProcess | Cluster_Thread | [task.execute] 20 seconds sleep after failover, for nodes to go pending....
2013-12-26 23:07:25 | INFO | MainProcess | MainThread | [biXDCR.load_with_failover_then_add_back]  Add back Destination Non-Master Node 10.1.3.94:8091
2013-12-26 23:07:26 | INFO | MainProcess | MainThread | [rest_client.add_back_node] add_back_node successful
2013-12-26 23:07:27 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.1.3.94%2Cns_1%4010.1.3.93%2Cns_1%4010.1.3.95
2013-12-26 23:07:27 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2013-12-26 23:07:27 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0 %
2013-12-26 23:07:38 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 8.99061524994 %
2013-12-26 23:07:48 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 21.0433549939 %
2013-12-26 23:07:58 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 32.0863115563 %
2013-12-26 23:08:09 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 42.8360765027 %
2013-12-26 23:08:19 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 54.2424889539 %
2013-12-26 23:08:29 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 65.3895039818 %
2013-12-26 23:08:39 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 75.6889546784 %
2013-12-26 23:08:49 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 85.2713041879 %
2013-12-26 23:09:00 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 93.6467097032 %
2013-12-26 23:09:10 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 103.289789915 sec
2013-12-26 23:09:10 | INFO | MainProcess | MainThread | [xdcrbasetests.sleep] sleep for 60 secs.  ...
2013-12-26 23:10:10 | INFO | MainProcess | MainThread | [xdcrbasetests._async_update_delete_data] The tasks:-
2013-12-26 23:10:14 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.2.12:11210 default
2013-12-26 23:10:19 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.96:11210 default
2013-12-26 23:10:24 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.97:11210 default
2013-12-26 23:10:29 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.99:11210 default
2013-12-26 23:10:37 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.2.12:11210 default
2013-12-26 23:10:42 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.96:11210 default
2013-12-26 23:10:47 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.97:11210 default
2013-12-26 23:10:51 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.99:11210 default
2013-12-26 23:10:56 | INFO | MainProcess | MainThread | [xdcrbasetests.sleep] sleep for 10 secs.  ...
2013-12-26 23:11:10 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.93:11210 default
2013-12-26 23:11:15 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.94:11210 default
2013-12-26 23:11:19 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.95:11210 default
2013-12-26 23:11:27 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.93:11210 default
2013-12-26 23:11:31 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.94:11210 default
2013-12-26 23:11:36 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.95:11210 default
2013-12-26 23:11:40 | INFO | MainProcess | MainThread | [xdcrbasetests.sleep] sleep for 10 secs.  ...
2013-12-26 23:13:41 | INFO | MainProcess | MainThread | [xdcrbasetests.merge_buckets] merge buckets 10.1.3.96->10.1.3.93, bidirection:True
2013-12-26 23:14:03 | INFO | MainProcess | MainThread | [xdcrbasetests.sleep] sleep for 120 secs.  ...
2013-12-26 23:16:07 | INFO | MainProcess | MainThread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'10.1.3.99', 'password': 'password', 'port': u'8091'}
2013-12-26 23:16:07 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.99:11210 default
2013-12-26 23:16:11 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.99', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2013-12-26 23:16:11 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.99', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (4278210872, 0, '')
2013-12-26 23:16:13 | INFO | MainProcess | MainThread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'10.1.2.12', 'password': 'password', 'port': u'8091'}
2013-12-26 23:16:13 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.2.12:11210 default
2013-12-26 23:16:17 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.2.12', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2013-12-26 23:16:17 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.2.12', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3830694094, 0, '')
2013-12-26 23:16:17 | INFO | MainProcess | MainThread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'10.1.3.96', 'password': 'password', 'port': u'8091'}
2013-12-26 23:16:17 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.96:11210 default
2013-12-26 23:16:21 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.96', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2013-12-26 23:16:21 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.96', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3878245687, 0, '')
2013-12-26 23:16:22 | INFO | MainProcess | MainThread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'10.1.3.97', 'password': 'password', 'port': u'8091'}
2013-12-26 23:16:22 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.97:11210 default
2013-12-26 23:16:27 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.97', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2013-12-26 23:16:27 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.97', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2640727345, 0, '')
2013-12-26 23:16:27 | INFO | MainProcess | MainThread | [xdcrbasetests._expiry_pager] wait for expiry pager to run on all these nodes
2013-12-26 23:16:27 | INFO | MainProcess | MainThread | [xdcrbasetests.sleep] sleep for 30 secs.  ...
2013-12-26 23:16:59 | INFO | MainProcess | MainThread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'10.1.3.95', 'password': 'password', 'port': u'8091'}
2013-12-26 23:16:59 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.95:11210 default
2013-12-26 23:17:02 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.95', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2013-12-26 23:17:02 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.95', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (4069219609, 0, '')
2013-12-26 23:17:04 | INFO | MainProcess | MainThread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'10.1.3.93', 'password': 'password', 'port': u'8091'}
2013-12-26 23:17:04 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.93:11210 default
2013-12-26 23:17:07 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.93', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2013-12-26 23:17:07 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.93', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (70299695, 0, '')
2013-12-26 23:17:08 | INFO | MainProcess | MainThread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'10.1.3.94', 'password': 'password', 'port': u'8091'}
2013-12-26 23:17:08 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.94:11210 default
2013-12-26 23:17:12 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.94', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
2013-12-26 23:17:12 | INFO | MainProcess | MainThread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'10.1.3.94', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (913107776, 0, '')
2013-12-26 23:17:12 | INFO | MainProcess | MainThread | [xdcrbasetests._expiry_pager] wait for expiry pager to run on all these nodes
2013-12-26 23:17:12 | INFO | MainProcess | MainThread | [xdcrbasetests.sleep] sleep for 30 secs.  ...
2013-12-26 23:17:42 | INFO | MainProcess | MainThread | [xdcrbasetests.verify_xdcr_stats] and Verify xdcr replication stats at Source Cluster : 10.1.3.96
2013-12-26 23:17:43 | WARNING | MainProcess | MainThread | [xdcrbasetests._get_cluster_buckets] No bucket(s) found on the server ip:10.1.3.97 port:8091 ssh_username:root
2013-12-26 23:17:43 | INFO | MainProcess | MainThread | [xdcrbasetests.verify_xdcr_stats] Verify xdcr replication stats at Destination Cluster : 10.1.3.93
2013-12-26 23:17:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.93:11210 default
2013-12-26 23:17:48 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '10.1.3.93:8091',default bucket
2013-12-26 23:17:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.95:11210 default
2013-12-26 23:17:54 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '10.1.3.95:8091',default bucket
2013-12-26 23:17:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.94:11210 default
2013-12-26 23:17:59 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '10.1.3.94:8091',default bucket
2013-12-26 23:18:00 | WARNING | MainProcess | MainThread | [xdcrbasetests._get_cluster_buckets] No bucket(s) found on the server ip:10.1.3.97 port:8091 ssh_username:root
2013-12-26 23:18:08 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.2.12:11210 default
2013-12-26 23:18:13 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.96:11210 default
2013-12-26 23:18:17 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.97:11210 default
2013-12-26 23:18:21 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 10.1.3.99:11210 default
2013-12-26 23:18:26 | INFO | MainProcess | MainThread | [task.__init__] 20000 items will be verified on default bucket
2013-12-26 23:18:27 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified
2013-12-26 23:20:43 | INFO | MainProcess | load_gen_task | [task.has_next] 10000 items were verified
2013-12-26 23:22:43 | INFO | MainProcess | load_gen_task | [task.has_next] 20000 items were verified
2013-12-26 23:22:43 | INFO | MainProcess | load_gen_task | [task.has_next] 20000 items were verified in 257.021759033 sec.the average number of ops                - 77.8144219603 per second
2013-12-26 23:22:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.93:11210 default
2013-12-26 23:22:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.95:11210 default
2013-12-26 23:22:58 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.94:11210 default
2013-12-26 23:23:03 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items 14000 == 14000 expected on '10.1.3.93:8091''10.1.3.95:8091''10.1.3.94:8091',default bucket
2013-12-26 23:23:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.93:11210 default
2013-12-26 23:23:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.95:11210 default
2013-12-26 23:23:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.1.3.94:11210 default
2013-12-26 23:23:19 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 14000 == 14000 expected on '10.1.3.93:8091''10.1.3.95:8091''10.1.3.94:8091',default bucket
FAIL
2013-12-26 23:27:23 | INFO | MainProcess | MainThread | [xdcrbasetests.tearDown] ==============  XDCRbasetests stats for test #1 load_with_failover_then_add_back ==============
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests.tearDown] Type of run: BIDIRECTIONAL XDCR
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] STATS with source at 10.1.3.96 and destination at 10.1.3.93
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] Bucket: default
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Average local replica creation rate for bucket 'default': 12.9539633831 KB per second
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Median XDC replication ops for bucket 'default': 0.127 K ops per second
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Mean XDC replication ops for bucket 'default': 0.14080246695 K ops per second
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Average XDCR data replication rate for bucket 'default': 12.9661243549 KB per second
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] STATS with source at 10.1.3.93 and destination at 10.1.3.96
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] Bucket: default
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Average local replica creation rate for bucket 'default': 12.9236893275 KB per second
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Median XDC replication ops for bucket 'default': 0.0795 K ops per second
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Mean XDC replication ops for bucket 'default': 0.149623565977 K ops per second
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Average XDCR data replication rate for bucket 'default': 12.961645804 KB per second
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests.tearDown] ============== = = = = = = = = END = = = = = = = = = = ==============
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests.tearDown] ==============  XDCRbasetests cleanup was started for test #1 load_with_failover_then_add_back ==============
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [rest_client.remove_remote_cluster] removing remote cluster name:cluster1
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [rest_client.remove_remote_cluster] removing remote cluster name:cluster0
2013-12-26 23:27:25 | INFO | MainProcess | MainThread | [xdcrbasetests._do_cleanup] cleanup cluster1: [ip:10.1.3.96 port:8091 ssh_username:root, ip:10.1.2.12 port:8091 ssh_username:root, ip:10.1.3.97 port:8091 ssh_username:root, ip:10.1.3.99 port:8091 ssh_username:root]
2013-12-26 23:27:26 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 10.1.3.96
2013-12-26 23:27:26 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ...
2013-12-26 23:27:28 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 10.1.3.96
2013-12-26 23:27:28 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2013-12-26 23:27:28 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] existing buckets : []
2013-12-26 23:27:28 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes
2013-12-26 23:27:28 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%4010.1.3.99%2Cns_1%4010.1.3.97%2Cns_1%4010.1.2.12&user=Administrator&knownNodes=ns_1%4010.1.3.99%2Cns_1%4010.1.3.97%2Cns_1%4010.1.3.96%2Cns_1%4010.1.2.12
2013-12-26 23:27:28 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started
2013-12-26 23:27:28 | INFO | MainProcess | MainThread | [rest_client._rebalance_progress] rebalance percentage : 0 %
2013-12-26 23:27:32 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 4.01596593857 seconds 
2013-12-26 23:27:32 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 4.01596593857 seconds after rebalance...
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:10.1.3.96 port:8091 ssh_username:root ? [(u'ns_1@10.1.3.99', 8091), (u'ns_1@10.1.3.97', 8091), (u'ns_1@10.1.2.12', 8091)]
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.96:8091
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.96:8091 is running
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.2.12
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.2.12:8091
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.2.12:8091 is running
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.3.97
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.97:8091
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.97:8091 is running
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.3.99
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.99:8091
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.99:8091 is running
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [xdcrbasetests._do_cleanup] cleanup cluster2: [ip:10.1.3.93 port:8091 ssh_username:root, ip:10.1.3.95 port:8091 ssh_username:root, ip:10.1.3.94 port:8091 ssh_username:root]
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 10.1.3.93
2013-12-26 23:27:36 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ...
2013-12-26 23:27:39 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 10.1.3.93
2013-12-26 23:27:39 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2013-12-26 23:27:39 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] existing buckets : []
2013-12-26 23:27:39 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes
2013-12-26 23:27:39 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%4010.1.3.94%2Cns_1%4010.1.3.95&user=Administrator&knownNodes=ns_1%4010.1.3.94%2Cns_1%4010.1.3.93%2Cns_1%4010.1.3.95
2013-12-26 23:27:39 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started
2013-12-26 23:27:39 | INFO | MainProcess | MainThread | [rest_client._rebalance_progress] rebalance percentage : 0 %
2013-12-26 23:27:43 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 4.01625609398 seconds 
2013-12-26 23:27:43 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 4.01625609398 seconds after rebalance...
2013-12-26 23:27:47 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:10.1.3.93 port:8091 ssh_username:root ? [(u'ns_1@10.1.3.94', 8091), (u'ns_1@10.1.3.95', 8091)]
2013-12-26 23:27:47 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.93:8091
2013-12-26 23:27:47 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.93:8091 is running
2013-12-26 23:27:47 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.3.95
2013-12-26 23:27:47 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.95:8091
2013-12-26 23:27:47 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.95:8091 is running
2013-12-26 23:27:47 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.1.3.94
2013-12-26 23:27:47 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.1.3.94:8091
2013-12-26 23:27:47 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.1.3.94:8091 is running
2013-12-26 23:27:47 | INFO | MainProcess | MainThread | [xdcrbasetests.tearDown] ==============  XDCRbasetests cleanup was finished for test #1 load_with_failover_then_add_back ==============

======================================================================
FAIL: load_with_failover_then_add_back (xdcr.biXDCR.bidirectional)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/xdcr/biXDCR.py", line 283, in load_with_failover_then_add_back
    self.verify_results(verify_src=True)
  File "pytests/xdcr/xdcrbasetests.py", line 717, in verify_results
    self.verify_xdcr_stats(src_nodes, dest_nodes, verify_src)
  File "pytests/xdcr/xdcrbasetests.py", line 697, in verify_xdcr_stats
    self.__wait_for_mutation_to_replicate(self.dest_master)
  File "pytests/xdcr/xdcrbasetests.py", line 662, in __wait_for_mutation_to_replicate
    self.fail("Timeout occurs while waiting for mutations to be replicated")
AssertionError: Timeout occurs while waiting for mutations to be replicated

----------------------------------------------------------------------
Ran 1 test in 1934.479s

FAILED (failures=1)
summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1
failures so far...
xdcr.biXDCR.bidirectional.load_with_failover_then_add_back
testrunner logs, diags and results are available under logs/testrunner-13-Dec-26_22-55-33
Run after suite setup for xdcr.biXDCR.bidirectional.load_with_failover_then_add_back