./testrunner -i /tmp/testexec.4730.ini stop-on-failure=False,fail_on_errors=1,get-cbcollect-info=True -t xdcr.uniXDCR.unidirectional.test_verify_mb19802_1,items=100000,rdirection=unidirection,ctopology=chain,GROUP=CHAIN;P0

Test Input params:
{'fail_on_errors': '1', 'logs_folder': '/data/workspace/centos-p0-xdcr-vset00-00-unidirectional-2/logs/testrunner-18-Jan-09_02-02-04/test_14', 'GROUP': 'CHAIN;P0', 'items': '100000', 'conf_file': 'py-xdcr-unidirectional-2.conf', 'num_nodes': 6, 'cluster_name': 'testexec.4730', 'ctopology': 'chain', 'rdirection': 'unidirection', 'stop-on-failure': 'False', 'ini': '/tmp/testexec.4730.ini', 'case_number': 14, 'get-cbcollect-info': 'True', 'spec': 'py-xdcr-unidirectional-2'}
[2018-01-09 08:23:56,128] - [rest_client:1438] INFO - /diag/eval status on 172.23.105.212:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2018-01-09 08:23:56,131] - [remote_util:232] INFO - connecting to 172.23.105.212 with username:root password:couchbase ssh_key:
[2018-01-09 08:23:56,269] - [remote_util:266] INFO - Connected to 172.23.105.212
[2018-01-09 08:23:56,690] - [remote_util:2990] INFO - running command.raw on 172.23.105.212: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:23:56,811] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:23:56,812] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:23:56,930] - [remote_util:232] INFO - connecting to 172.23.105.212 with username:root password:couchbase ssh_key:
[2018-01-09 08:23:57,068] - [remote_util:266] INFO - Connected to 172.23.105.212
[2018-01-09 08:23:57,468] - [remote_util:2990] INFO - running command.raw on 172.23.105.212: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:23:57,580] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:23:57,581] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:23:57,700] - [remote_util:232] INFO - connecting to 172.23.105.214 with username:root password:couchbase ssh_key:
[2018-01-09 08:23:57,836] - [remote_util:266] INFO - Connected to 172.23.105.214
[2018-01-09 08:23:58,443] - [remote_util:2990] INFO - running command.raw on 172.23.105.214: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:23:58,576] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:23:58,577] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:23:58,695] - [remote_util:232] INFO - connecting to 172.23.105.214 with username:root password:couchbase ssh_key:
[2018-01-09 08:23:58,827] - [remote_util:266] INFO - Connected to 172.23.105.214
[2018-01-09 08:23:59,431] - [remote_util:2990] INFO - running command.raw on 172.23.105.214: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:23:59,607] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:23:59,607] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:23:59,725] - [remote_util:232] INFO - connecting to 172.23.105.218 with username:root password:couchbase ssh_key:
[2018-01-09 08:23:59,860] - [remote_util:266] INFO - Connected to 172.23.105.218
[2018-01-09 08:24:00,448] - [remote_util:2990] INFO - running command.raw on 172.23.105.218: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:00,570] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:00,571] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:00,690] - [remote_util:232] INFO - connecting to 172.23.105.218 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:00,824] - [remote_util:266] INFO - Connected to 172.23.105.218
[2018-01-09 08:24:01,419] - [remote_util:2990] INFO - running command.raw on 172.23.105.218: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:01,538] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:01,539] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:01,658] - [remote_util:232] INFO - connecting to 172.23.105.223 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:01,831] - [remote_util:266] INFO - Connected to 172.23.105.223
[2018-01-09 08:24:02,268] - [remote_util:2990] INFO - running command.raw on 172.23.105.223: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:02,358] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:02,358] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:02,476] - [remote_util:232] INFO - connecting to 172.23.105.223 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:02,645] - [remote_util:266] INFO - Connected to 172.23.105.223
[2018-01-09 08:24:03,054] - [remote_util:2990] INFO - running command.raw on 172.23.105.223: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:03,140] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:03,141] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:03,260] - [remote_util:232] INFO - connecting to 172.23.105.243 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:03,365] - [remote_util:266] INFO - Connected to 172.23.105.243
[2018-01-09 08:24:03,855] - [remote_util:2990] INFO - running command.raw on 172.23.105.243: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:03,946] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:03,946] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:04,064] - [remote_util:232] INFO - connecting to 172.23.105.243 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:04,200] - [remote_util:266] INFO - Connected to 172.23.105.243
[2018-01-09 08:24:04,674] - [remote_util:2990] INFO - running command.raw on 172.23.105.243: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:04,764] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:04,765] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:04,882] - [remote_util:232] INFO - connecting to 172.23.105.27 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:05,015] - [remote_util:266] INFO - Connected to 172.23.105.27
[2018-01-09 08:24:05,569] - [remote_util:2990] INFO - running command.raw on 172.23.105.27: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:05,678] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:05,679] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:05,797] - [remote_util:232] INFO - connecting to 172.23.105.27 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:05,928] - [remote_util:266] INFO - Connected to 172.23.105.27
[2018-01-09 08:24:06,485] - [remote_util:2990] INFO - running command.raw on 172.23.105.27: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:06,592] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:06,594] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:06,708] - [xdcrnewbasetests:2705] INFO - {'172.23.105.243': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.27': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.223': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.218': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.212': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.214': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}}
[2018-01-09 08:24:06,721] - [rest_client:1438] INFO - /diag/eval status on 172.23.105.212:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2018-01-09 08:24:06,724] - [remote_util:232] INFO - connecting to 172.23.105.212 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:06,861] - [remote_util:266] INFO - Connected to 172.23.105.212
[2018-01-09 08:24:07,256] - [remote_util:2990] INFO - running command.raw on 172.23.105.212: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:07,369] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:07,370] - [xdcrnewbasetests:428] INFO - 7
[2018-01-09 08:24:07,375] - [remote_util:232] INFO - connecting to 172.23.105.214 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:07,510] - [remote_util:266] INFO - Connected to 172.23.105.214
[2018-01-09 08:24:08,113] - [remote_util:2990] INFO - running command.raw on 172.23.105.214: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:08,284] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:08,286] - [xdcrnewbasetests:428] INFO - 6
[2018-01-09 08:24:08,404] - [remote_util:232] INFO - connecting to 172.23.105.218 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:08,537] - [remote_util:266] INFO - Connected to 172.23.105.218
[2018-01-09 08:24:09,141] - [remote_util:2990] INFO - running command.raw on 172.23.105.218: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:09,263] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:09,264] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:09,382] - [remote_util:232] INFO - connecting to 172.23.105.223 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:09,519] - [remote_util:266] INFO - Connected to 172.23.105.223
[2018-01-09 08:24:09,929] - [remote_util:2990] INFO - running command.raw on 172.23.105.223: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:10,017] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:10,018] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:10,135] - [remote_util:232] INFO - connecting to 172.23.105.243 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:10,241] - [remote_util:266] INFO - Connected to 172.23.105.243
[2018-01-09 08:24:10,676] - [remote_util:2990] INFO - running command.raw on 172.23.105.243: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:10,764] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:10,765] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:10,883] - [remote_util:232] INFO - connecting to 172.23.105.27 with username:root password:couchbase ssh_key:
[2018-01-09 08:24:11,014] - [remote_util:266] INFO - Connected to 172.23.105.27
[2018-01-09 08:24:11,546] - [remote_util:2990] INFO - running command.raw on 172.23.105.27: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:24:11,651] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:24:11,652] - [xdcrnewbasetests:428] INFO - 0
[2018-01-09 08:24:11,767] - [xdcrnewbasetests:2723] INFO - {'172.23.105.243': 0, '172.23.105.27': 0, '172.23.105.223': 0, '172.23.105.218': 0, '172.23.105.212': 7, '172.23.105.214': 6}
[2018-01-09 08:24:11,767] - [xdcrnewbasetests:2486] INFO - ==== XDCRNewbasetests setup is started for test #14 test_verify_mb19802_1 ====
[2018-01-09 08:24:11,769] - [xdcrnewbasetests:1183] INFO - removing xdcr/nodes settings
[2018-01-09 08:24:11,802] - [xdcrnewbasetests:1193] INFO - cleanup [ip:172.23.105.212 port:8091 ssh_username:root, ip:172.23.105.214 port:8091 ssh_username:root]
[2018-01-09 08:24:11,813] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.105.212
[2018-01-09 08:24:11,813] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2018-01-09 08:24:13,847] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.212:8091
[2018-01-09 08:24:13,853] - [cluster_helper:80] INFO - ns_server @ 172.23.105.212:8091 is running
[2018-01-09 08:24:13,871] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.105.214
[2018-01-09 08:24:13,872] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2018-01-09 08:24:15,911] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.214:8091
[2018-01-09 08:24:15,920] - [cluster_helper:80] INFO - ns_server @ 172.23.105.214:8091 is running
[2018-01-09 08:24:15,920] - [xdcrnewbasetests:1183] INFO - removing xdcr/nodes settings
[2018-01-09 08:24:15,972] - [xdcrnewbasetests:1193] INFO - cleanup [ip:172.23.105.218 port:8091 ssh_username:root, ip:172.23.105.223 port:8091 ssh_username:root]
[2018-01-09 08:24:15,993] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.105.218
[2018-01-09 08:24:15,993] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2018-01-09 08:24:18,047] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.218:8091
[2018-01-09 08:24:18,057] - [cluster_helper:80] INFO - ns_server @ 172.23.105.218:8091 is running
[2018-01-09 08:24:18,086] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.105.223
[2018-01-09 08:24:18,087] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2018-01-09 08:24:20,120] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.223:8091
[2018-01-09 08:24:20,128] - [cluster_helper:80] INFO - ns_server @ 172.23.105.223:8091 is running
[2018-01-09 08:24:20,128] - [xdcrnewbasetests:2733] INFO - Initializing all clusters...
[2018-01-09 08:24:20,140] - [rest_client:1438] INFO - /diag/eval status on 172.23.105.212:8091: True content: [5,5] command: cluster_compat_mode:get_compat_version().
[2018-01-09 08:24:20,793] - [task:125] INFO - server: ip:172.23.105.212 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.212', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.212', 'uptime': u'4893', 'mcdMemoryReserved': 3104, 'storageTotalRam': 3880, 'hostname': u'172.23.105.212:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7facb41552d0>], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-1632-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3479732224, 'memoryTotal': 4069322752, 'memoryQuota': 2069, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-01-09 08:24:20,794] - [rest_client:904] INFO - pools/default params : memoryQuota=2069
[2018-01-09 08:24:20,800] - [rest_client:939] INFO - settings/indexes params : storageMode=forestdb
[2018-01-09 08:24:20,805] - [rest_client:804] ERROR - POST http://172.23.105.212:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in 5.0 enterprise edition."}} auth: Administrator:password
[2018-01-09 08:24:20,805] - [rest_client:822] INFO - settings/web params on 172.23.105.212:8091:username=Administrator&password=password&port=8091
[2018-01-09 08:24:20,939] - [task:125] INFO - server: ip:172.23.105.214 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.214', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.214', 'uptime': u'22961', 'mcdMemoryReserved': 3047, 'storageTotalRam': 3809, 'hostname': u'172.23.105.214:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7facb41550d0>], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-1632-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3604615168, 'memoryTotal': 3994292224, 'memoryQuota': 2285, 'mcdMemoryAllocated': 3047, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-01-09 08:24:20,940] - [rest_client:904] INFO - pools/default params : memoryQuota=2031
[2018-01-09 08:24:20,951] - [rest_client:939] INFO - settings/indexes params : storageMode=forestdb
[2018-01-09 08:24:20,959] - [rest_client:804] ERROR - POST http://172.23.105.214:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in 5.0 enterprise edition."}} auth: Administrator:password
[2018-01-09 08:24:20,960] - [rest_client:822] INFO - settings/web params on 172.23.105.214:8091:username=Administrator&password=password&port=8091
[2018-01-09 08:24:22,172] - [task:425] INFO - adding node 172.23.105.214:8091 to cluster
[2018-01-09 08:24:22,172] - [rest_client:1202] INFO - adding remote node @172.23.105.214:8091 to this cluster @172.23.105.212:8091
[2018-01-09 08:24:26,417] - [rest_client:1420] INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.212,ns_1@172.23.105.214'}
[2018-01-09 08:24:26,424] - [rest_client:1425] INFO - rebalance operation started
[2018-01-09 08:24:26,429] - [rest_client:1573] INFO - rebalance percentage : 0.00 %
[2018-01-09 08:24:26,430] - [task:491] INFO - Rebalance - status: running, progress: 0
[2018-01-09 08:24:36,456] - [task:491] INFO - Rebalance - status: none, progress: 100
[2018-01-09 08:24:36,464] - [task:550] INFO - rebalancing was completed with progress: 100% in 10.0394639969 sec
[2018-01-09 08:24:36,480] - [rest_client:1438] INFO - /diag/eval status on 172.23.105.218:8091: True content: [5,5] command: cluster_compat_mode:get_compat_version().
[2018-01-09 08:24:36,823] - [task:125] INFO - server: ip:172.23.105.218 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.218', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.218', 'uptime': u'2976', 'mcdMemoryReserved': 3047, 'storageTotalRam': 3809, 'hostname': u'172.23.105.218:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7facb4155b10>], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-1632-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3543351296, 'memoryTotal': 3994292224, 'memoryQuota': 2031, 'mcdMemoryAllocated': 3047, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-01-09 08:24:36,824] - [rest_client:904] INFO - pools/default params : memoryQuota=2031
[2018-01-09 08:24:36,836] - [rest_client:939] INFO - settings/indexes params : storageMode=forestdb
[2018-01-09 08:24:36,842] - [rest_client:804] ERROR - POST http://172.23.105.218:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in 5.0 enterprise edition."}} auth: Administrator:password
[2018-01-09 08:24:36,843] - [rest_client:822] INFO - settings/web params on 172.23.105.218:8091:username=Administrator&password=password&port=8091
[2018-01-09 08:24:37,079] - [task:125] INFO - server: ip:172.23.105.223 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.223', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.223', 'uptime': u'23009', 'mcdMemoryReserved': 3104, 'storageTotalRam': 3880, 'hostname': u'172.23.105.223:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7facb41552d0>], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-1632-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3672809472, 'memoryTotal': 4069212160, 'memoryQuota': 2328, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-01-09 08:24:37,080] - [rest_client:904] INFO - pools/default params : memoryQuota=2069
[2018-01-09 08:24:37,096] - [rest_client:939] INFO - settings/indexes params : storageMode=forestdb
[2018-01-09 08:24:37,109] - [rest_client:804] ERROR - POST http://172.23.105.223:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in 5.0 enterprise edition."}} auth: Administrator:password
[2018-01-09 08:24:37,110] - [rest_client:822] INFO - settings/web params on 172.23.105.223:8091:username=Administrator&password=password&port=8091
[2018-01-09 08:24:38,244] - [task:425] INFO - adding node 172.23.105.223:8091 to cluster
[2018-01-09 08:24:38,245] - [rest_client:1202] INFO - adding remote node @172.23.105.223:8091 to this cluster @172.23.105.218:8091
[2018-01-09 08:24:40,835] - [rest_client:1420] INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.223,ns_1@172.23.105.218'}
[2018-01-09 08:24:40,850] - [rest_client:1425] INFO - rebalance operation started
[2018-01-09 08:24:40,856] - [rest_client:1573] INFO - rebalance percentage : 0.00 %
[2018-01-09 08:24:40,856] - [task:491] INFO - Rebalance - status: running, progress: 0
[2018-01-09 08:24:50,887] - [task:491] INFO - Rebalance - status: none, progress: 100
[2018-01-09 08:24:50,901] - [task:550] INFO - rebalancing was completed with progress: 100% in 10.0509660244 sec
[2018-01-09 08:24:50,928] - [rest_client:804] ERROR - DELETE http://172.23.105.212:8091/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password
[2018-01-09 08:24:50,929] - [internal_user:36] INFO - Exception while deleting user. Exception is -"User was not found."
[2018-01-09 08:25:11,120] - [rest_client:804] ERROR - DELETE http://172.23.105.218:8091/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password
[2018-01-09 08:25:11,121] - [internal_user:36] INFO - Exception while deleting user. Exception is -"User was not found."
[2018-01-09 08:25:31,560] - [rest_client:1959] INFO - Node versions in cluster [u'5.5.0-1632-enterprise', u'5.5.0-1632-enterprise']
[2018-01-09 08:25:31,560] - [rest_client:2222] INFO - http://172.23.105.212:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=sasl&replicaIndex=1&name=default&saslPassword=None&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=1531
[2018-01-09 08:25:31,570] - [rest_client:2247] INFO - 0.01 seconds to create bucket default
[2018-01-09 08:25:31,570] - [bucket_helper:343] INFO - waiting for memcached bucket : default in 172.23.105.212 to accept set ops
[2018-01-09 08:25:32,841] - [data_helper:295] INFO - creating direct client 172.23.105.212:11210 default
[2018-01-09 08:25:32,935] - [data_helper:295] INFO - creating direct client 172.23.105.214:11210 default
[2018-01-09 08:25:33,086] - [data_helper:295] INFO - creating direct client 172.23.105.212:11210 default
[2018-01-09 08:25:33,170] - [data_helper:295] INFO - creating direct client 172.23.105.214:11210 default
[2018-01-09 08:25:33,817] - [task:312] INFO - bucket 'default' was created with per node RAM quota: 1531
[2018-01-09 08:25:34,000] - [rest_client:1959] INFO - Node versions in cluster [u'5.5.0-1632-enterprise', u'5.5.0-1632-enterprise']
[2018-01-09 08:25:34,001] - [rest_client:2222] INFO - http://172.23.105.218:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=sasl&replicaIndex=1&name=default&saslPassword=None&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=1531
[2018-01-09 08:25:34,025] - [rest_client:2247] INFO - 0.02 seconds to create bucket default
[2018-01-09 08:25:34,026] - [bucket_helper:343] INFO - waiting for memcached bucket : default in 172.23.105.218 to accept set ops
[2018-01-09 08:25:35,768] - [data_helper:295] INFO - creating direct client 172.23.105.218:11210 default
[2018-01-09 08:25:35,894] - [data_helper:295] INFO - creating direct client 172.23.105.223:11210 default
[2018-01-09 08:25:36,144] - [data_helper:295] INFO - creating direct client 172.23.105.218:11210 default
[2018-01-09 08:25:36,252] - [data_helper:295] INFO - creating direct client 172.23.105.223:11210 default
[2018-01-09 08:25:36,944] - [task:312] INFO - bucket 'default' was created with per node RAM quota: 1531
[2018-01-09 08:25:36,971] - [xdcrnewbasetests:2492] INFO - ==== XDCRNewbasetests setup is finished for test #14 test_verify_mb19802_1 ====
[2018-01-09 08:25:36,979] - [rest_client:1087] INFO - adding remote cluster hostname:172.23.105.218:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.105.212:8091
[2018-01-09 08:25:37,007] - [rest_client:1134] INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {}
[2018-01-09 08:25:37,036] - [rest_client:1147] INFO - Replication created with id: f278812563dc5c9befda0fc017bc2f70/default/default
[2018-01-09 08:25:47,071] - [rest_client:2529] INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.105.212
[2018-01-09 08:25:47,131] - [data_helper:295] INFO - creating direct client 172.23.105.212:11210 default
[2018-01-09 08:25:47,216] - [data_helper:295] INFO - creating direct client 172.23.105.214:11210 default
[2018-01-09 08:25:47,424] - [rest_client:1438] INFO - /diag/eval status on 172.23.105.212:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2018-01-09 08:25:47,430] - [remote_util:232] INFO - connecting to 172.23.105.218 with username:root password:couchbase ssh_key:
[2018-01-09 08:25:47,566] - [remote_util:266] INFO - Connected to 172.23.105.218
[2018-01-09 08:25:48,447] - [remote_util:2990] INFO - running command.raw on 172.23.105.218: /etc/init.d/couchbase-server stop
[2018-01-09 08:25:53,590] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:25:53,591] - [remote_util:2825] INFO - Stopping couchbase-server
[2018-01-09 08:25:53,592] - [remote_util:2825] INFO - [60G[[0;32m  OK  [0;39m]
[2018-01-09 08:25:53,593] - [remote_util:2825] INFO - 
[2018-01-09 08:26:33,400] - [remote_util:2990] INFO - running command.raw on 172.23.105.218: /etc/init.d/couchbase-server start
[2018-01-09 08:26:36,262] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:26:36,263] - [remote_util:2825] INFO - Starting couchbase-server
[2018-01-09 08:26:36,263] - [remote_util:2825] INFO - [  OK  ]
[2018-01-09 08:26:36,264] - [xdcrnewbasetests:3468] INFO - sleep for 300 secs.  ...
[2018-01-09 08:31:36,368] - [remote_util:232] INFO - connecting to 172.23.105.212 with username:root password:couchbase ssh_key:
[2018-01-09 08:31:36,507] - [remote_util:266] INFO - Connected to 172.23.105.212
[2018-01-09 08:31:36,909] - [remote_util:2990] INFO - running command.raw on 172.23.105.212: zgrep "batchGetMeta timed out" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2018-01-09 08:31:37,022] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:31:37,023] - [xdcrnewbasetests:428] INFO - 2
FAIL
[2018-01-09 08:31:37,138] - [xdcrnewbasetests:2526] INFO - Collecting logs @ 172.23.105.212
grabbing cbcollect from 172.23.105.212
[2018-01-09 08:31:37,144] - [remote_util:232] INFO - connecting to 172.23.105.212 with username:root password:couchbase ssh_key:
[2018-01-09 08:31:37,280] - [remote_util:266] INFO - Connected to 172.23.105.212
Collecting logs from 172.23.105.212

[2018-01-09 08:31:37,675] - [remote_util:2990] INFO - running command.raw on 172.23.105.212: /opt/couchbase/bin/cbcollect_info 172.23.105.212-20180109-0831-diag.zip 
[2018-01-09 08:33:30,578] - [remote_util:3027] INFO - command executed successfully

Downloading zipped logs from 172.23.105.212
[2018-01-09 08:33:31,985] - [remote_util:2990] INFO - running command.raw on 172.23.105.212: rm -f /root/172.23.105.212-20180109-0831-diag.zip
[2018-01-09 08:33:32,069] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:33:32,188] - [remote_util:232] INFO - connecting to 172.23.105.212 with username:root password:couchbase ssh_key:
[2018-01-09 08:33:32,325] - [remote_util:266] INFO - Connected to 172.23.105.212
Collecting data files from 172.23.105.212

[2018-01-09 08:33:32,739] - [remote_util:2990] INFO - running command.raw on 172.23.105.212: tar -zcvf 172.23.105.212-192018-833-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2018-01-09 08:33:34,151] - [remote_util:3027] INFO - command executed successfully

[2018-01-09 08:33:34,895] - [remote_util:2990] INFO - running command.raw on 172.23.105.212: rm -f /root/172.23.105.212-192018-833-couch.tar.gz
[2018-01-09 08:33:34,977] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:33:34,979] - [xdcrnewbasetests:2526] INFO - Collecting logs @ 172.23.105.214
grabbing cbcollect from 172.23.105.214
[2018-01-09 08:33:34,982] - [remote_util:232] INFO - connecting to 172.23.105.214 with username:root password:couchbase ssh_key:
[2018-01-09 08:33:35,120] - [remote_util:266] INFO - Connected to 172.23.105.214
Collecting logs from 172.23.105.214

[2018-01-09 08:33:35,736] - [remote_util:2990] INFO - running command.raw on 172.23.105.214: /opt/couchbase/bin/cbcollect_info 172.23.105.214-20180109-0833-diag.zip 
[2018-01-09 08:36:17,598] - [remote_util:3027] INFO - command executed successfully

Downloading zipped logs from 172.23.105.214
[2018-01-09 08:36:18,810] - [remote_util:2990] INFO - running command.raw on 172.23.105.214: rm -f /root/172.23.105.214-20180109-0833-diag.zip
[2018-01-09 08:36:18,863] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:36:18,868] - [remote_util:232] INFO - connecting to 172.23.105.214 with username:root password:couchbase ssh_key:
[2018-01-09 08:36:19,003] - [remote_util:266] INFO - Connected to 172.23.105.214
Collecting data files from 172.23.105.214

[2018-01-09 08:36:19,689] - [remote_util:2990] INFO - running command.raw on 172.23.105.214: tar -zcvf 172.23.105.214-192018-836-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2018-01-09 08:36:23,491] - [remote_util:3027] INFO - command executed successfully

[2018-01-09 08:36:24,427] - [remote_util:2990] INFO - running command.raw on 172.23.105.214: rm -f /root/172.23.105.214-192018-836-couch.tar.gz
[2018-01-09 08:36:24,476] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:36:24,479] - [xdcrnewbasetests:2526] INFO - Collecting logs @ 172.23.105.218
grabbing cbcollect from 172.23.105.218
[2018-01-09 08:36:24,482] - [remote_util:232] INFO - connecting to 172.23.105.218 with username:root password:couchbase ssh_key:
[2018-01-09 08:36:24,618] - [remote_util:266] INFO - Connected to 172.23.105.218
Collecting logs from 172.23.105.218

[2018-01-09 08:36:25,255] - [remote_util:2990] INFO - running command.raw on 172.23.105.218: /opt/couchbase/bin/cbcollect_info 172.23.105.218-20180109-0836-diag.zip 
[2018-01-09 08:39:15,349] - [remote_util:3027] INFO - command executed successfully

Downloading zipped logs from 172.23.105.218
[2018-01-09 08:39:16,287] - [remote_util:2990] INFO - running command.raw on 172.23.105.218: rm -f /root/172.23.105.218-20180109-0836-diag.zip
[2018-01-09 08:39:16,346] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:39:16,350] - [remote_util:232] INFO - connecting to 172.23.105.218 with username:root password:couchbase ssh_key:
[2018-01-09 08:39:16,485] - [remote_util:266] INFO - Connected to 172.23.105.218
Collecting data files from 172.23.105.218

[2018-01-09 08:39:17,171] - [remote_util:2990] INFO - running command.raw on 172.23.105.218: tar -zcvf 172.23.105.218-192018-839-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2018-01-09 08:39:19,846] - [remote_util:3027] INFO - command executed successfully

[2018-01-09 08:39:20,654] - [remote_util:2990] INFO - running command.raw on 172.23.105.218: rm -f /root/172.23.105.218-192018-839-couch.tar.gz
[2018-01-09 08:39:20,705] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:39:20,708] - [xdcrnewbasetests:2526] INFO - Collecting logs @ 172.23.105.223
grabbing cbcollect from 172.23.105.223
[2018-01-09 08:39:20,711] - [remote_util:232] INFO - connecting to 172.23.105.223 with username:root password:couchbase ssh_key:
[2018-01-09 08:39:20,847] - [remote_util:266] INFO - Connected to 172.23.105.223
Collecting logs from 172.23.105.223

[2018-01-09 08:39:21,269] - [remote_util:2990] INFO - running command.raw on 172.23.105.223: /opt/couchbase/bin/cbcollect_info 172.23.105.223-20180109-0839-diag.zip 
[2018-01-09 08:41:07,428] - [remote_util:3027] INFO - command executed successfully

Downloading zipped logs from 172.23.105.223
[2018-01-09 08:41:08,394] - [remote_util:2990] INFO - running command.raw on 172.23.105.223: rm -f /root/172.23.105.223-20180109-0839-diag.zip
[2018-01-09 08:41:08,416] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:41:08,421] - [remote_util:232] INFO - connecting to 172.23.105.223 with username:root password:couchbase ssh_key:
[2018-01-09 08:41:08,558] - [remote_util:266] INFO - Connected to 172.23.105.223
Collecting data files from 172.23.105.223

[2018-01-09 08:41:08,987] - [remote_util:2990] INFO - running command.raw on 172.23.105.223: tar -zcvf 172.23.105.223-192018-841-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2018-01-09 08:41:11,113] - [remote_util:3027] INFO - command executed successfully

[2018-01-09 08:41:11,985] - [remote_util:2990] INFO - running command.raw on 172.23.105.223: rm -f /root/172.23.105.223-192018-841-couch.tar.gz
[2018-01-09 08:41:12,007] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:41:12,009] - [xdcrnewbasetests:2526] INFO - Collecting logs @ 172.23.105.243
grabbing cbcollect from 172.23.105.243
[2018-01-09 08:41:12,012] - [remote_util:232] INFO - connecting to 172.23.105.243 with username:root password:couchbase ssh_key:
[2018-01-09 08:41:12,120] - [remote_util:266] INFO - Connected to 172.23.105.243
Collecting logs from 172.23.105.243

[2018-01-09 08:41:12,526] - [remote_util:2990] INFO - running command.raw on 172.23.105.243: /opt/couchbase/bin/cbcollect_info 172.23.105.243-20180109-0841-diag.zip 
[2018-01-09 08:42:27,259] - [remote_util:3027] INFO - command executed successfully

Downloading zipped logs from 172.23.105.243
[2018-01-09 08:42:27,514] - [remote_util:2990] INFO - running command.raw on 172.23.105.243: rm -f /root/172.23.105.243-20180109-0841-diag.zip
[2018-01-09 08:42:27,594] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:42:27,599] - [remote_util:232] INFO - connecting to 172.23.105.243 with username:root password:couchbase ssh_key:
[2018-01-09 08:42:27,702] - [remote_util:266] INFO - Connected to 172.23.105.243
Collecting data files from 172.23.105.243

[2018-01-09 08:42:28,114] - [remote_util:2990] INFO - running command.raw on 172.23.105.243: tar -zcvf 172.23.105.243-192018-842-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2018-01-09 08:42:28,162] - [remote_util:3027] INFO - command executed successfully

[2018-01-09 08:42:28,365] - [remote_util:2990] INFO - running command.raw on 172.23.105.243: rm -f /root/172.23.105.243-192018-842-couch.tar.gz
[2018-01-09 08:42:28,442] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:42:28,444] - [xdcrnewbasetests:2526] INFO - Collecting logs @ 172.23.105.27
grabbing cbcollect from 172.23.105.27
[2018-01-09 08:42:28,448] - [remote_util:232] INFO - connecting to 172.23.105.27 with username:root password:couchbase ssh_key:
[2018-01-09 08:42:28,581] - [remote_util:266] INFO - Connected to 172.23.105.27
Collecting logs from 172.23.105.27

[2018-01-09 08:42:29,116] - [remote_util:2990] INFO - running command.raw on 172.23.105.27: /opt/couchbase/bin/cbcollect_info 172.23.105.27-20180109-0842-diag.zip 
[2018-01-09 08:43:58,253] - [remote_util:3027] INFO - command executed successfully

Downloading zipped logs from 172.23.105.27
[2018-01-09 08:43:58,538] - [remote_util:2990] INFO - running command.raw on 172.23.105.27: rm -f /root/172.23.105.27-20180109-0842-diag.zip
[2018-01-09 08:43:58,566] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:43:58,573] - [remote_util:232] INFO - connecting to 172.23.105.27 with username:root password:couchbase ssh_key:
[2018-01-09 08:43:58,704] - [remote_util:266] INFO - Connected to 172.23.105.27
Collecting data files from 172.23.105.27

[2018-01-09 08:43:59,253] - [remote_util:2990] INFO - running command.raw on 172.23.105.27: tar -zcvf 172.23.105.27-192018-843-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2018-01-09 08:43:59,313] - [remote_util:3027] INFO - command executed successfully

[2018-01-09 08:43:59,505] - [remote_util:2990] INFO - running command.raw on 172.23.105.27: rm -f /root/172.23.105.27-192018-843-couch.tar.gz
[2018-01-09 08:43:59,590] - [remote_util:3027] INFO - command executed successfully
[2018-01-09 08:43:59,670] - [xdcrnewbasetests:2544] INFO - ====  XDCRNewbasetests cleanup is started for test #14 test_verify_mb19802_1 ====
[2018-01-09 08:43:59,671] - [xdcrnewbasetests:1183] INFO - removing xdcr/nodes settings
[2018-01-09 08:43:59,687] - [rest_client:1170] INFO - Deleting replication /controller/cancelXDCR/f278812563dc5c9befda0fc017bc2f70%2Fdefault%2Fdefault
[2018-01-09 08:43:59,699] - [rest_client:1174] INFO - Replication deleted successfully
[2018-01-09 08:43:59,706] - [rest_client:1118] INFO - removing remote cluster name:remote_cluster_C1-C2
[2018-01-09 08:43:59,733] - [xdcrnewbasetests:1193] INFO - cleanup [ip:172.23.105.212 port:8091 ssh_username:root, ip:172.23.105.214 port:8091 ssh_username:root]
[2018-01-09 08:43:59,758] - [bucket_helper:142] INFO - deleting existing buckets [u'default'] on 172.23.105.212
[2018-01-09 08:43:59,758] - [bucket_helper:144] INFO - remove bucket default ...
[2018-01-09 08:44:01,381] - [bucket_helper:158] INFO - deleted bucket : default from 172.23.105.212
[2018-01-09 08:44:01,382] - [bucket_helper:236] INFO - waiting for bucket deletion to complete....
[2018-01-09 08:44:01,388] - [rest_client:133] INFO - node 172.23.105.212 existing buckets : []
[2018-01-09 08:44:01,389] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2018-01-09 08:44:03,415] - [cluster_helper:254] INFO - rebalancing all nodes in order to remove nodes
[2018-01-09 08:44:03,422] - [rest_client:1420] INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.105.214', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.212,ns_1@172.23.105.214'}
[2018-01-09 08:44:03,433] - [rest_client:1425] INFO - rebalance operation started
[2018-01-09 08:44:03,439] - [rest_client:1573] INFO - rebalance percentage : 0.00 %
[2018-01-09 08:44:23,471] - [rest_client:1512] INFO - rebalance progress took 20.0369491577 seconds 
[2018-01-09 08:44:23,471] - [rest_client:1513] INFO - sleep for 10 seconds after rebalance...
[2018-01-09 08:44:33,499] - [cluster_helper:294] INFO - removed all the nodes from cluster associated with ip:172.23.105.212 port:8091 ssh_username:root ? [(u'ns_1@172.23.105.214', 8091)]
[2018-01-09 08:44:33,508] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.212:8091
[2018-01-09 08:44:33,514] - [cluster_helper:80] INFO - ns_server @ 172.23.105.212:8091 is running
[2018-01-09 08:44:33,527] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.105.214
[2018-01-09 08:44:33,528] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2018-01-09 08:44:35,575] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.214:8091
[2018-01-09 08:44:35,583] - [cluster_helper:80] INFO - ns_server @ 172.23.105.214:8091 is running
Cluster instance shutdown with force
[2018-01-09 08:44:35,584] - [xdcrnewbasetests:1183] INFO - removing xdcr/nodes settings
[2018-01-09 08:44:35,665] - [xdcrnewbasetests:1193] INFO - cleanup [ip:172.23.105.218 port:8091 ssh_username:root, ip:172.23.105.223 port:8091 ssh_username:root]
[2018-01-09 08:44:35,702] - [bucket_helper:142] INFO - deleting existing buckets [u'default'] on 172.23.105.218
[2018-01-09 08:44:35,702] - [bucket_helper:144] INFO - remove bucket default ...
[2018-01-09 08:44:36,901] - [bucket_helper:158] INFO - deleted bucket : default from 172.23.105.218
[2018-01-09 08:44:36,901] - [bucket_helper:236] INFO - waiting for bucket deletion to complete....
[2018-01-09 08:44:36,909] - [rest_client:133] INFO - node 172.23.105.218 existing buckets : []
[2018-01-09 08:44:36,910] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2018-01-09 08:44:38,959] - [cluster_helper:254] INFO - rebalancing all nodes in order to remove nodes
[2018-01-09 08:44:38,966] - [rest_client:1420] INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.105.223', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.223,ns_1@172.23.105.218'}
[2018-01-09 08:44:38,974] - [rest_client:1425] INFO - rebalance operation started
[2018-01-09 08:44:38,988] - [rest_client:1573] INFO - rebalance percentage : 0.00 %
[2018-01-09 08:44:59,026] - [rest_client:1512] INFO - rebalance progress took 20.0517349243 seconds 
[2018-01-09 08:44:59,027] - [rest_client:1513] INFO - sleep for 10 seconds after rebalance...
[2018-01-09 08:45:09,051] - [cluster_helper:294] INFO - removed all the nodes from cluster associated with ip:172.23.105.218 port:8091 ssh_username:root ? [(u'ns_1@172.23.105.223', 8091)]
[2018-01-09 08:45:09,062] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.218:8091
[2018-01-09 08:45:09,078] - [cluster_helper:80] INFO - ns_server @ 172.23.105.218:8091 is running
[2018-01-09 08:45:09,088] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.105.223
[2018-01-09 08:45:09,089] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2018-01-09 08:45:11,125] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.223:8091
[2018-01-09 08:45:11,133] - [cluster_helper:80] INFO - ns_server @ 172.23.105.223:8091 is running
Cluster instance shutdown with force
[2018-01-09 08:45:11,133] - [xdcrnewbasetests:2549] INFO - ====  XDCRNewbasetests cleanup is finished for test #14 test_verify_mb19802_1 ===

======================================================================
FAIL: test_verify_mb19802_1 (xdcr.uniXDCR.unidirectional)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/xdcr/uniXDCR.py", line 699, in test_verify_mb19802_1
    self.assertEqual(count, 0, "batchGetMeta timed out error message found in " + str(node.ip))
AssertionError: batchGetMeta timed out error message found in 172.23.105.212

----------------------------------------------------------------------