./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,fail_on_errors=1,GROUP=P1 -t xdcr.pauseResumeXDCR.PauseResumeTest.replication_with_pause_and_resume,items=30000,rdirection=bidirection,ctopology=chain,replication_type=xmem,rebalance_in=C1-C2,pause_wait=10,pause=C1-C2,GROUP=P1

Test Input params:
{'get-coredumps': 'True', 'fail_on_errors': '1', 'pause': 'C1-C2', 'GROUP': 'P1', 'cluster_name': 'INI_FILE', 'items': '30000', 'case_number': 6, 'conf_file': 'conf/py-xdcr-pause-resume.conf', 'num_nodes': 8, 'pause_wait': '10', 'ctopology': 'chain', 'rdirection': 'bidirection', 'stop-on-failure': 'False', 'ini': 'INI_FILE.ini', 'replication_type': 'xmem', 'get-logs': 'False', 'rebalance_in': 'C1-C2', 'get-cbcollect-info': 'True', 'logs_folder': '/data/workspace/cen006-p1-xxdcr-vset09-01-goxdcr-pause-resume/logs/testrunner-17-Mar-05_09-36-16/test_6', 'spec': 'py-xdcr-pause-resume'}
[2017-03-05 11:04:18,663] - [remote_util:195] INFO - connecting to 172.23.107.198 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:18,831] - [remote_util:229] INFO - Connected to 172.23.107.198
[2017-03-05 11:04:19,387] - [remote_util:564] INFO - 172.23.107.198 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for Erlang/Memcached crashes on 172.23.107.198 ... 
dump files not found on 172.23.107.198
[2017-03-05 11:04:19,539] - [remote_util:195] INFO - connecting to 172.23.107.171 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:19,709] - [remote_util:229] INFO - Connected to 172.23.107.171
[2017-03-05 11:04:20,443] - [remote_util:564] INFO - 172.23.107.171 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for Erlang/Memcached crashes on 172.23.107.171 ... 
dump files not found on 172.23.107.171
[2017-03-05 11:04:20,622] - [remote_util:195] INFO - connecting to 172.23.107.120 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:20,793] - [remote_util:229] INFO - Connected to 172.23.107.120
[2017-03-05 11:04:21,456] - [remote_util:564] INFO - 172.23.107.120 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for Erlang/Memcached crashes on 172.23.107.120 ... 
dump files not found on 172.23.107.120
[2017-03-05 11:04:21,637] - [remote_util:195] INFO - connecting to 172.23.107.169 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:21,769] - [remote_util:229] INFO - Connected to 172.23.107.169
[2017-03-05 11:04:22,347] - [remote_util:564] INFO - 172.23.107.169 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for Erlang/Memcached crashes on 172.23.107.169 ... 
dump files not found on 172.23.107.169
[2017-03-05 11:04:22,522] - [remote_util:195] INFO - connecting to 172.23.105.178 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:22,653] - [remote_util:229] INFO - Connected to 172.23.105.178
[2017-03-05 11:04:23,322] - [remote_util:564] INFO - 172.23.105.178 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for Erlang/Memcached crashes on 172.23.105.178 ... 
dump files not found on 172.23.105.178
[2017-03-05 11:04:23,500] - [remote_util:195] INFO - connecting to 172.23.105.179 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:23,633] - [remote_util:229] INFO - Connected to 172.23.105.179
[2017-03-05 11:04:24,230] - [remote_util:564] INFO - 172.23.105.179 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for Erlang/Memcached crashes on 172.23.105.179 ... 
dump files not found on 172.23.105.179
[2017-03-05 11:04:24,377] - [remote_util:195] INFO - connecting to 172.23.105.180 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:24,510] - [remote_util:229] INFO - Connected to 172.23.105.180
[2017-03-05 11:04:25,074] - [remote_util:564] INFO - 172.23.105.180 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for Erlang/Memcached crashes on 172.23.105.180 ... 
dump files not found on 172.23.105.180
[2017-03-05 11:04:25,233] - [remote_util:195] INFO - connecting to 172.23.105.181 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:25,364] - [remote_util:229] INFO - Connected to 172.23.105.181
[2017-03-05 11:04:25,969] - [remote_util:564] INFO - 172.23.105.181 **** The version file /opt/couchbase/ VERSION.txt  exists
replication_with_pause_and_resume (xdcr.pauseResumeXDCR.PauseResumeTest) ... looking for Erlang/Memcached crashes on 172.23.105.181 ... 
dump files not found on 172.23.105.181
[2017-03-05 11:04:26,168] - [rest_client:1338] INFO - /diag/eval status on 172.23.107.198:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2017-03-05 11:04:26,172] - [remote_util:195] INFO - connecting to 172.23.107.198 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:26,340] - [remote_util:229] INFO - Connected to 172.23.107.198
[2017-03-05 11:04:26,901] - [remote_util:2632] INFO - running command.raw on 172.23.107.198: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:26,974] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:26,974] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:27,093] - [remote_util:195] INFO - connecting to 172.23.107.198 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:27,260] - [remote_util:229] INFO - Connected to 172.23.107.198
[2017-03-05 11:04:27,810] - [remote_util:2632] INFO - running command.raw on 172.23.107.198: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:27,932] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:27,933] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:28,050] - [remote_util:195] INFO - connecting to 172.23.107.171 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:28,183] - [remote_util:229] INFO - Connected to 172.23.107.171
[2017-03-05 11:04:28,777] - [remote_util:2632] INFO - running command.raw on 172.23.107.171: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:28,855] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:28,855] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:28,972] - [remote_util:195] INFO - connecting to 172.23.107.171 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:29,142] - [remote_util:229] INFO - Connected to 172.23.107.171
[2017-03-05 11:04:29,752] - [remote_util:2632] INFO - running command.raw on 172.23.107.171: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:29,822] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:29,822] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:29,940] - [remote_util:195] INFO - connecting to 172.23.107.120 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:30,111] - [remote_util:229] INFO - Connected to 172.23.107.120
[2017-03-05 11:04:30,756] - [remote_util:2632] INFO - running command.raw on 172.23.107.120: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:30,819] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:30,820] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:30,826] - [remote_util:195] INFO - connecting to 172.23.107.120 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:30,996] - [remote_util:229] INFO - Connected to 172.23.107.120
[2017-03-05 11:04:31,609] - [remote_util:2632] INFO - running command.raw on 172.23.107.120: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:31,711] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:31,711] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:31,830] - [remote_util:195] INFO - connecting to 172.23.107.169 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:32,001] - [remote_util:229] INFO - Connected to 172.23.107.169
[2017-03-05 11:04:32,597] - [remote_util:2632] INFO - running command.raw on 172.23.107.169: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:32,658] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:32,658] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:32,777] - [remote_util:195] INFO - connecting to 172.23.107.169 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:32,945] - [remote_util:229] INFO - Connected to 172.23.107.169
[2017-03-05 11:04:33,552] - [remote_util:2632] INFO - running command.raw on 172.23.107.169: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:33,609] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:33,610] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:33,728] - [remote_util:195] INFO - connecting to 172.23.105.178 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:33,899] - [remote_util:229] INFO - Connected to 172.23.105.178
[2017-03-05 11:04:34,439] - [remote_util:2632] INFO - running command.raw on 172.23.105.178: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:34,486] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:34,487] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:34,606] - [remote_util:195] INFO - connecting to 172.23.105.178 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:34,737] - [remote_util:229] INFO - Connected to 172.23.105.178
[2017-03-05 11:04:35,293] - [remote_util:2632] INFO - running command.raw on 172.23.105.178: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:35,411] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:35,411] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:35,417] - [remote_util:195] INFO - connecting to 172.23.105.179 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:35,529] - [remote_util:229] INFO - Connected to 172.23.105.179
[2017-03-05 11:04:36,077] - [remote_util:2632] INFO - running command.raw on 172.23.105.179: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:36,192] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:36,192] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:36,312] - [remote_util:195] INFO - connecting to 172.23.105.179 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:36,456] - [remote_util:229] INFO - Connected to 172.23.105.179
[2017-03-05 11:04:37,005] - [remote_util:2632] INFO - running command.raw on 172.23.105.179: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:37,111] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:37,112] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:37,230] - [remote_util:195] INFO - connecting to 172.23.105.180 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:37,398] - [remote_util:229] INFO - Connected to 172.23.105.180
[2017-03-05 11:04:37,967] - [remote_util:2632] INFO - running command.raw on 172.23.105.180: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:38,085] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:38,085] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:38,204] - [remote_util:195] INFO - connecting to 172.23.105.180 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:38,336] - [remote_util:229] INFO - Connected to 172.23.105.180
[2017-03-05 11:04:38,948] - [remote_util:2632] INFO - running command.raw on 172.23.105.180: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:39,066] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:39,066] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:39,185] - [remote_util:195] INFO - connecting to 172.23.105.181 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:39,360] - [remote_util:229] INFO - Connected to 172.23.105.181
[2017-03-05 11:04:39,931] - [remote_util:2632] INFO - running command.raw on 172.23.105.181: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:40,045] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:40,045] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:40,164] - [remote_util:195] INFO - connecting to 172.23.105.181 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:40,296] - [remote_util:229] INFO - Connected to 172.23.105.181
[2017-03-05 11:04:40,868] - [remote_util:2632] INFO - running command.raw on 172.23.105.181: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:40,983] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:40,983] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:41,098] - [xdcrnewbasetests:2618] INFO - {'172.23.107.169': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.171': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.198': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.120': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.180': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.181': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.179': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.178': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}}
[2017-03-05 11:04:41,110] - [rest_client:1338] INFO - /diag/eval status on 172.23.107.198:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2017-03-05 11:04:41,114] - [remote_util:195] INFO - connecting to 172.23.107.198 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:41,281] - [remote_util:229] INFO - Connected to 172.23.107.198
[2017-03-05 11:04:41,821] - [remote_util:2632] INFO - running command.raw on 172.23.107.198: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:41,952] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:41,952] - [xdcrnewbasetests:414] INFO - 8
[2017-03-05 11:04:42,071] - [remote_util:195] INFO - connecting to 172.23.107.171 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:42,244] - [remote_util:229] INFO - Connected to 172.23.107.171
[2017-03-05 11:04:42,870] - [remote_util:2632] INFO - running command.raw on 172.23.107.171: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:42,941] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:42,942] - [xdcrnewbasetests:414] INFO - 1
[2017-03-05 11:04:43,060] - [remote_util:195] INFO - connecting to 172.23.107.120 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:43,231] - [remote_util:229] INFO - Connected to 172.23.107.120
[2017-03-05 11:04:43,884] - [remote_util:2632] INFO - running command.raw on 172.23.107.120: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:44,016] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:44,017] - [xdcrnewbasetests:414] INFO - 4
[2017-03-05 11:04:44,136] - [remote_util:195] INFO - connecting to 172.23.107.169 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:44,268] - [remote_util:229] INFO - Connected to 172.23.107.169
[2017-03-05 11:04:44,915] - [remote_util:2632] INFO - running command.raw on 172.23.107.169: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:44,974] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:44,975] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:45,093] - [remote_util:195] INFO - connecting to 172.23.105.178 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:45,262] - [remote_util:229] INFO - Connected to 172.23.105.178
[2017-03-05 11:04:45,846] - [remote_util:2632] INFO - running command.raw on 172.23.105.178: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:45,955] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:45,956] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:46,074] - [remote_util:195] INFO - connecting to 172.23.105.179 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:46,243] - [remote_util:229] INFO - Connected to 172.23.105.179
[2017-03-05 11:04:46,828] - [remote_util:2632] INFO - running command.raw on 172.23.105.179: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:46,943] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:46,944] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:46,949] - [remote_util:195] INFO - connecting to 172.23.105.180 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:47,096] - [remote_util:229] INFO - Connected to 172.23.105.180
[2017-03-05 11:04:47,644] - [remote_util:2632] INFO - running command.raw on 172.23.105.180: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:47,698] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:47,699] - [xdcrnewbasetests:414] INFO - 0
[2017-03-05 11:04:47,817] - [remote_util:195] INFO - connecting to 172.23.105.181 with username:root password:couchbase ssh_key:
[2017-03-05 11:04:47,953] - [remote_util:229] INFO - Connected to 172.23.105.181
[2017-03-05 11:04:48,531] - [remote_util:2632] INFO - running command.raw on 172.23.105.181: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-03-05 11:04:48,650] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:04:48,651] - [xdcrnewbasetests:414] INFO - 1
[2017-03-05 11:04:48,766] - [xdcrnewbasetests:2636] INFO - {'172.23.107.169': 0, '172.23.107.171': 1, '172.23.107.198': 8, '172.23.107.120': 4, '172.23.105.180': 0, '172.23.105.181': 1, '172.23.105.179': 0, '172.23.105.178': 0}
[2017-03-05 11:04:48,766] - [xdcrnewbasetests:2433] INFO - ==== XDCRNewbasetests setup is started for test #6 replication_with_pause_and_resume ====
[2017-03-05 11:04:48,769] - [xdcrnewbasetests:1149] INFO - removing xdcr/nodes settings
[2017-03-05 11:04:48,806] - [xdcrnewbasetests:1159] INFO - cleanup [ip:172.23.107.198 port:8091 ssh_username:root, ip:172.23.107.171 port:8091 ssh_username:root]
[2017-03-05 11:04:48,820] - [bucket_helper:138] INFO - deleting existing buckets [] on 172.23.107.198
[2017-03-05 11:04:48,853] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.198:8091
[2017-03-05 11:04:48,860] - [cluster_helper:80] INFO - ns_server @ 172.23.107.198:8091 is running
[2017-03-05 11:04:48,871] - [bucket_helper:138] INFO - deleting existing buckets [] on 172.23.107.171
[2017-03-05 11:04:48,910] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.171:8091
[2017-03-05 11:04:48,919] - [cluster_helper:80] INFO - ns_server @ 172.23.107.171:8091 is running
[2017-03-05 11:04:48,919] - [xdcrnewbasetests:1149] INFO - removing xdcr/nodes settings
[2017-03-05 11:04:49,005] - [xdcrnewbasetests:1159] INFO - cleanup [ip:172.23.107.120 port:8091 ssh_username:root, ip:172.23.107.169 port:8091 ssh_username:root]
[2017-03-05 11:04:49,042] - [bucket_helper:138] INFO - deleting existing buckets [] on 172.23.107.120
[2017-03-05 11:04:49,122] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.120:8091
[2017-03-05 11:04:49,130] - [cluster_helper:80] INFO - ns_server @ 172.23.107.120:8091 is running
[2017-03-05 11:04:49,143] - [bucket_helper:138] INFO - deleting existing buckets [] on 172.23.107.169
[2017-03-05 11:04:49,176] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.169:8091
[2017-03-05 11:04:49,183] - [cluster_helper:80] INFO - ns_server @ 172.23.107.169:8091 is running
[2017-03-05 11:04:49,183] - [xdcrnewbasetests:2646] INFO - Initializing all clusters...
[2017-03-05 11:04:49,195] - [rest_client:1338] INFO - /diag/eval status on 172.23.107.198:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2017-03-05 11:04:49,782] - [task:117] INFO - server: ip:172.23.107.198 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.198', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.198', 'uptime': u'5336', 'mcdMemoryReserved': 3104, 'hostname': u'172.23.107.198:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x25c2e10>], 'moxi': 11211, 'port': u'8091', 'version': u'5.0.0-2194-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327680, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3473195008, 'memoryTotal': 4069322752, 'memoryQuota': 2069, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2017-03-05 11:04:49,783] - [rest_client:891] INFO - pools/default params : memoryQuota=2069
[2017-03-05 11:04:49,791] - [rest_client:928] INFO - settings/indexes params : storageMode=forestdb
[2017-03-05 11:04:49,801] - [rest_client:807] INFO - settings/web params on 172.23.107.198:8091:username=Administrator&password=welcome&port=8091
[2017-03-05 11:04:49,839] - [task:117] INFO - server: ip:172.23.107.171 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.171', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.171', 'uptime': u'5467', 'mcdMemoryReserved': 3104, 'hostname': u'172.23.107.171:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x25c2e10>], 'moxi': 11211, 'port': u'8091', 'version': u'5.0.0-2194-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327680, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3647000576, 'memoryTotal': 4069695488, 'memoryQuota': 2328, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2017-03-05 11:04:49,840] - [rest_client:891] INFO - pools/default params : memoryQuota=2069
[2017-03-05 11:04:49,847] - [rest_client:928] INFO - settings/indexes params : storageMode=forestdb
[2017-03-05 11:04:49,855] - [rest_client:807] INFO - settings/web params on 172.23.107.171:8091:username=Administrator&password=welcome&port=8091
[2017-03-05 11:04:51,416] - [task:414] INFO - adding node 172.23.107.171:8091 to cluster
[2017-03-05 11:04:51,417] - [rest_client:1104] INFO - adding remote node @172.23.107.171:8091 to this cluster @172.23.107.198:8091
[2017-03-05 11:04:58,098] - [rest_client:1322] INFO - rebalance params : password=welcome&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.171%2Cns_1%40172.23.107.198
[2017-03-05 11:04:58,107] - [rest_client:1326] INFO - rebalance operation started
[2017-03-05 11:04:58,123] - [rest_client:1457] INFO - rebalance percentage : 0.00 %
[2017-03-05 11:05:08,160] - [task:537] INFO - rebalancing was completed with progress: 100% in 10.0525650978 sec
[2017-03-05 11:05:08,174] - [rest_client:1338] INFO - /diag/eval status on 172.23.107.120:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2017-03-05 11:05:08,810] - [task:117] INFO - server: ip:172.23.107.120 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.120', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.120', 'uptime': u'5494', 'mcdMemoryReserved': 3104, 'hostname': u'172.23.107.120:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x25c2b90>], 'moxi': 11211, 'port': u'8091', 'version': u'5.0.0-2194-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327680, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3486670848, 'memoryTotal': 4069695488, 'memoryQuota': 2069, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2017-03-05 11:05:08,811] - [rest_client:891] INFO - pools/default params : memoryQuota=2069
[2017-03-05 11:05:08,820] - [rest_client:928] INFO - settings/indexes params : storageMode=forestdb
[2017-03-05 11:05:08,830] - [rest_client:807] INFO - settings/web params on 172.23.107.120:8091:username=Administrator&password=welcome&port=8091
[2017-03-05 11:05:08,869] - [task:117] INFO - server: ip:172.23.107.169 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.169', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.169', 'uptime': u'5492', 'mcdMemoryReserved': 3104, 'hostname': u'172.23.107.169:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x25c2b90>], 'moxi': 11211, 'port': u'8091', 'version': u'5.0.0-2194-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327680, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3665039360, 'memoryTotal': 4069695488, 'memoryQuota': 2328, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2017-03-05 11:05:08,870] - [rest_client:891] INFO - pools/default params : memoryQuota=2069
[2017-03-05 11:05:08,874] - [rest_client:928] INFO - settings/indexes params : storageMode=forestdb
[2017-03-05 11:05:08,882] - [rest_client:807] INFO - settings/web params on 172.23.107.169:8091:username=Administrator&password=welcome&port=8091
[2017-03-05 11:05:09,902] - [task:414] INFO - adding node 172.23.107.169:8091 to cluster
[2017-03-05 11:05:09,902] - [rest_client:1104] INFO - adding remote node @172.23.107.169:8091 to this cluster @172.23.107.120:8091
[2017-03-05 11:05:12,876] - [rest_client:1322] INFO - rebalance params : password=welcome&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.107.120
[2017-03-05 11:05:12,887] - [rest_client:1326] INFO - rebalance operation started
[2017-03-05 11:05:12,899] - [rest_client:1457] INFO - rebalance percentage : 0.00 %
[2017-03-05 11:05:22,944] - [task:537] INFO - rebalancing was completed with progress: 100% in 10.0564720631 sec
[2017-03-05 11:05:23,200] - [rest_client:1963] INFO - http://172.23.107.198:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1569&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None
[2017-03-05 11:05:23,213] - [rest_client:1988] INFO - 0.01 seconds to create bucket default
[2017-03-05 11:05:23,213] - [bucket_helper:322] INFO - waiting for memcached bucket : default in 172.23.107.198 to accept set ops
[2017-03-05 11:05:24,606] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2017-03-05 11:05:24,758] - [data_helper:295] INFO - creating direct client 172.23.107.198:11210 default
[2017-03-05 11:05:24,947] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2017-03-05 11:05:25,571] - [data_helper:295] INFO - creating direct client 172.23.107.198:11210 default
[2017-03-05 11:05:26,308] - [task:298] INFO - bucket 'default' was created with per node RAM quota: 1569
[2017-03-05 11:05:26,974] - [rest_client:1963] INFO - http://172.23.107.120:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1569&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None
[2017-03-05 11:05:26,994] - [rest_client:1988] INFO - 0.02 seconds to create bucket default
[2017-03-05 11:05:26,995] - [bucket_helper:322] INFO - waiting for memcached bucket : default in 172.23.107.120 to accept set ops
[2017-03-05 11:05:28,373] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2017-03-05 11:05:28,478] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2017-03-05 11:05:28,629] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2017-03-05 11:05:28,727] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2017-03-05 11:05:29,791] - [task:298] INFO - bucket 'default' was created with per node RAM quota: 1569
[2017-03-05 11:05:29,815] - [xdcrnewbasetests:2439] INFO - ==== XDCRNewbasetests setup is finished for test #6 replication_with_pause_and_resume ====
[2017-03-05 11:05:29,825] - [rest_client:994] INFO - adding remote cluster hostname:172.23.107.120:8091 with username:password Administrator:welcome name:remote_cluster_C1-C2 to source node: 172.23.107.198:8091
[2017-03-05 11:05:29,869] - [rest_client:994] INFO - adding remote cluster hostname:172.23.107.198:8091 with username:password Administrator:welcome name:remote_cluster_C2-C1 to source node: 172.23.107.120:8091
[2017-03-05 11:05:29,906] - [rest_client:1041] INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {}
[2017-03-05 11:05:29,950] - [rest_client:1054] INFO - Replication created with id: 99d1ac9e65b72b50d2012b604c5cabf0/default/default
[2017-03-05 11:05:29,957] - [rest_client:1041] INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C2-C1 with settings {}
[2017-03-05 11:05:30,023] - [rest_client:1054] INFO - Replication created with id: 25fb4ccb374095c30d53e6509fe418db/default/default
[2017-03-05 11:05:30,105] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2017-03-05 11:05:30,221] - [data_helper:295] INFO - creating direct client 172.23.107.198:11210 default
[2017-03-05 11:05:30,466] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2017-03-05 11:05:30,608] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2017-03-05 11:05:30,720] - [xdcrnewbasetests:3327] INFO - sleep for 20 secs.  ...
[2017-03-05 11:05:50,858] - [rest_client:2238] INFO - Updated pauseRequested=true on bucket'default' on 172.23.107.198
[2017-03-05 11:05:51,022] - [rest_client:2238] INFO - Updated pauseRequested=true on bucket'default' on 172.23.107.120
[2017-03-05 11:05:51,072] - [xdcrnewbasetests:1896] INFO - Starting rebalance-in nodes:[ip:172.23.105.181 port:8091 ssh_username:root] at C1 cluster 172.23.107.198
[2017-03-05 11:05:51,089] - [xdcrnewbasetests:1896] INFO - Starting rebalance-in nodes:[ip:172.23.105.180 port:8091 ssh_username:root] at C2 cluster 172.23.107.120
[2017-03-05 11:05:51,106] - [xdcrnewbasetests:3327] INFO - sleep for 10 secs.  ...
[2017-03-05 11:05:51,350] - [task:414] INFO - adding node 172.23.105.181:8091 to cluster
[2017-03-05 11:05:51,351] - [rest_client:1104] INFO - adding remote node @172.23.105.181:8091 to this cluster @172.23.107.198:8091
[2017-03-05 11:05:51,851] - [task:414] INFO - adding node 172.23.105.180:8091 to cluster
[2017-03-05 11:05:51,852] - [rest_client:1104] INFO - adding remote node @172.23.105.180:8091 to this cluster @172.23.107.120:8091
[2017-03-05 11:05:54,781] - [rest_client:1322] INFO - rebalance params : password=welcome&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.181%2Cns_1%40172.23.107.171%2Cns_1%40172.23.107.198
[2017-03-05 11:05:54,796] - [rest_client:1326] INFO - rebalance operation started
[2017-03-05 11:05:54,830] - [rest_client:1457] INFO - rebalance percentage : 0.00 %
[2017-03-05 11:05:55,806] - [rest_client:1322] INFO - rebalance params : password=welcome&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.105.180%2Cns_1%40172.23.107.120
[2017-03-05 11:05:55,860] - [rest_client:1326] INFO - rebalance operation started
[2017-03-05 11:05:55,890] - [rest_client:1457] INFO - rebalance percentage : 0.00 %
[2017-03-05 11:05:58,609] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:05:58,741] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:05:58,890] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:05:59,039] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:05:59,099] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:05:59,234] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:05:59,263] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:05:59,467] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:05:59,510] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:05:59,642] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:05:59,664] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:05:59,872] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:05:59,874] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:00,030] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:00,048] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:00,238] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:00,248] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:00,418] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:00,435] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:00,648] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:00,651] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:00,842] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:00,854] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:01,065] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:01,069] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:01,237] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:01,249] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:01,270] - [rest_client:2238] INFO - Updated pauseRequested=false on bucket'default' on 172.23.107.198
[2017-03-05 11:06:01,475] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:01,486] - [xdcrnewbasetests:897] INFO - Outbound mutations on 172.23.107.198 is 3571
[2017-03-05 11:06:01,486] - [xdcrnewbasetests:898] INFO - Node 172.23.107.198 is replicating
[2017-03-05 11:06:01,502] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:01,680] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:01,731] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:01,733] - [rest_client:2238] INFO - Updated pauseRequested=false on bucket'default' on 172.23.107.120
[2017-03-05 11:06:01,912] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:01,960] - [xdcrnewbasetests:890] INFO - Outbound mutations on 172.23.107.120 is 0
[2017-03-05 11:06:02,000] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:02,066] - [xdcrnewbasetests:890] INFO - Outbound mutations on 172.23.107.120 is 0
[2017-03-05 11:06:02,111] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:02,167] - [xdcrnewbasetests:890] INFO - Outbound mutations on 172.23.107.120 is 0
[2017-03-05 11:06:02,167] - [xdcrnewbasetests:904] INFO - Outbound mutations on 172.23.107.120 is 0
[2017-03-05 11:06:02,167] - [xdcrnewbasetests:907] INFO - Cluster with node 172.23.107.120 is not replicating
[2017-03-05 11:06:02,167] - [xdcrnewbasetests:922] INFO - XDCR completed on 172.23.107.120
[2017-03-05 11:06:02,168] - [pauseResumeXDCR:115] INFO - Waiting for rebalance to complete...
[2017-03-05 11:06:02,274] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:02,288] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:02,514] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:02,530] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:02,690] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:02,738] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:02,900] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:02,999] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:03,065] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:03,230] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:03,335] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:03,458] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:03,572] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:03,700] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:03,849] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:03,865] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:04,044] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:04,107] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:04,301] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:04,351] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:04,512] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:04,609] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:04,773] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:04,838] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:04,915] - [rest_client:1457] INFO - rebalance percentage : 22.97 %
[2017-03-05 11:06:04,964] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:05,128] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:05,190] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:05,348] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:05,408] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:05,609] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:05,629] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:05,800] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:05,847] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:05,970] - [rest_client:1457] INFO - rebalance percentage : 21.47 %
[2017-03-05 11:06:06,073] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:06,076] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:06,264] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:06,308] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:06,527] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:06,586] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:06,699] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:06,832] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:06,875] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:07,073] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:07,144] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:07,274] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:07,353] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:07,500] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:07,635] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:07,683] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:07,809] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:07,905] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:08,012] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:08,131] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:08,208] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:08,354] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:08,449] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:08,554] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:08,628] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:08,766] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:08,848] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:09,004] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:09,017] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:09,244] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:09,265] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:09,432] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:09,455] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:09,628] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:09,639] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:09,791] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:09,830] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:10,018] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:10,088] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:10,257] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:10,310] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:10,492] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:10,565] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:10,694] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:10,759] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:10,936] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:11,030] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:11,089] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:11,206] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:11,313] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:11,443] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:11,498] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:11,632] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:11,735] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:11,900] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:11,910] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:12,100] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:12,104] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:12,331] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:12,348] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:12,533] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:12,551] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:12,752] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:12,796] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:12,940] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:13,039] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:13,204] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:13,258] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:13,409] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:13,520] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:13,687] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:13,749] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:13,849] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:14,008] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:14,081] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:14,195] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:14,251] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:14,426] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:14,479] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:14,598] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:14,679] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:14,830] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:14,944] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:15,019] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:15,023] - [rest_client:1457] INFO - rebalance percentage : 51.53 %
[2017-03-05 11:06:15,147] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:15,226] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:15,387] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:15,480] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:15,576] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:15,705] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:15,749] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:15,871] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:15,898] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:16,079] - [rest_client:1457] INFO - rebalance percentage : 55.62 %
[2017-03-05 11:06:16,091] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:16,161] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:16,266] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:16,370] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:16,504] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:16,600] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:16,665] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:16,770] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:16,826] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:17,002] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:17,054] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:17,193] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:17,224] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:17,439] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:17,500] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:17,640] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:17,721] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:17,905] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:18,153] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:18,284] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:18,459] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:25,101] - [rest_client:1457] INFO - rebalance percentage : 81.97 %
[2017-03-05 11:06:26,163] - [rest_client:1457] INFO - rebalance percentage : 90.38 %
[2017-03-05 11:06:31,098] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2017-03-05 11:06:31,254] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2017-03-05 11:06:31,357] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:32,810] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2017-03-05 11:06:32,927] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2017-03-05 11:06:33,021] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2017-03-05 11:06:33,574] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2017-03-05 11:06:33,760] - [data_helper:295] INFO - creating direct client 172.23.107.198:11210 default
[2017-03-05 11:06:33,965] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:35,179] - [task:537] INFO - rebalancing was completed with progress: 100% in 40.3820409775 sec
[2017-03-05 11:06:35,180] - [pauseResumeXDCR:115] INFO - Waiting for rebalance to complete...
[2017-03-05 11:06:35,855] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2017-03-05 11:06:36,111] - [data_helper:295] INFO - creating direct client 172.23.107.198:11210 default
[2017-03-05 11:06:36,275] - [task:537] INFO - rebalancing was completed with progress: 100% in 40.4143009186 sec
[2017-03-05 11:06:36,276] - [pauseResumeXDCR:121] INFO - Waiting for loading to complete...
[2017-03-05 11:06:36,403] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:39,691] - [pauseResumeXDCR:121] INFO - Waiting for loading to complete...
[2017-03-05 11:06:39,746] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.105.181', 'password': 'welcome', 'port': u'8091'}
[2017-03-05 11:06:39,747] - [data_helper:293] INFO - creating direct client 172.23.105.181:11210 default
[2017-03-05 11:06:39,819] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.105.181', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
ERROR
setting param: exp_pager_stime 10
[2017-03-05 11:06:39,821] - [xdcrnewbasetests:2473] INFO - Collecting logs @ 172.23.107.198
grabbing cbcollect from 172.23.107.198
[2017-03-05 11:06:39,825] - [remote_util:195] INFO - connecting to 172.23.107.198 with username:root password:couchbase ssh_key:
[2017-03-05 11:06:39,992] - [remote_util:229] INFO - Connected to 172.23.107.198
Collecting logs from 172.23.107.198

[2017-03-05 11:06:40,564] - [remote_util:2632] INFO - running command.raw on 172.23.107.198: /opt/couchbase/bin/cbcollect_info 172.23.107.198-20170305-1106-diag.zip
[2017-03-05 11:09:22,432] - [remote_util:2669] INFO - command executed successfully

Downloading zipped logs from 172.23.107.198
[2017-03-05 11:09:23,183] - [remote_util:2632] INFO - running command.raw on 172.23.107.198: rm -f /root/172.23.107.198-20170305-1106-diag.zip
[2017-03-05 11:09:23,230] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:09:23,236] - [remote_util:195] INFO - connecting to 172.23.107.198 with username:root password:couchbase ssh_key:
[2017-03-05 11:09:23,405] - [remote_util:229] INFO - Connected to 172.23.107.198
Collecting data files from 172.23.107.198

[2017-03-05 11:09:24,007] - [remote_util:2632] INFO - running command.raw on 172.23.107.198: tar -zcvf 172.23.107.198-352017-119-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2017-03-05 11:09:27,713] - [remote_util:2669] INFO - command executed successfully

[2017-03-05 11:09:28,324] - [remote_util:2632] INFO - running command.raw on 172.23.107.198: rm -f /root/172.23.107.198-352017-119-couch.tar.gz
[2017-03-05 11:09:28,370] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:09:28,372] - [xdcrnewbasetests:2473] INFO - Collecting logs @ 172.23.107.171
grabbing cbcollect from 172.23.107.171
[2017-03-05 11:09:28,376] - [remote_util:195] INFO - connecting to 172.23.107.171 with username:root password:couchbase ssh_key:
[2017-03-05 11:09:28,544] - [remote_util:229] INFO - Connected to 172.23.107.171
Collecting logs from 172.23.107.171

[2017-03-05 11:09:29,088] - [remote_util:2632] INFO - running command.raw on 172.23.107.171: /opt/couchbase/bin/cbcollect_info 172.23.107.171-20170305-1109-diag.zip
[2017-03-05 11:11:47,872] - [remote_util:2669] INFO - command executed successfully

Downloading zipped logs from 172.23.107.171
[2017-03-05 11:11:48,414] - [remote_util:2632] INFO - running command.raw on 172.23.107.171: rm -f /root/172.23.107.171-20170305-1109-diag.zip
[2017-03-05 11:11:48,485] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:11:48,497] - [remote_util:195] INFO - connecting to 172.23.107.171 with username:root password:couchbase ssh_key:
[2017-03-05 11:11:48,669] - [remote_util:229] INFO - Connected to 172.23.107.171
Collecting data files from 172.23.107.171

[2017-03-05 11:11:49,308] - [remote_util:2632] INFO - running command.raw on 172.23.107.171: tar -zcvf 172.23.107.171-352017-1111-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2017-03-05 11:11:50,737] - [remote_util:2669] INFO - command executed successfully

[2017-03-05 11:11:51,206] - [remote_util:2632] INFO - running command.raw on 172.23.107.171: rm -f /root/172.23.107.171-352017-1111-couch.tar.gz
[2017-03-05 11:11:51,255] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:11:51,370] - [xdcrnewbasetests:2473] INFO - Collecting logs @ 172.23.107.120
grabbing cbcollect from 172.23.107.120
[2017-03-05 11:11:51,375] - [remote_util:195] INFO - connecting to 172.23.107.120 with username:root password:couchbase ssh_key:
[2017-03-05 11:11:51,506] - [remote_util:229] INFO - Connected to 172.23.107.120
Collecting logs from 172.23.107.120

[2017-03-05 11:11:52,119] - [remote_util:2632] INFO - running command.raw on 172.23.107.120: /opt/couchbase/bin/cbcollect_info 172.23.107.120-20170305-1111-diag.zip
[2017-03-05 11:14:23,851] - [remote_util:2669] INFO - command executed successfully

Downloading zipped logs from 172.23.107.120
[2017-03-05 11:14:24,439] - [remote_util:2632] INFO - running command.raw on 172.23.107.120: rm -f /root/172.23.107.120-20170305-1111-diag.zip
[2017-03-05 11:14:24,480] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:14:24,485] - [remote_util:195] INFO - connecting to 172.23.107.120 with username:root password:couchbase ssh_key:
[2017-03-05 11:14:24,630] - [remote_util:229] INFO - Connected to 172.23.107.120
Collecting data files from 172.23.107.120

[2017-03-05 11:14:25,181] - [remote_util:2632] INFO - running command.raw on 172.23.107.120: tar -zcvf 172.23.107.120-352017-1114-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2017-03-05 11:14:27,892] - [remote_util:2669] INFO - command executed successfully

[2017-03-05 11:14:28,471] - [remote_util:2632] INFO - running command.raw on 172.23.107.120: rm -f /root/172.23.107.120-352017-1114-couch.tar.gz
[2017-03-05 11:14:28,511] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:14:28,513] - [xdcrnewbasetests:2473] INFO - Collecting logs @ 172.23.107.169
grabbing cbcollect from 172.23.107.169
[2017-03-05 11:14:28,517] - [remote_util:195] INFO - connecting to 172.23.107.169 with username:root password:couchbase ssh_key:
[2017-03-05 11:14:28,659] - [remote_util:229] INFO - Connected to 172.23.107.169
Collecting logs from 172.23.107.169

[2017-03-05 11:14:29,138] - [remote_util:2632] INFO - running command.raw on 172.23.107.169: /opt/couchbase/bin/cbcollect_info 172.23.107.169-20170305-1114-diag.zip
[2017-03-05 11:16:14,513] - [remote_util:2669] INFO - command executed successfully

Downloading zipped logs from 172.23.107.169
[2017-03-05 11:16:14,947] - [remote_util:2632] INFO - running command.raw on 172.23.107.169: rm -f /root/172.23.107.169-20170305-1114-diag.zip
[2017-03-05 11:16:14,973] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:16:15,093] - [remote_util:195] INFO - connecting to 172.23.107.169 with username:root password:couchbase ssh_key:
[2017-03-05 11:16:15,230] - [remote_util:229] INFO - Connected to 172.23.107.169
Collecting data files from 172.23.107.169

[2017-03-05 11:16:15,774] - [remote_util:2632] INFO - running command.raw on 172.23.107.169: tar -zcvf 172.23.107.169-352017-1116-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2017-03-05 11:16:17,680] - [remote_util:2669] INFO - command executed successfully

[2017-03-05 11:16:18,183] - [remote_util:2632] INFO - running command.raw on 172.23.107.169: rm -f /root/172.23.107.169-352017-1116-couch.tar.gz
[2017-03-05 11:16:18,217] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:16:18,219] - [xdcrnewbasetests:2473] INFO - Collecting logs @ 172.23.105.178
grabbing cbcollect from 172.23.105.178
[2017-03-05 11:16:18,223] - [remote_util:195] INFO - connecting to 172.23.105.178 with username:root password:couchbase ssh_key:
[2017-03-05 11:16:18,372] - [remote_util:229] INFO - Connected to 172.23.105.178
Collecting logs from 172.23.105.178

[2017-03-05 11:16:18,871] - [remote_util:2632] INFO - running command.raw on 172.23.105.178: /opt/couchbase/bin/cbcollect_info 172.23.105.178-20170305-1116-diag.zip
[2017-03-05 11:17:44,112] - [remote_util:2669] INFO - command executed successfully

Downloading zipped logs from 172.23.105.178
[2017-03-05 11:17:44,529] - [remote_util:2632] INFO - running command.raw on 172.23.105.178: rm -f /root/172.23.105.178-20170305-1116-diag.zip
[2017-03-05 11:17:44,559] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:17:44,565] - [remote_util:195] INFO - connecting to 172.23.105.178 with username:root password:couchbase ssh_key:
[2017-03-05 11:17:44,705] - [remote_util:229] INFO - Connected to 172.23.105.178
Collecting data files from 172.23.105.178

[2017-03-05 11:17:45,187] - [remote_util:2632] INFO - running command.raw on 172.23.105.178: tar -zcvf 172.23.105.178-352017-1117-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2017-03-05 11:17:45,243] - [remote_util:2669] INFO - command executed successfully

[2017-03-05 11:17:45,406] - [remote_util:2632] INFO - running command.raw on 172.23.105.178: rm -f /root/172.23.105.178-352017-1117-couch.tar.gz
[2017-03-05 11:17:45,429] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:17:45,431] - [xdcrnewbasetests:2473] INFO - Collecting logs @ 172.23.105.179
grabbing cbcollect from 172.23.105.179
[2017-03-05 11:17:45,435] - [remote_util:195] INFO - connecting to 172.23.105.179 with username:root password:couchbase ssh_key:
[2017-03-05 11:17:45,576] - [remote_util:229] INFO - Connected to 172.23.105.179
Collecting logs from 172.23.105.179

[2017-03-05 11:17:46,104] - [remote_util:2632] INFO - running command.raw on 172.23.105.179: /opt/couchbase/bin/cbcollect_info 172.23.105.179-20170305-1117-diag.zip
[2017-03-05 11:18:54,919] - [remote_util:2669] INFO - command executed successfully

Downloading zipped logs from 172.23.105.179
[2017-03-05 11:18:55,150] - [remote_util:2632] INFO - running command.raw on 172.23.105.179: rm -f /root/172.23.105.179-20170305-1117-diag.zip
[2017-03-05 11:18:55,179] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:18:55,200] - [remote_util:195] INFO - connecting to 172.23.105.179 with username:root password:couchbase ssh_key:
[2017-03-05 11:18:55,339] - [remote_util:229] INFO - Connected to 172.23.105.179
Collecting data files from 172.23.105.179

[2017-03-05 11:18:55,826] - [remote_util:2632] INFO - running command.raw on 172.23.105.179: tar -zcvf 172.23.105.179-352017-1118-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2017-03-05 11:18:55,880] - [remote_util:2669] INFO - command executed successfully

[2017-03-05 11:18:56,036] - [remote_util:2632] INFO - running command.raw on 172.23.105.179: rm -f /root/172.23.105.179-352017-1118-couch.tar.gz
[2017-03-05 11:18:56,065] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:18:56,066] - [xdcrnewbasetests:2473] INFO - Collecting logs @ 172.23.105.180
grabbing cbcollect from 172.23.105.180
[2017-03-05 11:18:56,070] - [remote_util:195] INFO - connecting to 172.23.105.180 with username:root password:couchbase ssh_key:
[2017-03-05 11:18:56,207] - [remote_util:229] INFO - Connected to 172.23.105.180
Collecting logs from 172.23.105.180

[2017-03-05 11:18:56,707] - [remote_util:2632] INFO - running command.raw on 172.23.105.180: /opt/couchbase/bin/cbcollect_info 172.23.105.180-20170305-1118-diag.zip
[2017-03-05 11:20:46,927] - [remote_util:2669] INFO - command executed successfully

Downloading zipped logs from 172.23.105.180
[2017-03-05 11:20:47,379] - [remote_util:2632] INFO - running command.raw on 172.23.105.180: rm -f /root/172.23.105.180-20170305-1118-diag.zip
[2017-03-05 11:20:47,418] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:20:47,423] - [remote_util:195] INFO - connecting to 172.23.105.180 with username:root password:couchbase ssh_key:
[2017-03-05 11:20:47,555] - [remote_util:229] INFO - Connected to 172.23.105.180
Collecting data files from 172.23.105.180

[2017-03-05 11:20:48,113] - [remote_util:2632] INFO - running command.raw on 172.23.105.180: tar -zcvf 172.23.105.180-352017-1120-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2017-03-05 11:20:49,509] - [remote_util:2669] INFO - command executed successfully

[2017-03-05 11:20:49,991] - [remote_util:2632] INFO - running command.raw on 172.23.105.180: rm -f /root/172.23.105.180-352017-1120-couch.tar.gz
[2017-03-05 11:20:50,034] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:20:50,036] - [xdcrnewbasetests:2473] INFO - Collecting logs @ 172.23.105.181
grabbing cbcollect from 172.23.105.181
[2017-03-05 11:20:50,040] - [remote_util:195] INFO - connecting to 172.23.105.181 with username:root password:couchbase ssh_key:
[2017-03-05 11:20:50,209] - [remote_util:229] INFO - Connected to 172.23.105.181
Collecting logs from 172.23.105.181

[2017-03-05 11:20:50,777] - [remote_util:2632] INFO - running command.raw on 172.23.105.181: /opt/couchbase/bin/cbcollect_info 172.23.105.181-20170305-1120-diag.zip
[2017-03-05 11:22:42,749] - [remote_util:2669] INFO - command executed successfully

Downloading zipped logs from 172.23.105.181
[2017-03-05 11:22:43,174] - [remote_util:2632] INFO - running command.raw on 172.23.105.181: rm -f /root/172.23.105.181-20170305-1120-diag.zip
[2017-03-05 11:22:43,221] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:22:43,233] - [remote_util:195] INFO - connecting to 172.23.105.181 with username:root password:couchbase ssh_key:
[2017-03-05 11:22:43,367] - [remote_util:229] INFO - Connected to 172.23.105.181
Collecting data files from 172.23.105.181

[2017-03-05 11:22:43,909] - [remote_util:2632] INFO - running command.raw on 172.23.105.181: tar -zcvf 172.23.105.181-352017-1122-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1
[2017-03-05 11:22:45,076] - [remote_util:2669] INFO - command executed successfully

[2017-03-05 11:22:45,528] - [remote_util:2632] INFO - running command.raw on 172.23.105.181: rm -f /root/172.23.105.181-352017-1122-couch.tar.gz
[2017-03-05 11:22:45,572] - [remote_util:2669] INFO - command executed successfully
[2017-03-05 11:22:45,574] - [xdcrnewbasetests:2482] INFO - ====  XDCRNewbasetests cleanup is started for test #6 replication_with_pause_and_resume ====
[2017-03-05 11:22:45,574] - [xdcrnewbasetests:1149] INFO - removing xdcr/nodes settings
[2017-03-05 11:22:45,599] - [rest_client:1077] INFO - Deleting replication /controller/cancelXDCR/99d1ac9e65b72b50d2012b604c5cabf0%2Fdefault%2Fdefault
[2017-03-05 11:22:45,651] - [rest_client:1025] INFO - removing remote cluster name:remote_cluster_C1-C2
[2017-03-05 11:22:45,691] - [xdcrnewbasetests:1159] INFO - cleanup [ip:172.23.107.198 port:8091 ssh_username:root, ip:172.23.107.171 port:8091 ssh_username:root, ip:172.23.105.181 port:8091 ssh_username:root]
[2017-03-05 11:22:45,726] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 172.23.107.198
[2017-03-05 11:22:45,726] - [bucket_helper:140] INFO - remove bucket default ...
[2017-03-05 11:22:47,502] - [bucket_helper:154] INFO - deleted bucket : default from 172.23.107.198
[2017-03-05 11:22:47,502] - [bucket_helper:230] INFO - waiting for bucket deletion to complete....
[2017-03-05 11:22:47,507] - [rest_client:134] INFO - node 172.23.107.198 existing buckets : []
[2017-03-05 11:22:47,539] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes
[2017-03-05 11:22:47,544] - [rest_client:1322] INFO - rebalance params : password=welcome&ejectedNodes=ns_1%40172.23.105.181%2Cns_1%40172.23.107.171&user=Administrator&knownNodes=ns_1%40172.23.105.181%2Cns_1%40172.23.107.171%2Cns_1%40172.23.107.198
[2017-03-05 11:22:47,558] - [rest_client:1326] INFO - rebalance operation started
[2017-03-05 11:22:47,574] - [rest_client:1457] INFO - rebalance percentage : 0.00 %
[2017-03-05 11:23:07,604] - [rest_client:1412] INFO - rebalance progress took 20.045593977 seconds 
[2017-03-05 11:23:07,605] - [rest_client:1413] INFO - sleep for 10 seconds after rebalance...
[2017-03-05 11:23:17,640] - [cluster_helper:295] INFO - removed all the nodes from cluster associated with ip:172.23.107.198 port:8091 ssh_username:root ? [(u'ns_1@172.23.105.181', 8091), (u'ns_1@172.23.107.171', 8091)]
[2017-03-05 11:23:17,648] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.198:8091
[2017-03-05 11:23:17,656] - [cluster_helper:80] INFO - ns_server @ 172.23.107.198:8091 is running
[2017-03-05 11:23:17,667] - [bucket_helper:138] INFO - deleting existing buckets [] on 172.23.107.171
[2017-03-05 11:23:17,704] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.171:8091
[2017-03-05 11:23:17,709] - [cluster_helper:80] INFO - ns_server @ 172.23.107.171:8091 is running
[2017-03-05 11:23:17,721] - [bucket_helper:138] INFO - deleting existing buckets [] on 172.23.105.181
[2017-03-05 11:23:17,744] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.181:8091
[2017-03-05 11:23:17,750] - [cluster_helper:80] INFO - ns_server @ 172.23.105.181:8091 is running
Cluster instance shutdown with force
[2017-03-05 11:23:17,750] - [xdcrnewbasetests:1149] INFO - removing xdcr/nodes settings
[2017-03-05 11:23:17,813] - [rest_client:1025] INFO - removing remote cluster name:remote_cluster_C2-C1
[2017-03-05 11:23:17,848] - [xdcrnewbasetests:1159] INFO - cleanup [ip:172.23.107.120 port:8091 ssh_username:root, ip:172.23.107.169 port:8091 ssh_username:root, ip:172.23.105.180 port:8091 ssh_username:root]
[2017-03-05 11:23:17,886] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 172.23.107.120
[2017-03-05 11:23:17,886] - [bucket_helper:140] INFO - remove bucket default ...
[2017-03-05 11:23:47,906] - [rest_client:790] ERROR - http://172.23.107.120:8091/pools/default/buckets/default error 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"}
[2017-03-05 11:23:47,906] - [rest_client:1898] WARNING - Bucket deletion timed out waiting for all nodes
[2017-03-05 11:23:47,931] - [bucket_helper:153] ERROR - Unable to get timings for bucket
[2017-03-05 11:23:47,931] - [bucket_helper:154] INFO - deleted bucket : default from 172.23.107.120
[2017-03-05 11:23:47,931] - [bucket_helper:230] INFO - waiting for bucket deletion to complete....
[2017-03-05 11:23:47,955] - [rest_client:134] INFO - node 172.23.107.120 existing buckets : []
[2017-03-05 11:23:48,005] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes
[2017-03-05 11:23:48,010] - [rest_client:1322] INFO - rebalance params : password=welcome&ejectedNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.105.180&user=Administrator&knownNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.105.180%2Cns_1%40172.23.107.120
[2017-03-05 11:23:48,017] - [rest_client:1326] INFO - rebalance operation started
[2017-03-05 11:23:48,047] - [rest_client:1457] INFO - rebalance percentage : 0.00 %
[2017-03-05 11:23:58,064] - [rest_client:1457] INFO - rebalance percentage : 0.00 %
[2017-03-05 11:24:08,083] - [rest_client:1440] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
[2017-03-05 11:24:08,208] - [rest_client:2726] INFO - Latest logs from UI on 172.23.107.120:
[2017-03-05 11:24:08,208] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.120', u'code': 0, u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n                              [{\'ns_1@172.23.107.120\',\n                                {\'EXIT\',\n                                 {old_buckets_shutdown_wait_failed,\n                                  ["default"]}}}]}', u'shortText': u'message', u'serverTime': u'2017-03-05T11:24:08.052Z', u'module': u'ns_orchestrator', u'tstamp': 1488741848052, u'type': u'critical'}
[2017-03-05 11:24:08,208] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.120', u'code': 0, u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.107.120\',\n                                                         {\'EXIT\',\n                                                          {old_buckets_shutdown_wait_failed,\n                                                           ["default"]}}}]\n', u'shortText': u'message', u'serverTime': u'2017-03-05T11:24:08.052Z', u'module': u'ns_rebalancer', u'tstamp': 1488741848052, u'type': u'critical'}
[2017-03-05 11:24:08,209] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.120', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.107.120'], EjectNodes = ['ns_1@172.23.107.169',\n                                                                       'ns_1@172.23.105.180'], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2017-03-05T11:23:48.005Z', u'module': u'ns_orchestrator', u'tstamp': 1488741828005, u'type': u'info'}
[2017-03-05 11:24:08,209] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.120', u'code': 102, u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.107.120\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.107.169\', u\'ns_1@172.23.105.180\']\n', u'shortText': u'client-side error report', u'serverTime': u'2017-03-05T11:23:47.999Z', u'module': u'menelaus_web', u'tstamp': 1488741827999, u'type': u'warning'}
[2017-03-05 11:24:08,209] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.120', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.107.120\' for deletion', u'shortText': u'message', u'serverTime': u'2017-03-05T11:23:18.468Z', u'module': u'ns_memcached', u'tstamp': 1488741798468, u'type': u'info'}
[2017-03-05 11:24:08,209] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.169', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.107.169\' for deletion', u'shortText': u'message', u'serverTime': u'2017-03-05T11:23:18.054Z', u'module': u'ns_memcached', u'tstamp': 1488741798054, u'type': u'info'}
[2017-03-05 11:24:08,210] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.120', u'code': 0, u'text': u'Remote cluster reference "remote_cluster_C2-C1" known via 172.23.107.198:8091 removed.', u'shortText': u'message', u'serverTime': u'2017-03-05T11:23:17.817Z', u'module': u'xdcr', u'tstamp': 1488741797817, u'type': u'info'}
[2017-03-05 11:24:08,210] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.169', u'code': 0, u'text': u'Replication from bucket "default" to bucket "default" on cluster "remote_cluster_C2-C1" removed, since the target bucket "default" has been deleted', u'shortText': u'message', u'serverTime': u'2017-03-05T11:22:46.103Z', u'module': u'xdcr', u'tstamp': 1488741766103, u'type': u'info'}
[2017-03-05 11:24:08,210] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.105.180', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.105.180\' for deletion', u'shortText': u'message', u'serverTime': u'2017-03-05T11:19:53.376Z', u'module': u'ns_memcached', u'tstamp': 1488741593376, u'type': u'info'}
[2017-03-05 11:24:08,210] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.120', u'code': 0, u'text': u'Reset auto-failover count', u'shortText': u'message', u'serverTime': u'2017-03-05T11:06:29.336Z', u'module': u'auto_failover', u'tstamp': 1488740789336, u'type': u'info'}
ERROR

======================================================================
ERROR: replication_with_pause_and_resume (xdcr.pauseResumeXDCR.PauseResumeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/xdcr/pauseResumeXDCR.py", line 126, in replication_with_pause_and_resume
    self._wait_for_replication_to_catchup(timeout=500)
  File "pytests/xdcr/xdcrnewbasetests.py", line 3273, in _wait_for_replication_to_catchup
    cb_cluster.run_expiry_pager()
  File "pytests/xdcr/xdcrnewbasetests.py", line 1673, in run_expiry_pager
    bucket)
  File "lib/membase/helper/cluster_helper.py", line 333, in flushctl_set
    ClusterOperationHelper.flushctl_set_per_node(_server, key, val, bucket)
  File "lib/membase/helper/cluster_helper.py", line 353, in flushctl_set_per_node
    rv = mc.set_param(key, str(val), type)
  File "lib/mc_bin_client.py", line 541, in set_param
    return self._doCmd(memcacheConstants.CMD_SET_FLUSH_PARAM, key, val, type)
  File "lib/mc_bin_client.py", line 139, in _doCmd
    return self._handleSingleResponse(opaque)
  File "lib/mc_bin_client.py", line 131, in _handleSingleResponse
    cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque)
  File "lib/mc_bin_client.py", line 122, in _handleKeyedResponse
    cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg()
  File "lib/mc_bin_client.py", line 98, in _recvMsg
    raise exceptions.EOFError("Got empty data (remote died?). from {0}".format(self.host))
EOFError: Got empty data (remote died?). from 172.23.105.181

======================================================================
ERROR: replication_with_pause_and_resume (xdcr.pauseResumeXDCR.PauseResumeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/xdcr/pauseResumeXDCR.py", line 26, in tearDown
    super(PauseResumeTest, self).tearDown()
  File "pytests/xdcr/xdcrnewbasetests.py", line 2484, in tearDown
    cb_cluster.cleanup_cluster(self)
  File "pytests/xdcr/xdcrnewbasetests.py", line 1174, in cleanup_cluster
    ClusterOperationHelper.cleanup_cluster([node])
  File "lib/membase/helper/cluster_helper.py", line 260, in cleanup_cluster
    wait_for_rebalance=wait_for_rebalance)
  File "lib/membase/api/rest_client.py", line 113, in remove_nodes
    return self.rest.monitorRebalance()
  File "lib/membase/api/rest_client.py", line 1382, in monitorRebalance
    progress = self._rebalance_progress()
  File "lib/membase/api/rest_client.py", line 1442, in _rebalance_progress
    raise RebalanceFailedException(msg)
RebalanceFailedException: Rebalance Failed: {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed

----------------------------------------------------------------------
Ran 1 test in 1182.096s

FAILED (errors=2)
Cluster instance shutdown with force
Cluster instance shutdown with force
grabbing core dumps files from 172.23.107.198
[2017-03-05 11:24:08,253] - [remote_util:195] INFO - connecting to 172.23.107.198 with username:root password:couchbase ssh_key:
[2017-03-05 11:24:08,420] - [remote_util:229] INFO - Connected to 172.23.107.198
[2017-03-05 11:24:09,021] - [remote_util:564] INFO - 172.23.107.198 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for crashes on 172.23.107.198 ... 
erl_crash files under /opt/couchbase/var/lib/couchbase/
core* files under /opt/couchbase/var/lib/couchbase/
core* files under /tmp/
breakpad *dmp files under /opt/couchbase/var/lib/couchbase/
crashes not found on 172.23.107.198
grabbing core dumps files from 172.23.107.171
[2017-03-05 11:24:09,189] - [remote_util:195] INFO - connecting to 172.23.107.171 with username:root password:couchbase ssh_key:
[2017-03-05 11:24:09,331] - [remote_util:229] INFO - Connected to 172.23.107.171
[2017-03-05 11:24:09,963] - [remote_util:564] INFO - 172.23.107.171 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for crashes on 172.23.107.171 ... 
erl_crash files under /opt/couchbase/var/lib/couchbase/
core* files under /opt/couchbase/var/lib/couchbase/
core* files under /tmp/
breakpad *dmp files under /opt/couchbase/var/lib/couchbase/
crashes not found on 172.23.107.171
grabbing core dumps files from 172.23.107.120
[2017-03-05 11:24:10,127] - [remote_util:195] INFO - connecting to 172.23.107.120 with username:root password:couchbase ssh_key:
[2017-03-05 11:24:10,258] - [remote_util:229] INFO - Connected to 172.23.107.120
[2017-03-05 11:24:10,900] - [remote_util:564] INFO - 172.23.107.120 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for crashes on 172.23.107.120 ... 
erl_crash files under /opt/couchbase/var/lib/couchbase/
core* files under /opt/couchbase/var/lib/couchbase/
core* files under /tmp/
breakpad *dmp files under /opt/couchbase/var/lib/couchbase/
crashes not found on 172.23.107.120
grabbing core dumps files from 172.23.107.169
[2017-03-05 11:24:11,072] - [remote_util:195] INFO - connecting to 172.23.107.169 with username:root password:couchbase ssh_key:
[2017-03-05 11:24:11,241] - [remote_util:229] INFO - Connected to 172.23.107.169
[2017-03-05 11:24:11,854] - [remote_util:564] INFO - 172.23.107.169 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for crashes on 172.23.107.169 ... 
erl_crash files under /opt/couchbase/var/lib/couchbase/
core* files under /opt/couchbase/var/lib/couchbase/
core* files under /tmp/
breakpad *dmp files under /opt/couchbase/var/lib/couchbase/
crashes not found on 172.23.107.169
grabbing core dumps files from 172.23.105.178
[2017-03-05 11:24:12,020] - [remote_util:195] INFO - connecting to 172.23.105.178 with username:root password:couchbase ssh_key:
[2017-03-05 11:24:12,193] - [remote_util:229] INFO - Connected to 172.23.105.178
[2017-03-05 11:24:12,849] - [remote_util:564] INFO - 172.23.105.178 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for crashes on 172.23.105.178 ... 
erl_crash files under /opt/couchbase/var/lib/couchbase/
core* files under /opt/couchbase/var/lib/couchbase/
core* files under /tmp/
breakpad *dmp files under /opt/couchbase/var/lib/couchbase/
crashes not found on 172.23.105.178
grabbing core dumps files from 172.23.105.179
[2017-03-05 11:24:13,047] - [remote_util:195] INFO - connecting to 172.23.105.179 with username:root password:couchbase ssh_key:
[2017-03-05 11:24:13,185] - [remote_util:229] INFO - Connected to 172.23.105.179
[2017-03-05 11:24:13,764] - [remote_util:564] INFO - 172.23.105.179 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for crashes on 172.23.105.179 ... 
erl_crash files under /opt/couchbase/var/lib/couchbase/
core* files under /opt/couchbase/var/lib/couchbase/
core* files under /tmp/
breakpad *dmp files under /opt/couchbase/var/lib/couchbase/
crashes not found on 172.23.105.179
grabbing core dumps files from 172.23.105.180
[2017-03-05 11:24:13,893] - [remote_util:195] INFO - connecting to 172.23.105.180 with username:root password:couchbase ssh_key:
[2017-03-05 11:24:14,036] - [remote_util:229] INFO - Connected to 172.23.105.180
[2017-03-05 11:24:14,685] - [remote_util:564] INFO - 172.23.105.180 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for crashes on 172.23.105.180 ... 
erl_crash files under /opt/couchbase/var/lib/couchbase/
core* files under /opt/couchbase/var/lib/couchbase/
core* files under /tmp/
breakpad *dmp files under /opt/couchbase/var/lib/couchbase/
crashes not found on 172.23.105.180
grabbing core dumps files from 172.23.105.181
[2017-03-05 11:24:14,820] - [remote_util:195] INFO - connecting to 172.23.105.181 with username:root password:couchbase ssh_key:
[2017-03-05 11:24:14,951] - [remote_util:229] INFO - Connected to 172.23.105.181
[2017-03-05 11:24:15,687] - [remote_util:564] INFO - 172.23.105.181 **** The version file /opt/couchbase/ VERSION.txt  exists
looking for crashes on 172.23.105.181 ... 
erl_crash files under /opt/couchbase/var/lib/couchbase/
core* files under /opt/couchbase/var/lib/couchbase/
core* files under /tmp/
breakpad *dmp files under /opt/couchbase/var/lib/couchbase/
crashes not found on 172.23.105.181
grabbing diags from 
http://172.23.107.198:8091/diag
unable to obtain diags from http://172.23.107.198:8091/diag Invalid header value 'Basic QWRtaW5pc3RyYXRvcjp3ZWxjb21l\n'
grabbing diags from 
http://172.23.107.171:8091/diag
unable to obtain diags from http://172.23.107.171:8091/diag Invalid header value 'Basic QWRtaW5pc3RyYXRvcjp3ZWxjb21l\n'
grabbing diags from 
http://172.23.107.120:8091/diag
unable to obtain diags from http://172.23.107.120:8091/diag Invalid header value 'Basic QWRtaW5pc3RyYXRvcjp3ZWxjb21l\n'
grabbing diags from 
http://172.23.107.169:8091/diag
unable to obtain diags from http://172.23.107.169:8091/diag Invalid header value 'Basic QWRtaW5pc3RyYXRvcjp3ZWxjb21l\n'
grabbing diags from 
http://172.23.105.178:8091/diag
unable to obtain diags from http://172.23.105.178:8091/diag Invalid header value 'Basic QWRtaW5pc3RyYXRvcjp3ZWxjb21l\n'
grabbing diags from 
http://172.23.105.179:8091/diag
unable to obtain diags from http://172.23.105.179:8091/diag Invalid header value 'Basic QWRtaW5pc3RyYXRvcjp3ZWxjb21l\n'
grabbing diags from 
http://172.23.105.180:8091/diag
unable to obtain diags from http://172.23.105.180:8091/diag Invalid header value 'Basic QWRtaW5pc3RyYXRvcjp3ZWxjb21l\n'
grabbing diags from 
http://172.23.105.181:8091/diag
unable to obtain diags from http://172.23.105.181:8091/diag Invalid header value 'Basic QWRtaW5pc3RyYXRvcjp3ZWxjb21l\n'
summary so far suite xdcr.pauseResumeXDCR.PauseResumeTest , pass 0 , fail 6
failures so far...
xdcr.pauseResumeXDCR.PauseResumeTest.replication_with_pause_and_resume
xdcr.pauseResumeXDCR.PauseResumeTest.replication_with_pause_and_resume
xdcr.pauseResumeXDCR.PauseResumeTest.replication_with_pause_and_resume
xdcr.pauseResumeXDCR.PauseResumeTest.replication_with_pause_and_resume
xdcr.pauseResumeXDCR.PauseResumeTest.replication_with_pause_and_resume
xdcr.pauseResumeXDCR.PauseResumeTest.replication_with_pause_and_resume
testrunner logs, diags and results are available under /data/workspace/cen006-p1-xxdcr-vset09-01-goxdcr-pause-resume/logs/testrunner-17-Mar-05_09-36-16/test_6