./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,fail_on_errors=1,GROUP=P1 -t xdcr.pauseResumeXDCR.PauseResumeTest.replication_with_pause_and_resume,items=30000,rdirection=bidirection,ctopology=chain,replication_type=xmem,rebalance_in=C1-C2,pause_wait=10,pause=C1-C2,GROUP=P1

Test Input params:
{'get-coredumps': 'True', 'fail_on_errors': '1', 'pause': 'C1-C2', 'GROUP': 'P1', 'cluster_name': 'INI_FILE', 'items': '30000', 'case_number': 6, 'conf_file': 'conf/py-xdcr-pause-resume.conf', 'num_nodes': 8, 'pause_wait': '10', 'ctopology': 'chain', 'rdirection': 'bidirection', 'stop-on-failure': 'False', 'ini': 'INI_FILE.ini', 'replication_type': 'xmem', 'get-logs': 'False', 'rebalance_in': 'C1-C2', 'get-cbcollect-info': 'True', 'logs_folder': '/data/workspace/cen006-p1-xxdcr-vset09-01-goxdcr-pause-resume/logs/testrunner-16-Feb-11_21-06-09/test_6', 'spec': 'py-xdcr-pause-resume'}
[2016-02-11 21:45:54,411] - [remote_util:165] INFO - connecting to 172.23.107.118 with username : root password : couchbase ssh_key: 
[2016-02-11 21:45:54,631] - [remote_util:198] INFO - Connected to 172.23.107.118
looking for Erlang/Memcached crashes on 172.23.107.118 ... 
dump files not found on 172.23.107.118
[2016-02-11 21:45:55,690] - [remote_util:165] INFO - connecting to 172.23.107.171 with username : root password : couchbase ssh_key: 
[2016-02-11 21:45:55,905] - [remote_util:198] INFO - Connected to 172.23.107.171
looking for Erlang/Memcached crashes on 172.23.107.171 ... 
dump files not found on 172.23.107.171
[2016-02-11 21:45:56,727] - [remote_util:165] INFO - connecting to 172.23.107.120 with username : root password : couchbase ssh_key: 
[2016-02-11 21:45:56,950] - [remote_util:198] INFO - Connected to 172.23.107.120
looking for Erlang/Memcached crashes on 172.23.107.120 ... 
dump files not found on 172.23.107.120
[2016-02-11 21:45:57,842] - [remote_util:165] INFO - connecting to 172.23.107.169 with username : root password : couchbase ssh_key: 
[2016-02-11 21:45:58,071] - [remote_util:198] INFO - Connected to 172.23.107.169
looking for Erlang/Memcached crashes on 172.23.107.169 ... 
dump files not found on 172.23.107.169
[2016-02-11 21:45:58,872] - [remote_util:165] INFO - connecting to 172.23.105.178 with username : root password : couchbase ssh_key: 
[2016-02-11 21:45:59,076] - [remote_util:198] INFO - Connected to 172.23.105.178
looking for Erlang/Memcached crashes on 172.23.105.178 ... 
dump files not found on 172.23.105.178
[2016-02-11 21:45:59,773] - [remote_util:165] INFO - connecting to 172.23.105.179 with username : root password : couchbase ssh_key: 
[2016-02-11 21:45:59,991] - [remote_util:198] INFO - Connected to 172.23.105.179
looking for Erlang/Memcached crashes on 172.23.105.179 ... 
dump files not found on 172.23.105.179
[2016-02-11 21:46:00,722] - [remote_util:165] INFO - connecting to 172.23.105.180 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:00,928] - [remote_util:198] INFO - Connected to 172.23.105.180
looking for Erlang/Memcached crashes on 172.23.105.180 ... 
dump files not found on 172.23.105.180
[2016-02-11 21:46:01,642] - [remote_util:165] INFO - connecting to 172.23.105.181 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:01,848] - [remote_util:198] INFO - Connected to 172.23.105.181
replication_with_pause_and_resume (xdcr.pauseResumeXDCR.PauseResumeTest) ... looking for Erlang/Memcached crashes on 172.23.105.181 ... 
dump files not found on 172.23.105.181
[2016-02-11 21:46:02,611] - [remote_util:165] INFO - connecting to 172.23.107.118 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:02,839] - [remote_util:198] INFO - Connected to 172.23.107.118
[2016-02-11 21:46:02,852] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.118:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:03,559] - [remote_util:2222] INFO - running command.raw on 172.23.107.118: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:03,667] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:03,668] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:03,670] - [remote_util:165] INFO - connecting to 172.23.107.118 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:03,885] - [remote_util:198] INFO - Connected to 172.23.107.118
[2016-02-11 21:46:03,898] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.118:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:04,634] - [remote_util:2222] INFO - running command.raw on 172.23.107.118: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:04,741] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:04,741] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:04,856] - [remote_util:165] INFO - connecting to 172.23.107.171 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:05,068] - [remote_util:198] INFO - Connected to 172.23.107.171
[2016-02-11 21:46:05,081] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.171:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:05,705] - [remote_util:2222] INFO - running command.raw on 172.23.107.171: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:05,821] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:05,822] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:05,937] - [remote_util:165] INFO - connecting to 172.23.107.171 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:06,161] - [remote_util:198] INFO - Connected to 172.23.107.171
[2016-02-11 21:46:06,178] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.171:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:06,823] - [remote_util:2222] INFO - running command.raw on 172.23.107.171: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:06,949] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:06,950] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:06,951] - [remote_util:165] INFO - connecting to 172.23.107.120 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:07,188] - [remote_util:198] INFO - Connected to 172.23.107.120
[2016-02-11 21:46:07,201] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.120:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:07,831] - [remote_util:2222] INFO - running command.raw on 172.23.107.120: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:07,946] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:07,947] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:08,061] - [remote_util:165] INFO - connecting to 172.23.107.120 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:08,279] - [remote_util:198] INFO - Connected to 172.23.107.120
[2016-02-11 21:46:08,294] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.120:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:08,956] - [remote_util:2222] INFO - running command.raw on 172.23.107.120: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:09,005] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:09,006] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:09,121] - [remote_util:165] INFO - connecting to 172.23.107.169 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:09,325] - [remote_util:198] INFO - Connected to 172.23.107.169
[2016-02-11 21:46:09,335] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.169:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:09,857] - [remote_util:2222] INFO - running command.raw on 172.23.107.169: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:09,955] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:09,957] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:09,959] - [remote_util:165] INFO - connecting to 172.23.107.169 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:10,171] - [remote_util:198] INFO - Connected to 172.23.107.169
[2016-02-11 21:46:10,183] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.169:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:10,726] - [remote_util:2222] INFO - running command.raw on 172.23.107.169: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:10,823] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:10,823] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:10,938] - [remote_util:165] INFO - connecting to 172.23.105.178 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:11,144] - [remote_util:198] INFO - Connected to 172.23.105.178
[2016-02-11 21:46:11,151] - [rest_client:757] ERROR - http://172.23.105.178:8091/nodes/self error 401 reason: status: 401, content:  
http://172.23.105.178:8091/nodes/self: 
[2016-02-11 21:46:11,157] - [rest_client:757] ERROR - http://172.23.105.178:8091/diag/eval/ error 401 reason: status: 401, content:  
[2016-02-11 21:46:11,157] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.178:8091: False content:  command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:11,668] - [remote_util:2222] INFO - running command.raw on 172.23.105.178: zgrep "panic:" /goxdcr.log* | wc -l
[2016-02-11 21:46:11,764] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:11,765] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:11,880] - [remote_util:165] INFO - connecting to 172.23.105.178 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:12,091] - [remote_util:198] INFO - Connected to 172.23.105.178
[2016-02-11 21:46:12,096] - [rest_client:757] ERROR - http://172.23.105.178:8091/nodes/self error 401 reason: status: 401, content:  
http://172.23.105.178:8091/nodes/self: 
[2016-02-11 21:46:12,100] - [rest_client:757] ERROR - http://172.23.105.178:8091/diag/eval/ error 401 reason: status: 401, content:  
[2016-02-11 21:46:12,101] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.178:8091: False content:  command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:12,634] - [remote_util:2222] INFO - running command.raw on 172.23.105.178: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /goxdcr.log* | wc -l
[2016-02-11 21:46:12,728] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:12,729] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:12,843] - [remote_util:165] INFO - connecting to 172.23.105.179 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:13,052] - [remote_util:198] INFO - Connected to 172.23.105.179
[2016-02-11 21:46:13,068] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.179:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:13,668] - [remote_util:2222] INFO - running command.raw on 172.23.105.179: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:13,763] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:13,764] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:13,766] - [remote_util:165] INFO - connecting to 172.23.105.179 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:13,975] - [remote_util:198] INFO - Connected to 172.23.105.179
[2016-02-11 21:46:13,988] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.179:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:14,493] - [remote_util:2222] INFO - running command.raw on 172.23.105.179: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:14,587] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:14,589] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:14,704] - [remote_util:165] INFO - connecting to 172.23.105.180 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:14,913] - [remote_util:198] INFO - Connected to 172.23.105.180
[2016-02-11 21:46:14,925] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.180:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:15,432] - [remote_util:2222] INFO - running command.raw on 172.23.105.180: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:15,527] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:15,528] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:15,643] - [remote_util:165] INFO - connecting to 172.23.105.180 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:15,848] - [remote_util:198] INFO - Connected to 172.23.105.180
[2016-02-11 21:46:15,861] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.180:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:16,347] - [remote_util:2222] INFO - running command.raw on 172.23.105.180: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:16,442] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:16,443] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:16,558] - [remote_util:165] INFO - connecting to 172.23.105.181 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:16,773] - [remote_util:198] INFO - Connected to 172.23.105.181
[2016-02-11 21:46:16,784] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.181:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:17,263] - [remote_util:2222] INFO - running command.raw on 172.23.105.181: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:17,361] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:17,362] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:17,477] - [remote_util:165] INFO - connecting to 172.23.105.181 with username : root password : couchbase ssh_key: 
[2016-02-11 21:46:17,688] - [remote_util:198] INFO - Connected to 172.23.105.181
[2016-02-11 21:46:17,699] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.181:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:46:18,240] - [remote_util:2222] INFO - running command.raw on 172.23.105.181: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:46:18,343] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:46:18,344] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:46:18,459] - [xdcrnewbasetests:2457] INFO - {'172.23.107.169': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.118': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.171': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.120': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.180': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.181': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.179': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.178': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}}
[2016-02-11 21:46:18,460] - [xdcrnewbasetests:2281] INFO - ==== XDCRNewbasetests setup is started for test #6 replication_with_pause_and_resume ====
[2016-02-11 21:46:18,462] - [xdcrnewbasetests:1090] INFO - removing xdcr/nodes settings
[2016-02-11 21:46:18,504] - [xdcrnewbasetests:1100] INFO - cleanup [ip:172.23.107.118 port:8091 ssh_username:root, ip:172.23.107.171 port:8091 ssh_username:root]
[2016-02-11 21:46:18,520] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.118
[2016-02-11 21:46:18,548] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.118:8091
[2016-02-11 21:46:18,554] - [cluster_helper:80] INFO - ns_server @ 172.23.107.118:8091 is running
[2016-02-11 21:46:18,568] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.171
[2016-02-11 21:46:18,603] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.171:8091
[2016-02-11 21:46:18,609] - [cluster_helper:80] INFO - ns_server @ 172.23.107.171:8091 is running
[2016-02-11 21:46:18,609] - [xdcrnewbasetests:1090] INFO - removing xdcr/nodes settings
[2016-02-11 21:46:18,681] - [xdcrnewbasetests:1100] INFO - cleanup [ip:172.23.107.120 port:8091 ssh_username:root, ip:172.23.107.169 port:8091 ssh_username:root]
[2016-02-11 21:46:18,694] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.120
[2016-02-11 21:46:18,727] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.120:8091
[2016-02-11 21:46:18,736] - [cluster_helper:80] INFO - ns_server @ 172.23.107.120:8091 is running
[2016-02-11 21:46:18,748] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.169
[2016-02-11 21:46:18,781] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.169:8091
[2016-02-11 21:46:18,788] - [cluster_helper:80] INFO - ns_server @ 172.23.107.169:8091 is running
[2016-02-11 21:46:18,788] - [xdcrnewbasetests:2467] INFO - Initializing all clusters...
[2016-02-11 21:46:18,799] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.118:8091: True content: [4,5] command: cluster_compat_mode:get_compat_version().
[2016-02-11 21:46:19,479] - [rest_client:774] INFO - settings/web params on 172.23.107.118:8091:username=Administrator&password=welcome&port=8091
[2016-02-11 21:46:19,508] - [rest_client:811] INFO - pools/default params : memoryQuota=2069
[2016-02-11 21:46:19,535] - [rest_client:774] INFO - settings/web params on 172.23.107.171:8091:username=Administrator&password=welcome&port=8091
[2016-02-11 21:46:19,870] - [rest_client:811] INFO - pools/default params : memoryQuota=2069
[2016-02-11 21:46:20,878] - [task:370] INFO - adding node 172.23.107.171:8091 to cluster
[2016-02-11 21:46:20,879] - [rest_client:974] INFO - adding remote node @172.23.107.171:8091 to this cluster @172.23.107.118:8091
[2016-02-11 21:46:24,785] - [rest_client:1192] INFO - rebalance params : password=welcome&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.171%2Cns_1%40172.23.107.118
[2016-02-11 21:46:24,793] - [rest_client:1196] INFO - rebalance operation started
[2016-02-11 21:46:24,806] - [rest_client:1324] INFO - rebalance percentage : 0.00 %
[2016-02-11 21:46:34,825] - [task:486] INFO - rebalancing was completed with progress: 100% in 10.0311319828 sec
[2016-02-11 21:46:34,842] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.120:8091: True content: [4,5] command: cluster_compat_mode:get_compat_version().
[2016-02-11 21:46:35,502] - [rest_client:774] INFO - settings/web params on 172.23.107.120:8091:username=Administrator&password=welcome&port=8091
[2016-02-11 21:46:35,543] - [rest_client:811] INFO - pools/default params : memoryQuota=2069
[2016-02-11 21:46:35,564] - [rest_client:774] INFO - settings/web params on 172.23.107.169:8091:username=Administrator&password=welcome&port=8091
[2016-02-11 21:46:35,588] - [rest_client:811] INFO - pools/default params : memoryQuota=2069
[2016-02-11 21:46:36,596] - [task:370] INFO - adding node 172.23.107.169:8091 to cluster
[2016-02-11 21:46:36,597] - [rest_client:974] INFO - adding remote node @172.23.107.169:8091 to this cluster @172.23.107.120:8091
[2016-02-11 21:46:39,756] - [rest_client:1192] INFO - rebalance params : password=welcome&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.107.120
[2016-02-11 21:46:39,764] - [rest_client:1196] INFO - rebalance operation started
[2016-02-11 21:46:39,778] - [rest_client:1324] INFO - rebalance percentage : 0.00 %
[2016-02-11 21:46:49,798] - [task:486] INFO - rebalancing was completed with progress: 100% in 10.0337018967 sec
[2016-02-11 21:46:49,862] - [rest_client:1743] INFO - http://172.23.107.118:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1569&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None
[2016-02-11 21:46:49,873] - [rest_client:1765] INFO - 0.01 seconds to create bucket default
[2016-02-11 21:46:49,873] - [bucket_helper:305] INFO - waiting for memcached bucket : default in 172.23.107.118 to accept set ops
[2016-02-11 21:46:51,155] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:46:51,272] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:46:51,444] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:46:51,537] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:46:53,040] - [task:254] INFO - bucket 'default' was created with per node RAM quota: 1569
[2016-02-11 21:46:53,833] - [rest_client:1743] INFO - http://172.23.107.120:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1569&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None
[2016-02-11 21:46:53,846] - [rest_client:1765] INFO - 0.01 seconds to create bucket default
[2016-02-11 21:46:53,847] - [bucket_helper:305] INFO - waiting for memcached bucket : default in 172.23.107.120 to accept set ops
[2016-02-11 21:46:55,152] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:46:55,275] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:46:55,441] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:46:55,569] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:46:56,310] - [task:254] INFO - bucket 'default' was created with per node RAM quota: 1569
[2016-02-11 21:46:56,332] - [rest_client:1966] INFO - Update internal setting xdcrCheckpointInterval=60
[2016-02-11 21:46:56,354] - [rest_client:1966] INFO - Update internal setting xdcrCheckpointInterval=60
[2016-02-11 21:46:56,355] - [xdcrnewbasetests:2287] INFO - ==== XDCRNewbasetests setup is finished for test #6 replication_with_pause_and_resume ====
[2016-02-11 21:46:56,362] - [rest_client:864] INFO - adding remote cluster hostname:172.23.107.120:8091 with username:password Administrator:welcome name:remote_cluster_C1-C2 to source node: 172.23.107.118:8091
[2016-02-11 21:46:56,436] - [rest_client:864] INFO - adding remote cluster hostname:172.23.107.118:8091 with username:password Administrator:welcome name:remote_cluster_C2-C1 to source node: 172.23.107.120:8091
[2016-02-11 21:46:56,490] - [rest_client:911] INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {}
[2016-02-11 21:46:56,634] - [rest_client:924] INFO - Replication created with id: 595013fdca7b192e8967cd7ba93da41e/default/default
[2016-02-11 21:46:56,644] - [rest_client:911] INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C2-C1 with settings {}
[2016-02-11 21:46:56,781] - [rest_client:924] INFO - Replication created with id: 2b962b0b62153f92102d3c2a7026848d/default/default
[2016-02-11 21:46:56,856] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:46:56,949] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:46:57,142] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:46:57,244] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:46:57,329] - [xdcrnewbasetests:3007] INFO - sleep for 20 secs.  ...
[2016-02-11 21:47:17,464] - [rest_client:1997] INFO - Updated pauseRequested=true on bucket'default' on 172.23.107.118
[2016-02-11 21:47:18,866] - [rest_client:1997] INFO - Updated pauseRequested=true on bucket'default' on 172.23.107.120
[2016-02-11 21:47:18,923] - [xdcrnewbasetests:1763] INFO - Starting rebalance-in nodes:[ip:172.23.105.181 port:8091 ssh_username:root] at C1 cluster 172.23.107.118
[2016-02-11 21:47:18,943] - [xdcrnewbasetests:1763] INFO - Starting rebalance-in nodes:[ip:172.23.105.180 port:8091 ssh_username:root] at C2 cluster 172.23.107.120
[2016-02-11 21:47:18,969] - [xdcrnewbasetests:3007] INFO - sleep for 10 secs.  ...
[2016-02-11 21:47:19,084] - [task:370] INFO - adding node 172.23.105.181:8091 to cluster
[2016-02-11 21:47:19,085] - [rest_client:974] INFO - adding remote node @172.23.105.181:8091 to this cluster @172.23.107.118:8091
[2016-02-11 21:47:19,359] - [task:370] INFO - adding node 172.23.105.180:8091 to cluster
[2016-02-11 21:47:19,361] - [rest_client:974] INFO - adding remote node @172.23.105.180:8091 to this cluster @172.23.107.120:8091
[2016-02-11 21:47:22,081] - [rest_client:1192] INFO - rebalance params : password=welcome&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.181%2Cns_1%40172.23.107.171%2Cns_1%40172.23.107.118
[2016-02-11 21:47:22,103] - [rest_client:1196] INFO - rebalance operation started
[2016-02-11 21:47:22,126] - [rest_client:1324] INFO - rebalance percentage : 0.00 %
[2016-02-11 21:47:22,711] - [rest_client:1192] INFO - rebalance params : password=welcome&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.105.180%2Cns_1%40172.23.107.120
[2016-02-11 21:47:22,745] - [rest_client:1196] INFO - rebalance operation started
[2016-02-11 21:47:22,763] - [rest_client:1324] INFO - rebalance percentage : 0.00 %
[2016-02-11 21:47:26,114] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:26,254] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:26,398] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:26,476] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:26,679] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:26,726] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:26,875] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:26,887] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:27,104] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:27,112] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:27,306] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:27,310] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:27,554] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:27,561] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:27,759] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:27,763] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:27,962] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:27,983] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:28,177] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:28,196] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:28,405] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:28,445] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:28,584] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:28,645] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:28,802] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:28,864] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:28,963] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:29,064] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:29,138] - [rest_client:1997] INFO - Updated pauseRequested=false on bucket'default' on 172.23.107.118
[2016-02-11 21:47:29,225] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:29,296] - [xdcrnewbasetests:874] INFO - Outbound mutations on 172.23.107.118 is 4827
[2016-02-11 21:47:29,297] - [xdcrnewbasetests:875] INFO - Node 172.23.107.118 is replicating
[2016-02-11 21:47:29,362] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:29,474] - [rest_client:1997] INFO - Updated pauseRequested=false on bucket'default' on 172.23.107.120
[2016-02-11 21:47:29,518] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:29,596] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:29,622] - [xdcrnewbasetests:874] INFO - Outbound mutations on 172.23.107.120 is 32
[2016-02-11 21:47:29,623] - [xdcrnewbasetests:875] INFO - Node 172.23.107.120 is replicating
[2016-02-11 21:47:29,624] - [pauseResumeXDCR:115] INFO - Waiting for rebalance to complete...
[2016-02-11 21:47:29,760] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:29,880] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:29,979] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:30,091] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:30,245] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:30,342] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:30,471] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:30,620] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:30,750] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:30,870] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:31,024] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:31,069] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:31,309] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:31,350] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:31,546] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:31,587] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:31,812] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:31,872] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:32,022] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:32,090] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:32,211] - [rest_client:1324] INFO - rebalance percentage : 28.14 %
[2016-02-11 21:47:32,287] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:32,344] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:32,534] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:32,548] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:32,794] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:32,808] - [rest_client:1324] INFO - rebalance percentage : 28.72 %
[2016-02-11 21:47:32,843] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:33,038] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:33,102] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:33,343] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:33,368] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:33,554] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:33,573] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:33,840] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:33,842] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:34,049] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:34,091] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:34,325] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:34,347] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:34,523] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:34,595] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:34,736] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:34,806] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:34,939] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:35,012] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:35,178] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:35,258] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:35,398] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:35,531] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:35,644] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:35,805] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:35,880] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:36,019] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:36,131] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:36,326] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:36,343] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:36,555] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:36,560] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:36,816] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:36,826] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:37,045] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:37,056] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:37,287] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:37,290] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:37,512] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:37,527] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:37,762] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:37,766] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:37,945] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:38,041] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:38,172] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:38,264] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:38,395] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:38,544] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:38,630] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:38,813] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:38,825] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:39,053] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:39,066] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:39,256] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:39,265] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:39,513] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:39,523] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:39,714] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:39,753] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:39,991] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:40,008] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:40,163] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:40,237] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:40,329] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:40,499] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:40,594] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:40,746] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:40,772] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:40,980] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:41,019] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:41,189] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:41,346] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:41,420] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:41,631] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:41,636] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:41,847] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:41,887] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:42,046] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:42,160] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:42,239] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:42,295] - [rest_client:1324] INFO - rebalance percentage : 58.45 %
[2016-02-11 21:47:42,451] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:42,486] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:42,714] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:42,716] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:42,875] - [rest_client:1324] INFO - rebalance percentage : 69.01 %
[2016-02-11 21:47:42,955] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:42,961] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:43,156] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:43,279] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:43,445] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:43,490] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:43,696] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:43,706] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:43,926] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:43,941] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:44,175] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:44,191] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:44,429] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:44,441] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:44,665] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:44,724] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:44,894] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:44,970] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:45,103] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:45,250] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:45,335] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:45,474] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:45,554] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:45,713] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:45,811] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:45,966] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:46,030] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:46,247] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:46,304] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:46,501] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:46,506] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:46,751] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:46,785] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:46,957] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:47:47,033] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:47,593] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:47,757] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:47,901] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:48,091] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:48,274] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:48,445] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:48,624] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:48,833] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:47:52,321] - [rest_client:1324] INFO - rebalance percentage : 75.11 %
[2016-02-11 21:47:52,907] - [rest_client:1324] INFO - rebalance percentage : 91.29 %
[2016-02-11 21:47:56,837] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:47:56,985] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:47:57,140] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:48:02,359] - [rest_client:1324] INFO - rebalance percentage : 99.61 %
[2016-02-11 21:48:02,953] - [task:486] INFO - rebalancing was completed with progress: 100% in 40.2075638771 sec
[2016-02-11 21:48:05,311] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:48:05,532] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:48:05,784] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:48:07,929] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:48:08,089] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:48:08,372] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:48:12,397] - [task:486] INFO - rebalancing was completed with progress: 100% in 50.2935581207 sec
[2016-02-11 21:48:12,399] - [pauseResumeXDCR:115] INFO - Waiting for rebalance to complete...
[2016-02-11 21:48:12,402] - [pauseResumeXDCR:121] INFO - Waiting for loading to complete...
[2016-02-11 21:48:20,033] - [pauseResumeXDCR:121] INFO - Waiting for loading to complete...
[2016-02-11 21:48:20,034] - [xdcrnewbasetests:2933] INFO - Merging keys for replication Replication C1:default -> C2:default
[2016-02-11 21:48:20,184] - [xdcrnewbasetests:2872] INFO - src_kvstore has 30000 valid and 0 deleted keys
[2016-02-11 21:48:20,185] - [xdcrnewbasetests:2874] INFO - dest kvstore has 30000 valid and 0 deleted keys
[2016-02-11 21:48:20,869] - [xdcrnewbasetests:2923] INFO - After merging: destination bucket's kv_store now has 60000 valid keys and 0 deleted keys
[2016-02-11 21:48:20,871] - [xdcrnewbasetests:2933] INFO - Merging keys for replication Replication C2:default -> C1:default
[2016-02-11 21:48:21,116] - [xdcrnewbasetests:2872] INFO - src_kvstore has 60000 valid and 0 deleted keys
[2016-02-11 21:48:21,117] - [xdcrnewbasetests:2874] INFO - dest kvstore has 30000 valid and 0 deleted keys
[2016-02-11 21:48:22,276] - [xdcrnewbasetests:2923] INFO - After merging: destination bucket's kv_store now has 60000 valid keys and 0 deleted keys
[2016-02-11 21:48:22,387] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.105.181', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:48:22,387] - [data_helper:293] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:48:22,474] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.105.181', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:48:22,475] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.105.181', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (1608476192, 0, '')
[2016-02-11 21:48:22,620] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.118', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:48:22,620] - [data_helper:293] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:48:22,718] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.118', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:48:22,719] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.118', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (3851611033, 0, '')
[2016-02-11 21:48:22,760] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.171', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:48:22,760] - [data_helper:293] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:48:22,884] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.171', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:48:22,892] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.171', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (2540969961, 0, '')
[2016-02-11 21:48:22,892] - [xdcrnewbasetests:1583] INFO - wait for expiry pager to run on all these nodes
[2016-02-11 21:48:32,949] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.105.180', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:48:32,950] - [data_helper:293] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:48:33,031] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.105.180', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:48:33,032] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.105.180', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (419470002, 0, '')
[2016-02-11 21:48:33,075] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.120', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:48:33,076] - [data_helper:293] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:48:33,237] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.120', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:48:33,239] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.120', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (1993483017, 0, '')
[2016-02-11 21:48:33,273] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.169', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:48:33,273] - [data_helper:293] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:48:33,358] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.169', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:48:33,360] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.169', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (1613472879, 0, '')
[2016-02-11 21:48:33,360] - [xdcrnewbasetests:1583] INFO - wait for expiry pager to run on all these nodes
[2016-02-11 21:48:43,464] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:48:43,545] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.118:8091',default bucket
[2016-02-11 21:48:43,561] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:48:43,668] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.171:8091',default bucket
[2016-02-11 21:48:43,681] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:48:43,758] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.181:8091',default bucket
[2016-02-11 21:48:44,030] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:48:44,132] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.120:8091',default bucket
[2016-02-11 21:48:44,155] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:48:44,247] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.169:8091',default bucket
[2016-02-11 21:48:44,263] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:48:44,349] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.180:8091',default bucket
[2016-02-11 21:48:44,367] - [xdcrnewbasetests:2170] INFO - Waiting for dcp queue to drain on cluster node: 172.23.107.118
[2016-02-11 21:48:44,466] - [xdcrnewbasetests:2180] INFO - Current dcp queue size on C1 for default is 0
[2016-02-11 21:48:44,467] - [xdcrnewbasetests:2170] INFO - Waiting for dcp queue to drain on cluster node: 172.23.107.120
[2016-02-11 21:48:44,598] - [xdcrnewbasetests:2180] INFO - Current dcp queue size on C2 for default is 0
[2016-02-11 21:48:44,599] - [xdcrnewbasetests:2200] INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.107.118
[2016-02-11 21:48:44,660] - [xdcrnewbasetests:2214] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0
[2016-02-11 21:48:44,661] - [xdcrnewbasetests:2200] INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.107.120
[2016-02-11 21:48:44,747] - [xdcrnewbasetests:2214] INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0
[2016-02-11 21:48:44,953] - [xdcrnewbasetests:2074] INFO - Saw: vb_active_curr_items 60000 == 60000 expected on C1, default bucket
[2016-02-11 21:48:45,110] - [xdcrnewbasetests:2108] INFO - Saw: vb_replica_curr_items 60000 == 60000 expected on C1, default bucket
[2016-02-11 21:48:45,286] - [xdcrnewbasetests:2074] INFO - Saw: vb_active_curr_items 60000 == 60000 expected on C2, default bucket
[2016-02-11 21:48:45,461] - [xdcrnewbasetests:2108] INFO - Saw: vb_replica_curr_items 60000 == 60000 expected on C2, default bucket
[2016-02-11 21:48:45,538] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:48:45,708] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:48:45,888] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:48:46,105] - [task:1514] INFO - 60000 items will be verified on default bucket
[2016-02-11 21:48:46,765] - [task:1523] INFO - 0 items were verified
[2016-02-11 21:48:52,950] - [task:1523] INFO - 10000 items were verified
[2016-02-11 21:48:59,025] - [task:1523] INFO - 20000 items were verified
[2016-02-11 21:49:05,281] - [task:1523] INFO - 30000 items were verified
[2016-02-11 21:49:11,351] - [task:1523] INFO - 40000 items were verified
[2016-02-11 21:49:17,492] - [task:1523] INFO - 50000 items were verified
[2016-02-11 21:49:23,653] - [task:1523] INFO - 60000 items were verified
[2016-02-11 21:49:23,653] - [task:1527] INFO - 60000 items were verified in 37.548055172 sec.the average number of ops                - 1597.95214835 per second
[2016-02-11 21:49:23,751] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:49:23,863] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:49:23,970] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:49:24,184] - [task:1514] INFO - 60000 items will be verified on default bucket
[2016-02-11 21:49:24,408] - [task:1523] INFO - 0 items were verified
[2016-02-11 21:49:30,630] - [task:1523] INFO - 10000 items were verified
[2016-02-11 21:49:37,140] - [task:1523] INFO - 20000 items were verified
[2016-02-11 21:49:43,406] - [task:1523] INFO - 30000 items were verified
[2016-02-11 21:49:49,984] - [task:1523] INFO - 40000 items were verified
[2016-02-11 21:49:56,559] - [task:1523] INFO - 50000 items were verified
[2016-02-11 21:50:02,993] - [task:1523] INFO - 60000 items were verified
[2016-02-11 21:50:02,994] - [task:1527] INFO - 60000 items were verified in 38.8088018894 sec.the average number of ops                - 1546.04090534 per second
[2016-02-11 21:50:02,994] - [xdcrnewbasetests:2838] INFO - Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.107.118 -> Couchbase Cluster: C2, Master Ip: 172.23.107.120, bucket default
[2016-02-11 21:50:03,104] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:50:03,229] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:50:03,425] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:50:03,612] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:50:03,726] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:50:03,849] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:50:04,027] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:50:04,146] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:50:04,260] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:50:04,673] - [task:1627] INFO - RevID verification: in progress for default ...
[2016-02-11 21:52:06,348] - [task:1665] INFO - 50000 items have been verified
[2016-02-11 21:52:29,686] - [task:1635] INFO - RevId Verification : 60000 existing items have been verified
[2016-02-11 21:52:29,687] - [task:1637] INFO - RevId Verification : 0 deleted items have been verified
[2016-02-11 21:52:29,687] - [task:1640] INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket
[2016-02-11 21:52:29,832] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.105.180', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:52:29,832] - [data_helper:293] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:52:29,978] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.105.180', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:52:29,979] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.105.180', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (910699825, 0, '')
[2016-02-11 21:52:30,020] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.120', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:52:30,021] - [data_helper:293] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:52:30,125] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.120', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:52:30,127] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.120', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (623534165, 0, '')
[2016-02-11 21:52:30,154] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.169', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:52:30,154] - [data_helper:293] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:52:30,233] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.169', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:52:30,235] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.169', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (3533158404, 0, '')
[2016-02-11 21:52:30,235] - [xdcrnewbasetests:1583] INFO - wait for expiry pager to run on all these nodes
[2016-02-11 21:52:40,286] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.105.181', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:52:40,286] - [data_helper:293] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:52:40,367] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.105.181', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:52:40,368] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.105.181', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (3657209095, 0, '')
[2016-02-11 21:52:40,397] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.118', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:52:40,398] - [data_helper:293] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:52:40,481] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.118', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:52:40,482] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.118', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (2879710013, 0, '')
[2016-02-11 21:52:40,525] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.171', 'password': 'welcome', 'port': u'8091'}
[2016-02-11 21:52:40,526] - [data_helper:293] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:52:40,631] - [cluster_helper:341] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.171', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2016-02-11 21:52:40,632] - [cluster_helper:355] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.171', 'password': 'welcome', 'port': u'8091'}, exp_pager_stime to 10, result: (4005965693, 0, '')
[2016-02-11 21:52:40,633] - [xdcrnewbasetests:1583] INFO - wait for expiry pager to run on all these nodes
[2016-02-11 21:52:50,686] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:52:50,800] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.120:8091',default bucket
[2016-02-11 21:52:50,816] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:52:50,912] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.169:8091',default bucket
[2016-02-11 21:52:50,932] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:52:51,028] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.180:8091',default bucket
[2016-02-11 21:52:51,086] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:52:51,222] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.118:8091',default bucket
[2016-02-11 21:52:51,239] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:52:51,341] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.171:8091',default bucket
[2016-02-11 21:52:51,356] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:52:51,446] - [task:538] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.181:8091',default bucket
[2016-02-11 21:52:51,492] - [xdcrnewbasetests:2170] INFO - Waiting for dcp queue to drain on cluster node: 172.23.107.120
[2016-02-11 21:52:51,604] - [xdcrnewbasetests:2180] INFO - Current dcp queue size on C2 for default is 0
[2016-02-11 21:52:51,605] - [xdcrnewbasetests:2170] INFO - Waiting for dcp queue to drain on cluster node: 172.23.107.118
[2016-02-11 21:52:51,724] - [xdcrnewbasetests:2180] INFO - Current dcp queue size on C1 for default is 0
[2016-02-11 21:52:51,724] - [xdcrnewbasetests:2200] INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.107.120
[2016-02-11 21:52:51,800] - [xdcrnewbasetests:2214] INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0
[2016-02-11 21:52:51,800] - [xdcrnewbasetests:2200] INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.107.118
[2016-02-11 21:52:51,858] - [xdcrnewbasetests:2214] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0
[2016-02-11 21:52:52,273] - [xdcrnewbasetests:2074] INFO - Saw: vb_active_curr_items 60000 == 60000 expected on C2, default bucket
[2016-02-11 21:52:52,475] - [xdcrnewbasetests:2108] INFO - Saw: vb_replica_curr_items 60000 == 60000 expected on C2, default bucket
[2016-02-11 21:52:52,646] - [xdcrnewbasetests:2074] INFO - Saw: vb_active_curr_items 60000 == 60000 expected on C1, default bucket
[2016-02-11 21:52:52,811] - [xdcrnewbasetests:2108] INFO - Saw: vb_replica_curr_items 60000 == 60000 expected on C1, default bucket
[2016-02-11 21:52:52,923] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:52:53,057] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:52:53,154] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:52:53,357] - [task:1514] INFO - 60000 items will be verified on default bucket
[2016-02-11 21:52:54,033] - [task:1523] INFO - 0 items were verified
[2016-02-11 21:53:00,514] - [task:1523] INFO - 10000 items were verified
[2016-02-11 21:53:06,654] - [task:1523] INFO - 20000 items were verified
[2016-02-11 21:53:12,776] - [task:1523] INFO - 30000 items were verified
[2016-02-11 21:53:19,171] - [task:1523] INFO - 40000 items were verified
[2016-02-11 21:53:25,783] - [task:1523] INFO - 50000 items were verified
[2016-02-11 21:53:32,528] - [task:1523] INFO - 60000 items were verified
[2016-02-11 21:53:32,529] - [task:1527] INFO - 60000 items were verified in 39.1717569828 sec.the average number of ops                - 1531.71570218 per second
[2016-02-11 21:53:32,608] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:53:32,730] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:53:32,924] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:53:33,147] - [task:1514] INFO - 60000 items will be verified on default bucket
[2016-02-11 21:53:33,524] - [task:1523] INFO - 0 items were verified
[2016-02-11 21:53:39,766] - [task:1523] INFO - 10000 items were verified
[2016-02-11 21:53:46,628] - [task:1523] INFO - 20000 items were verified
[2016-02-11 21:53:53,094] - [task:1523] INFO - 30000 items were verified
[2016-02-11 21:53:59,698] - [task:1523] INFO - 40000 items were verified
[2016-02-11 21:54:06,331] - [task:1523] INFO - 50000 items were verified
[2016-02-11 21:54:13,707] - [task:1523] INFO - 60000 items were verified
[2016-02-11 21:54:13,708] - [task:1527] INFO - 60000 items were verified in 40.5600438118 sec.the average number of ops                - 1479.28797773 per second
[2016-02-11 21:54:13,709] - [xdcrnewbasetests:2838] INFO - Verifying RevIds for Couchbase Cluster: C2, Master Ip: 172.23.107.120 -> Couchbase Cluster: C1, Master Ip: 172.23.107.118, bucket default
[2016-02-11 21:54:13,816] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:54:13,960] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:54:14,093] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:54:14,371] - [data_helper:295] INFO - creating direct client 172.23.107.120:11210 default
[2016-02-11 21:54:14,499] - [data_helper:295] INFO - creating direct client 172.23.107.169:11210 default
[2016-02-11 21:54:14,624] - [data_helper:295] INFO - creating direct client 172.23.105.180:11210 default
[2016-02-11 21:54:14,867] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:54:14,979] - [data_helper:295] INFO - creating direct client 172.23.107.171:11210 default
[2016-02-11 21:54:15,161] - [data_helper:295] INFO - creating direct client 172.23.105.181:11210 default
[2016-02-11 21:54:15,610] - [task:1627] INFO - RevID verification: in progress for default ...
[2016-02-11 21:56:14,730] - [task:1665] INFO - 50000 items have been verified
[2016-02-11 21:56:37,790] - [task:1635] INFO - RevId Verification : 60000 existing items have been verified
[2016-02-11 21:56:37,790] - [task:1637] INFO - RevId Verification : 0 deleted items have been verified
[2016-02-11 21:56:37,790] - [task:1640] INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket
[2016-02-11 21:56:37,934] - [remote_util:165] INFO - connecting to 172.23.107.118 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:38,158] - [remote_util:198] INFO - Connected to 172.23.107.118
[2016-02-11 21:56:38,173] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.118:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:38,975] - [remote_util:2222] INFO - running command.raw on 172.23.107.118: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:39,090] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:39,091] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:39,205] - [xdcrnewbasetests:2967] INFO - Initial panic: count on 172.23.107.118 :0, now :0
[2016-02-11 21:56:39,206] - [remote_util:165] INFO - connecting to 172.23.107.118 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:39,420] - [remote_util:198] INFO - Connected to 172.23.107.118
[2016-02-11 21:56:39,435] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.118:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:40,235] - [remote_util:2222] INFO - running command.raw on 172.23.107.118: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:40,341] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:40,342] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:40,457] - [xdcrnewbasetests:2967] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.118 :0, now :0
[2016-02-11 21:56:40,458] - [remote_util:165] INFO - connecting to 172.23.107.171 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:40,685] - [remote_util:198] INFO - Connected to 172.23.107.171
[2016-02-11 21:56:40,703] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.171:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:41,428] - [remote_util:2222] INFO - running command.raw on 172.23.107.171: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:41,560] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:41,561] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:41,676] - [xdcrnewbasetests:2967] INFO - Initial panic: count on 172.23.107.171 :0, now :0
[2016-02-11 21:56:41,677] - [remote_util:165] INFO - connecting to 172.23.107.171 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:41,891] - [remote_util:198] INFO - Connected to 172.23.107.171
[2016-02-11 21:56:41,906] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.171:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:42,607] - [remote_util:2222] INFO - running command.raw on 172.23.107.171: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:42,665] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:42,666] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:42,780] - [xdcrnewbasetests:2967] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.171 :0, now :0
[2016-02-11 21:56:42,781] - [remote_util:165] INFO - connecting to 172.23.107.120 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:43,011] - [remote_util:198] INFO - Connected to 172.23.107.120
[2016-02-11 21:56:43,030] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.120:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:43,701] - [remote_util:2222] INFO - running command.raw on 172.23.107.120: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:43,824] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:43,825] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:43,827] - [xdcrnewbasetests:2967] INFO - Initial panic: count on 172.23.107.120 :0, now :0
[2016-02-11 21:56:43,827] - [remote_util:165] INFO - connecting to 172.23.107.120 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:44,050] - [remote_util:198] INFO - Connected to 172.23.107.120
[2016-02-11 21:56:44,177] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.120:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:44,866] - [remote_util:2222] INFO - running command.raw on 172.23.107.120: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:44,996] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:44,997] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:45,112] - [xdcrnewbasetests:2967] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.120 :0, now :0
[2016-02-11 21:56:45,113] - [remote_util:165] INFO - connecting to 172.23.107.169 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:45,326] - [remote_util:198] INFO - Connected to 172.23.107.169
[2016-02-11 21:56:45,339] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.169:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:45,912] - [remote_util:2222] INFO - running command.raw on 172.23.107.169: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:46,015] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:46,017] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:46,132] - [xdcrnewbasetests:2967] INFO - Initial panic: count on 172.23.107.169 :0, now :0
[2016-02-11 21:56:46,132] - [remote_util:165] INFO - connecting to 172.23.107.169 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:46,348] - [remote_util:198] INFO - Connected to 172.23.107.169
[2016-02-11 21:56:46,361] - [rest_client:1208] INFO - /diag/eval status on 172.23.107.169:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:46,920] - [remote_util:2222] INFO - running command.raw on 172.23.107.169: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:47,050] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:47,051] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:47,053] - [xdcrnewbasetests:2967] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.169 :0, now :0
[2016-02-11 21:56:47,053] - [remote_util:165] INFO - connecting to 172.23.105.178 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:47,275] - [remote_util:198] INFO - Connected to 172.23.105.178
[2016-02-11 21:56:47,282] - [rest_client:757] ERROR - http://172.23.105.178:8091/nodes/self error 401 reason: status: 401, content:  
http://172.23.105.178:8091/nodes/self: 
[2016-02-11 21:56:47,288] - [rest_client:757] ERROR - http://172.23.105.178:8091/diag/eval/ error 401 reason: status: 401, content:  
[2016-02-11 21:56:47,288] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.178:8091: False content:  command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:47,826] - [remote_util:2222] INFO - running command.raw on 172.23.105.178: zgrep "panic:" /goxdcr.log* | wc -l
[2016-02-11 21:56:47,918] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:47,919] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:47,921] - [xdcrnewbasetests:2967] INFO - Initial panic: count on 172.23.105.178 :0, now :0
[2016-02-11 21:56:47,922] - [remote_util:165] INFO - connecting to 172.23.105.178 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:48,134] - [remote_util:198] INFO - Connected to 172.23.105.178
[2016-02-11 21:56:48,139] - [rest_client:757] ERROR - http://172.23.105.178:8091/nodes/self error 401 reason: status: 401, content:  
http://172.23.105.178:8091/nodes/self: 
[2016-02-11 21:56:48,144] - [rest_client:757] ERROR - http://172.23.105.178:8091/diag/eval/ error 401 reason: status: 401, content:  
[2016-02-11 21:56:48,144] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.178:8091: False content:  command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:48,688] - [remote_util:2222] INFO - running command.raw on 172.23.105.178: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /goxdcr.log* | wc -l
[2016-02-11 21:56:48,784] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:48,785] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:48,899] - [xdcrnewbasetests:2967] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.105.178 :0, now :0
[2016-02-11 21:56:48,900] - [remote_util:165] INFO - connecting to 172.23.105.179 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:49,115] - [remote_util:198] INFO - Connected to 172.23.105.179
[2016-02-11 21:56:49,127] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.179:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:49,619] - [remote_util:2222] INFO - running command.raw on 172.23.105.179: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:49,712] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:49,713] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:49,828] - [xdcrnewbasetests:2967] INFO - Initial panic: count on 172.23.105.179 :0, now :0
[2016-02-11 21:56:49,828] - [remote_util:165] INFO - connecting to 172.23.105.179 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:50,047] - [remote_util:198] INFO - Connected to 172.23.105.179
[2016-02-11 21:56:50,057] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.179:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:50,606] - [remote_util:2222] INFO - running command.raw on 172.23.105.179: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:50,704] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:50,705] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:50,707] - [xdcrnewbasetests:2967] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.105.179 :0, now :0
[2016-02-11 21:56:50,708] - [remote_util:165] INFO - connecting to 172.23.105.180 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:50,918] - [remote_util:198] INFO - Connected to 172.23.105.180
[2016-02-11 21:56:50,930] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.180:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:51,530] - [remote_util:2222] INFO - running command.raw on 172.23.105.180: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:51,629] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:51,630] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:51,745] - [xdcrnewbasetests:2967] INFO - Initial panic: count on 172.23.105.180 :0, now :0
[2016-02-11 21:56:51,745] - [remote_util:165] INFO - connecting to 172.23.105.180 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:51,955] - [remote_util:198] INFO - Connected to 172.23.105.180
[2016-02-11 21:56:51,969] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.180:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:52,507] - [remote_util:2222] INFO - running command.raw on 172.23.105.180: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:52,607] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:52,608] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:52,610] - [xdcrnewbasetests:2967] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.105.180 :0, now :0
[2016-02-11 21:56:52,610] - [remote_util:165] INFO - connecting to 172.23.105.181 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:52,825] - [remote_util:198] INFO - Connected to 172.23.105.181
[2016-02-11 21:56:52,840] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.181:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:53,370] - [remote_util:2222] INFO - running command.raw on 172.23.105.181: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:53,473] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:53,474] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:53,588] - [xdcrnewbasetests:2967] INFO - Initial panic: count on 172.23.105.181 :0, now :0
[2016-02-11 21:56:53,589] - [remote_util:165] INFO - connecting to 172.23.105.181 with username : root password : couchbase ssh_key: 
[2016-02-11 21:56:53,802] - [remote_util:198] INFO - Connected to 172.23.105.181
[2016-02-11 21:56:53,813] - [rest_client:1208] INFO - /diag/eval status on 172.23.105.181:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-02-11 21:56:54,364] - [remote_util:2222] INFO - running command.raw on 172.23.105.181: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2016-02-11 21:56:54,441] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:56:54,441] - [xdcrnewbasetests:403] INFO - 0
[2016-02-11 21:56:54,556] - [xdcrnewbasetests:2967] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.105.181 :0, now :0
[2016-02-11 21:56:54,559] - [xdcrnewbasetests:2330] INFO - ====  XDCRNewbasetests cleanup is started for test #6 replication_with_pause_and_resume ====
[2016-02-11 21:56:54,559] - [xdcrnewbasetests:1090] INFO - removing xdcr/nodes settings
[2016-02-11 21:56:54,578] - [rest_client:947] INFO - Deleting replication /controller/cancelXDCR/595013fdca7b192e8967cd7ba93da41e%2Fdefault%2Fdefault
[2016-02-11 21:56:54,617] - [rest_client:895] INFO - removing remote cluster name:remote_cluster_C1-C2
[2016-02-11 21:56:54,648] - [xdcrnewbasetests:1100] INFO - cleanup [ip:172.23.107.118 port:8091 ssh_username:root, ip:172.23.107.171 port:8091 ssh_username:root, ip:172.23.105.181 port:8091 ssh_username:root]
[2016-02-11 21:56:54,681] - [bucket_helper:137] INFO - deleting existing buckets [u'default'] on 172.23.107.118
[2016-02-11 21:56:54,682] - [bucket_helper:139] INFO - remove bucket default ...
[2016-02-11 21:57:24,697] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default error 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"}
[2016-02-11 21:57:24,697] - [rest_client:1687] WARNING - Bucket deletion timed out waiting for all nodes
[2016-02-11 21:57:24,709] - [remote_util:165] INFO - connecting to 172.23.107.118 with username : root password : couchbase ssh_key: 
[2016-02-11 21:57:24,929] - [remote_util:198] INFO - Connected to 172.23.107.118
[2016-02-11 21:57:25,746] - [remote_util:2222] INFO - running command.raw on 172.23.107.118: ls -LR '/opt/couchbase/var/lib/couchbase/data'
[2016-02-11 21:57:25,835] - [remote_util:2259] INFO - command executed successfully
[2016-02-11 21:57:25,836] - [remote_util:2069] INFO - /opt/couchbase/var/lib/couchbase/data:
[2016-02-11 21:57:25,836] - [remote_util:2069] INFO - _replicator.couch.1
[2016-02-11 21:57:25,837] - [remote_util:2069] INFO - _users.couch.1
[2016-02-11 21:57:25,966] - [data_helper:295] INFO - creating direct client 172.23.107.118:11210 default
[2016-02-11 21:57:25,973] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:26,480] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:26,988] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:27,497] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:28,005] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:28,512] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:29,019] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:29,527] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:30,035] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:30,544] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:31,056] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:31,563] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:32,071] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:32,580] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:33,088] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:33,594] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:34,103] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:34,611] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:35,118] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:35,634] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:36,171] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:36,683] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:37,208] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:37,733] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:38,258] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:38,771] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:39,280] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:39,789] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:40,295] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:40,803] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:41,308] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:41,814] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:42,320] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:42,827] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:43,336] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:43,842] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:44,348] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:44,855] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:45,362] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:45,870] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:46,380] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:46,887] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:47,395] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:47,903] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:48,409] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:48,915] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:49,422] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:49,928] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:50,435] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:50,943] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:51,450] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:51,956] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:52,464] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:52,970] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:53,476] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:53,983] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:54,491] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:55,000] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:55,510] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:56,018] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:56,526] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:57,035] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:57,543] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:58,053] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:58,560] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:59,066] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:57:59,572] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:00,078] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:00,585] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:01,090] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:01,597] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:02,105] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:02,612] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:03,118] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:03,629] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:04,135] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:04,642] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:05,157] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:05,663] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:06,169] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:06,677] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:07,184] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:07,691] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:08,198] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:08,704] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:09,211] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:09,717] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:10,226] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:10,732] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:11,240] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:11,747] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:12,257] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:12,764] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:13,272] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:13,779] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:14,285] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:14,793] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:15,301] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:15,808] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:16,315] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:16,824] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:17,332] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:17,839] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:18,345] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:18,851] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:19,357] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:19,866] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:20,375] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:20,881] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:21,388] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:21,895] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:22,760] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:23,270] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:23,776] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:24,283] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:24,790] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:25,296] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:25,803] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:26,304] - [rest_client:119] INFO - vbucket map is not ready for bucket default after waiting 60 seconds
[2016-02-11 21:58:26,317] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:26,327] - [rest_client:757] ERROR - http://172.23.107.118:8091/pools/default/buckets/default?basic_stats=true error 404 reason: status: 404, content: Requested resource not found.
 Requested resource not found.
[2016-02-11 21:58:26,327] - [bucket_helper:152] ERROR - Unable to get timings for bucket
[2016-02-11 21:58:26,328] - [bucket_helper:153] INFO - deleted bucket : default from 172.23.107.118
[2016-02-11 21:58:26,328] - [bucket_helper:226] INFO - waiting for bucket deletion to complete....
[2016-02-11 21:58:26,332] - [rest_client:127] INFO - node 172.23.107.118 existing buckets : []
[2016-02-11 21:58:26,358] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes
[2016-02-11 21:58:26,363] - [rest_client:1192] INFO - rebalance params : password=welcome&ejectedNodes=ns_1%40172.23.105.181%2Cns_1%40172.23.107.171&user=Administrator&knownNodes=ns_1%40172.23.105.181%2Cns_1%40172.23.107.171%2Cns_1%40172.23.107.118
[2016-02-11 21:58:26,374] - [rest_client:1196] INFO - rebalance operation started
[2016-02-11 21:58:26,382] - [rest_client:1324] INFO - rebalance percentage : 0.00 %
[2016-02-11 21:58:36,405] - [rest_client:1279] INFO - rebalance progress took 10.0300741196 seconds 
[2016-02-11 21:58:36,405] - [rest_client:1280] INFO - sleep for 10 seconds after rebalance...
[2016-02-11 21:58:46,424] - [rest_client:757] ERROR - http://172.23.105.181:8091/nodes/self error 404 reason: unknown "Node is unknown to this cluster."
http://172.23.105.181:8091/nodes/self with status False: Node is unknown to this cluster.
[2016-02-11 21:58:46,425] - [rest_client:292] ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry
[2016-02-11 21:58:51,440] - [rest_client:757] ERROR - http://172.23.105.181:8091/nodes/self error 404 reason: unknown "Node is unknown to this cluster."
http://172.23.105.181:8091/nodes/self with status False: Node is unknown to this cluster.
[2016-02-11 21:58:51,440] - [rest_client:292] ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry
[2016-02-11 21:58:56,453] - [rest_client:757] ERROR - http://172.23.105.181:8091/nodes/self error 404 reason: unknown "Node is unknown to this cluster."
http://172.23.105.181:8091/nodes/self with status False: Node is unknown to this cluster.
[2016-02-11 21:58:56,454] - [rest_client:292] ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry
[2016-02-11 21:59:01,459] - [rest_client:295] ERROR - node 172.23.105.181:8091 is in a broken state!
[2016-02-11 21:59:01,460] - [cluster_helper:269] ERROR - can't create rest connection after rebalance out for ejected nodes,                        will retry after 10 seconds according to MB-8430: unable to reach the host @ 172.23.105.181 
[2016-02-11 21:59:11,472] - [rest_client:757] ERROR - http://172.23.105.181:8091/nodes/self error 404 reason: unknown "Node is unknown to this cluster."
http://172.23.105.181:8091/nodes/self with status False: Node is unknown to this cluster.
[2016-02-11 21:59:11,472] - [rest_client:292] ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry
[2016-02-11 21:59:16,485] - [rest_client:757] ERROR - http://172.23.105.181:8091/nodes/self error 404 reason: unknown "Node is unknown to this cluster."
http://172.23.105.181:8091/nodes/self with status False: Node is unknown to this cluster.
[2016-02-11 21:59:16,486] - [rest_client:292] ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry
[2016-02-11 21:59:21,498] - [rest_client:757] ERROR - http://172.23.105.181:8091/nodes/self error 404 reason: unknown "Node is unknown to this cluster."
http://172.23.105.181:8091/nodes/self with status False: Node is unknown to this cluster.
[2016-02-11 21:59:21,498] - [rest_client:292] ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry
[2016-02-11 21:59:26,504] - [rest_client:295] ERROR - node 172.23.105.181:8091 is in a broken state!
ERROR

======================================================================
ERROR: replication_with_pause_and_resume (xdcr.pauseResumeXDCR.PauseResumeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/xdcr/pauseResumeXDCR.py", line 26, in tearDown
    super(PauseResumeTest, self).tearDown()
  File "pytests/xdcr/xdcrnewbasetests.py", line 2332, in tearDown
    cb_cluster.cleanup_cluster(self)
  File "pytests/xdcr/xdcrnewbasetests.py", line 1115, in cleanup_cluster
    ClusterOperationHelper.cleanup_cluster([node])
  File "lib/membase/helper/cluster_helper.py", line 271, in cleanup_cluster
    rest = RestConnection(removed)
  File "lib/membase/api/rest_client.py", line 296, in __init__
    raise ServerUnavailableException(self.ip)
ServerUnavailableException: unable to reach the host @ 172.23.105.181

----------------------------------------------------------------------
Ran 1 test in 803.950s

FAILED (errors=1)