jim@mancouch:~/couchbase/testrunner$ ./testrunner -i ./b/resources/dev-6-nodes-rebal-xdcr.ini -p stop-on-failure=False,fail_on_errors=1 -t "xdcr.rebalanceXDCR.Rebalance.swap_rebalance,items=100000,rdirection=bidirection,ctopology=chain,update=C1-C2,delete=C1-C2,rebalance=C2,GROUP=P2;xmem" Global Test input params: {'cluster_name': 'dev-6-nodes-rebal-xdcr', 'fail_on_errors': '1', 'ini': './b/resources/dev-6-nodes-rebal-xdcr.ini', 'num_nodes': 6, 'stop-on-failure': 'False'} Logs will be stored at /home/jim/couchbase/testrunner/logs/testrunner-19-May-22_09-34-52/test_1 ./testrunner -i ./b/resources/dev-6-nodes-rebal-xdcr.ini -p stop-on-failure=False,fail_on_errors=1 -t xdcr.rebalanceXDCR.Rebalance.swap_rebalance,items=100000,rdirection=bidirection,ctopology=chain,update=C1-C2,delete=C1-C2,rebalance=C2,GROUP=P2;xmem Test Input params: {'fail_on_errors': '1', 'GROUP': 'P2;xmem', 'items': '100000', 'update': 'C1-C2', 'num_nodes': 6, 'cluster_name': 'dev-6-nodes-rebal-xdcr', 'ctopology': 'chain', 'rdirection': 'bidirection', 'stop-on-failure': 'False', 'ini': './b/resources/dev-6-nodes-rebal-xdcr.ini', 'case_number': 1, 'logs_folder': '/home/jim/couchbase/testrunner/logs/testrunner-19-May-22_09-34-52/test_1', 'rebalance': 'C2', 'delete': 'C1-C2'} Run before suite setup for xdcr.rebalanceXDCR.Rebalance.swap_rebalance swap_rebalance (xdcr.rebalanceXDCR.Rebalance) ... 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9010: True content: "/home/jim/couchbase/ns_server/logs/n_10" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_10"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_10"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9011: True content: "/home/jim/couchbase/ns_server/logs/n_11" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_11"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_11"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9012: True content: "/home/jim/couchbase/ns_server/logs/n_12" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_12"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_12"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9013: True content: "/home/jim/couchbase/ns_server/logs/n_13" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_13"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_13"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9014: True content: "/home/jim/couchbase/ns_server/logs/n_14" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_14"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_14"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9015: True content: "/home/jim/couchbase/ns_server/logs/n_15" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_15"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_15"/goxdcr.log* 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__initialize_error_count_dict] {'127.0.0.1': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}} 2019-05-22 09:34:53 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9010: True content: "/home/jim/couchbase/ns_server/logs/n_10" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_10"/goxdcr.log* 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9011: True content: "/home/jim/couchbase/ns_server/logs/n_11" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_11"/goxdcr.log* 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9012: True content: "/home/jim/couchbase/ns_server/logs/n_12" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_12"/goxdcr.log* 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9013: True content: "/home/jim/couchbase/ns_server/logs/n_13" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_13"/goxdcr.log* 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9014: True content: "/home/jim/couchbase/ns_server/logs/n_14" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_14"/goxdcr.log* 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9015: True content: "/home/jim/couchbase/ns_server/logs/n_15" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_15"/goxdcr.log* 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__initialize_repl_restart_count_dict] {'127.0.0.1': 0} 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.setUp] ==== XDCRNewbasetests setup is started for test #1 swap_rebalance ==== 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] removing xdcr/nodes settings 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] cleanup [ip:127.0.0.1 port:9010 ssh_username:Administrator, ip:127.0.0.1 port:9011 ssh_username:Administrator] 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9010 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9010 is running 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9011 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9011 is running 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] removing xdcr/nodes settings 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] cleanup [ip:127.0.0.1 port:9012 ssh_username:Administrator, ip:127.0.0.1 port:9013 ssh_username:Administrator] 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9012 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9012 is running 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9013 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9013 is running 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__init_clusters] Initializing all clusters... 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:54 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://127.0.0.1:9010/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:asdasd 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [xdcrnewbasetests.enable_diag_eval_on_non_local_hosts] Running in compatibility mode, not enabled diag/eval for non-local hosts 2019-05-22 09:34:54 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9010: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9010 ssh_username:Administrator, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'n_10@cb.local', 'uptime': u'14', 'mcdMemoryReserved': 103106, 'storageTotalRam': 128883, 'hostname': u'127.0.0.1:9010', 'storage': [], 'moxi': 11211, 'port': u'9010', 'version': u'0.0.0-0000-community', 'memcached': 12020, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 120684609536, 'memoryTotal': 135144009728, 'memoryQuota': 76664, 'mcdMemoryAllocated': 103106, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-05-22 09:34:55 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://127.0.0.1:9010/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:asdasd 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=68737 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9010:username=Administrator&password=asdasd&port=9010 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl http://Administrator:asdasd@localhost:9010/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9010: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9011 ssh_username:Administrator, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'n_11@cb.local', 'uptime': u'13', 'mcdMemoryReserved': 103106, 'storageTotalRam': 128883, 'hostname': u'127.0.0.1:9011', 'storage': [], 'moxi': 11211, 'port': u'9011', 'version': u'0.0.0-0000-community', 'memcached': 12022, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 120684441600, 'memoryTotal': 135144009728, 'memoryQuota': 76664, 'mcdMemoryAllocated': 103106, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-05-22 09:34:55 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://127.0.0.1:9011/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:asdasd 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=68737 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9011:username=Administrator&password=asdasd&port=9011 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl http://Administrator:asdasd@localhost:9011/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9011: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-05-22 09:34:55 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb 2019-05-22 09:34:56 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 127.0.0.1:9011 to cluster 2019-05-22 09:34:56 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @127.0.0.1:9011 to this cluster @127.0.0.1:9010 2019-05-22 09:34:59 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'asdasd', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'n_11@127.0.0.1,n_10@192.168.4.39'} 2019-05-22 09:34:59 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2019-05-22 09:34:59 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-05-22 09:34:59 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-05-22 09:35:09 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2019-05-22 09:35:09 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.0419230461 sec 2019-05-22 09:35:09 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:35:09 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:35:09 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://127.0.0.1:9012/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:asdasd 2019-05-22 09:35:09 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases 2019-05-22 09:35:09 | INFO | MainProcess | test_thread | [xdcrnewbasetests.enable_diag_eval_on_non_local_hosts] Running in compatibility mode, not enabled diag/eval for non-local hosts 2019-05-22 09:35:09 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9012: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9012 ssh_username:Administrator, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'n_12@cb.local', 'uptime': u'28', 'mcdMemoryReserved': 103106, 'storageTotalRam': 128883, 'hostname': u'127.0.0.1:9012', 'storage': [], 'moxi': 11211, 'port': u'9012', 'version': u'0.0.0-0000-community', 'memcached': 12024, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 120624467968, 'memoryTotal': 135144009728, 'memoryQuota': 76664, 'mcdMemoryAllocated': 103106, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-05-22 09:35:10 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://127.0.0.1:9012/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:asdasd 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=68737 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9012:username=Administrator&password=asdasd&port=9012 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl http://Administrator:asdasd@localhost:9012/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9012: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9013 ssh_username:Administrator, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'n_13@cb.local', 'uptime': u'28', 'mcdMemoryReserved': 103106, 'storageTotalRam': 128883, 'hostname': u'127.0.0.1:9013', 'storage': [], 'moxi': 11211, 'port': u'9013', 'version': u'0.0.0-0000-community', 'memcached': 12026, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 120622161920, 'memoryTotal': 135144009728, 'memoryQuota': 76664, 'mcdMemoryAllocated': 103106, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-05-22 09:35:10 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://127.0.0.1:9013/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:asdasd 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=68737 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9013:username=Administrator&password=asdasd&port=9013 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl http://Administrator:asdasd@localhost:9013/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9013: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-05-22 09:35:10 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb 2019-05-22 09:35:11 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 127.0.0.1:9013 to cluster 2019-05-22 09:35:11 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @127.0.0.1:9013 to this cluster @127.0.0.1:9012 2019-05-22 09:35:14 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'asdasd', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'n_12@192.168.4.39,n_13@127.0.0.1'} 2019-05-22 09:35:14 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2019-05-22 09:35:14 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-05-22 09:35:14 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-05-22 09:35:24 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2019-05-22 09:35:24 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.0414600372 sec 2019-05-22 09:35:24 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://127.0.0.1:9010/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 404 reason: unknown "User was not found." auth: Administrator:asdasd 2019-05-22 09:35:24 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -"User was not found." 2019-05-22 09:35:25 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://127.0.0.1:9012/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 404 reason: unknown "User was not found." auth: Administrator:asdasd 2019-05-22 09:35:25 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -"User was not found." 2019-05-22 09:35:25 | ERROR | MainProcess | Cluster_Thread | [rest_client.is_enterprise_edition] IP(s) for node(s) with community edition [u'192.168.4.39', u'127.0.0.1'] 2019-05-22 09:35:25 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://127.0.0.1:9010/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=sasl&replicaIndex=1&name=default&saslPassword=None&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=68237 2019-05-22 09:35:25 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket default 2019-05-22 09:35:25 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops 2019-05-22 09:35:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:35:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:35:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:35:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:35:28 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 68237 2019-05-22 09:35:28 | ERROR | MainProcess | Cluster_Thread | [rest_client.is_enterprise_edition] IP(s) for node(s) with community edition [u'192.168.4.39', u'127.0.0.1'] 2019-05-22 09:35:28 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://127.0.0.1:9012/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=sasl&replicaIndex=1&name=default&saslPassword=None&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=68237 2019-05-22 09:35:28 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket default 2019-05-22 09:35:28 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops 2019-05-22 09:35:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:35:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12026 default 2019-05-22 09:35:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:35:31 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12026 default 2019-05-22 09:35:31 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 68237 2019-05-22 09:35:31 | INFO | MainProcess | test_thread | [xdcrnewbasetests.setUp] ==== XDCRNewbasetests setup is finished for test #1 swap_rebalance ==== 2019-05-22 09:35:31 | INFO | MainProcess | test_thread | [rest_client.add_remote_cluster] adding remote cluster hostname:127.0.0.1:9012 with username:password Administrator:asdasd name:remote_cluster_C1-C2 to source node: 127.0.0.1:9010 2019-05-22 09:35:31 | INFO | MainProcess | test_thread | [rest_client.add_remote_cluster] adding remote cluster hostname:127.0.0.1:9010 with username:password Administrator:asdasd name:remote_cluster_C2-C1 to source node: 127.0.0.1:9012 2019-05-22 09:35:31 | INFO | MainProcess | test_thread | [rest_client.start_replication] starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {} 2019-05-22 09:35:31 | INFO | MainProcess | test_thread | [rest_client.start_replication] Replication created with id: 23e43d73c537852f13795005af5e11a0/default/default 2019-05-22 09:35:41 | INFO | MainProcess | test_thread | [rest_client.start_replication] starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C2-C1 with settings {} 2019-05-22 09:35:41 | INFO | MainProcess | test_thread | [rest_client.start_replication] Replication created with id: 47ebe9bde86d837f246d6cb8539c7796/default/default 2019-05-22 09:35:51 | INFO | MainProcess | test_thread | [rest_client.set_xdcr_param] Updated checkpointInterval=60 on bucket'default' on 127.0.0.1 2019-05-22 09:35:51 | INFO | MainProcess | test_thread | [rest_client.set_xdcr_param] Updated checkpointInterval=60 on bucket'default' on 127.0.0.1 2019-05-22 09:35:51 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:35:51 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:35:52 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:35:52 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:35:52 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:35:52 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:35:52 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:35:52 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:35:52 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:35:52 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:36:04 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:36:04 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12026 default 2019-05-22 09:36:05 | INFO | Process-7 | load_gen_task | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:36:05 | INFO | Process-8 | load_gen_task | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:36:05 | INFO | Process-9 | load_gen_task | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:36:05 | INFO | Process-10 | load_gen_task | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:36:05 | INFO | Process-7 | load_gen_task | [data_helper.direct_client] creating direct client 127.0.0.1:12026 default 2019-05-22 09:36:05 | INFO | Process-9 | load_gen_task | [data_helper.direct_client] creating direct client 127.0.0.1:12026 default 2019-05-22 09:36:05 | INFO | Process-10 | load_gen_task | [data_helper.direct_client] creating direct client 127.0.0.1:12026 default 2019-05-22 09:36:05 | INFO | Process-8 | load_gen_task | [data_helper.direct_client] creating direct client 127.0.0.1:12026 default 2019-05-22 09:36:17 | INFO | MainProcess | test_thread | [xdcrnewbasetests.sleep] sleep for 10 secs. ... 2019-05-22 09:36:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__async_swap_rebalance] Starting swap-rebalance [remove_node:127.0.0.1] -> [add_node:127.0.0.1] at C2 cluster 127.0.0.1 2019-05-22 09:36:28 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes_versions] Node versions in cluster [u'0.0.0-0000-community', u'0.0.0-0000-community'] 2019-05-22 09:36:28 | INFO | MainProcess | Cluster_Thread | [task.execute] This is swap rebalance and we will monitor vbuckets shuffling 2019-05-22 09:36:28 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 127.0.0.1:9015 to cluster 2019-05-22 09:36:28 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @127.0.0.1:9015 to this cluster @127.0.0.1:9012 2019-05-22 09:36:31 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'asdasd', 'ejectedNodes': u'n_13@127.0.0.1', 'user': 'Administrator', 'knownNodes': u'n_12@192.168.4.39,n_13@127.0.0.1,n_15@127.0.0.1'} 2019-05-22 09:36:31 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2019-05-22 09:36:31 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling 2019-05-22 09:36:31 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-05-22 09:36:31 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-05-22 09:36:41 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling 2019-05-22 09:36:41 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 23.18 % 2019-05-22 09:36:41 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 23.18% 2019-05-22 09:36:51 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling 2019-05-22 09:36:51 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 51.50 % 2019-05-22 09:36:51 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 51.50% 2019-05-22 09:37:01 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling 2019-05-22 09:37:01 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 79.62 % 2019-05-22 09:37:01 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 79.62% 2019-05-22 09:37:11 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling 2019-05-22 09:37:11 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 100.00 % 2019-05-22 09:37:11 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 100.00% 2019-05-22 09:37:21 | INFO | MainProcess | Cluster_Thread | [task.check] This is swap rebalance and we will monitor vbuckets shuffling 2019-05-22 09:37:21 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2019-05-22 09:37:21 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 50.5685758591 sec 2019-05-22 09:37:21 | INFO | MainProcess | test_thread | [xdcrnewbasetests.perform_update_delete] Updating keys @ C1 2019-05-22 09:37:21 | INFO | MainProcess | test_thread | [xdcrnewbasetests.async_update_delete] At bucket 'default' @ C1: operation: update, key range 0 - 30000 2019-05-22 09:37:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:37:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:37:32 | INFO | MainProcess | test_thread | [xdcrnewbasetests.perform_update_delete] Updating keys @ C2 2019-05-22 09:37:32 | INFO | MainProcess | test_thread | [xdcrnewbasetests.async_update_delete] At bucket 'default' @ C2: operation: update, key range 0 - 30000 2019-05-22 09:37:32 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:37:32 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12030 default 2019-05-22 09:37:42 | INFO | MainProcess | test_thread | [xdcrnewbasetests.perform_update_delete] Deleting keys @ C1 2019-05-22 09:37:42 | INFO | MainProcess | test_thread | [xdcrnewbasetests.async_update_delete] At bucket 'default' @ C1: operation: delete, key range 70000 - 100000 2019-05-22 09:37:42 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:37:43 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:37:50 | INFO | MainProcess | test_thread | [xdcrnewbasetests.perform_update_delete] Deleting keys @ C2 2019-05-22 09:37:50 | INFO | MainProcess | test_thread | [xdcrnewbasetests.async_update_delete] At bucket 'default' @ C2: operation: delete, key range 70000 - 100000 2019-05-22 09:37:50 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:37:51 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12030 default 2019-05-22 09:37:58 | INFO | MainProcess | test_thread | [xdcrnewbasetests.sleep] sleep for 150 secs. ... 2019-05-22 09:40:28 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__merge_all_buckets] Merging keys for replication Replication C1:default -> C2:default 2019-05-22 09:40:29 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__merge_keys] src_kvstore has 70000 valid and 30000 deleted keys 2019-05-22 09:40:29 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__merge_keys] dest kvstore has 70000 valid and 30000 deleted keys 2019-05-22 09:42:01 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__merge_keys] After merging: destination bucket's kv_store now has 140000 valid keys and 30000 deleted keys 2019-05-22 09:42:01 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__merge_all_buckets] Merging keys for replication Replication C2:default -> C1:default 2019-05-22 09:42:02 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__merge_keys] src_kvstore has 140000 valid and 30000 deleted keys 2019-05-22 09:42:02 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__merge_keys] dest kvstore has 70000 valid and 30000 deleted keys 2019-05-22 09:44:38 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__merge_keys] After merging: destination bucket's kv_store now has 140000 valid keys and 30000 deleted keys 2019-05-22 09:44:38 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9010'} 2019-05-22 09:44:38 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:44:38 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9010'}, exp_pager_stime to 10 on default setting param: exp_pager_stime 10 2019-05-22 09:44:38 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9010'}, exp_pager_stime to 10, result: (3519529866, 0, '') 2019-05-22 09:44:38 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9011'} 2019-05-22 09:44:38 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:44:38 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9011'}, exp_pager_stime to 10 on default setting param: exp_pager_stime 10 2019-05-22 09:44:38 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9011'}, exp_pager_stime to 10, result: (4221710194, 0, '') 2019-05-22 09:44:38 | INFO | MainProcess | test_thread | [xdcrnewbasetests.run_expiry_pager] wait for expiry pager to run on all these nodes 2019-05-22 09:44:48 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9012'} 2019-05-22 09:44:48 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:44:48 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9012'}, exp_pager_stime to 10 on default setting param: exp_pager_stime 10 2019-05-22 09:44:48 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9012'}, exp_pager_stime to 10, result: (2577906123, 0, '') 2019-05-22 09:44:48 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9015'} 2019-05-22 09:44:48 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12030 default 2019-05-22 09:44:49 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9015'}, exp_pager_stime to 10 on default setting param: exp_pager_stime 10 2019-05-22 09:44:49 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9015'}, exp_pager_stime to 10, result: (2590200269, 0, '') 2019-05-22 09:44:49 | INFO | MainProcess | test_thread | [xdcrnewbasetests.run_expiry_pager] wait for expiry pager to run on all these nodes 2019-05-22 09:44:59 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12020 default 2019-05-22 09:45:00 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '127.0.0.1:9010',default bucket 2019-05-22 09:45:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:45:00 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '127.0.0.1:9011',default bucket 2019-05-22 09:45:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12024 default 2019-05-22 09:45:00 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '127.0.0.1:9012',default bucket 2019-05-22 09:45:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12030 default 2019-05-22 09:45:00 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '127.0.0.1:9015',default bucket 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_dcp_queue_drain] Waiting for dcp queue to drain on cluster node: 127.0.0.1 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9010/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_dcp_queue_drain] Current dcp queue size on C1 for default is 0 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_dcp_queue_drain] Waiting for dcp queue to drain on cluster node: 127.0.0.1 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9012/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_dcp_queue_drain] Current dcp queue size on C2 for default is 0 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Waiting for Outbound mutation to be zero on cluster node: 127.0.0.1 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Current Outbound mutations on cluster node: C1 for bucket default is 0 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Waiting for Outbound mutation to be zero on cluster node: 127.0.0.1 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Current Outbound mutations on cluster node: C2 for bucket default is 0 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9010/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:45:00 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Saw: vb_active_curr_items 140000 == 140000 expected on C1, default bucket 2019-05-22 09:45:01 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9010/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:45:01 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Saw: vb_replica_curr_items 140000 == 140000 expected on C1, default bucket 2019-05-22 09:45:01 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9012/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:45:01 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Saw: vb_active_curr_items 140000 == 140000 expected on C2, default bucket 2019-05-22 09:45:01 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9012/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:45:01 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Saw: vb_replica_curr_items 140000 == 140000 expected on C2, default bucket 2019-05-22 09:45:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:45:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:45:01 | INFO | MainProcess | test_thread | [task.__init__] 100000 items will be verified on default bucket 2019-05-22 09:45:02 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified 2019-05-22 09:45:05 | INFO | MainProcess | load_gen_task | [task.has_next] 10000 items were verified 2019-05-22 09:45:09 | INFO | MainProcess | load_gen_task | [task.has_next] 20000 items were verified 2019-05-22 09:45:12 | INFO | MainProcess | load_gen_task | [task.has_next] 30000 items were verified 2019-05-22 09:45:16 | INFO | MainProcess | load_gen_task | [task.has_next] 40000 items were verified 2019-05-22 09:45:19 | INFO | MainProcess | load_gen_task | [task.has_next] 50000 items were verified 2019-05-22 09:45:23 | INFO | MainProcess | load_gen_task | [task.has_next] 60000 items were verified 2019-05-22 09:45:26 | INFO | MainProcess | load_gen_task | [task.has_next] 70000 items were verified 2019-05-22 09:45:30 | INFO | MainProcess | load_gen_task | [task.has_next] 80000 items were verified 2019-05-22 09:45:33 | INFO | MainProcess | load_gen_task | [task.has_next] 90000 items were verified 2019-05-22 09:45:37 | INFO | MainProcess | load_gen_task | [task.has_next] 100000 items were verified 2019-05-22 09:45:37 | INFO | MainProcess | load_gen_task | [task.has_next] 100000 items were verified in 35.2342739105 sec.the average number of ops - 2838.14553827 per second 2019-05-22 09:45:37 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:45:37 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12030 default 2019-05-22 09:45:37 | INFO | MainProcess | test_thread | [task.__init__] 100000 items will be verified on default bucket 2019-05-22 09:45:38 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified 2019-05-22 09:45:42 | INFO | MainProcess | load_gen_task | [task.has_next] 10000 items were verified 2019-05-22 09:45:45 | INFO | MainProcess | load_gen_task | [task.has_next] 20000 items were verified 2019-05-22 09:45:48 | INFO | MainProcess | load_gen_task | [task.has_next] 30000 items were verified 2019-05-22 09:45:52 | INFO | MainProcess | load_gen_task | [task.has_next] 40000 items were verified 2019-05-22 09:45:55 | INFO | MainProcess | load_gen_task | [task.has_next] 50000 items were verified 2019-05-22 09:45:59 | INFO | MainProcess | load_gen_task | [task.has_next] 60000 items were verified 2019-05-22 09:46:02 | INFO | MainProcess | load_gen_task | [task.has_next] 70000 items were verified 2019-05-22 09:46:05 | INFO | MainProcess | load_gen_task | [task.has_next] 80000 items were verified 2019-05-22 09:46:09 | INFO | MainProcess | load_gen_task | [task.has_next] 90000 items were verified 2019-05-22 09:46:12 | INFO | MainProcess | load_gen_task | [task.has_next] 100000 items were verified 2019-05-22 09:46:12 | INFO | MainProcess | load_gen_task | [task.has_next] 100000 items were verified in 34.8203139305 sec.the average number of ops - 2871.88672281 per second 2019-05-22 09:46:12 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_rev_ids] Verifying RevIds for Couchbase Cluster: C1, Master Ip: 127.0.0.1 -> Couchbase Cluster: C2, Master Ip: 127.0.0.1, bucket default 2019-05-22 09:46:12 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:46:12 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:46:13 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:46:13 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:46:13 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:46:13 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12030 default 2019-05-22 09:46:14 | INFO | MainProcess | test_thread | [task.__init__] RevID verification: in progress for default ... 2019-05-22 09:46:36 | INFO | MainProcess | load_gen_task | [task.next] 50000 items have been verified 2019-05-22 09:46:58 | INFO | MainProcess | load_gen_task | [task.next] 100000 items have been verified 2019-05-22 09:46:58 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 100000 existing items have been verified 2019-05-22 09:46:58 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 0 deleted items have been verified 2019-05-22 09:46:58 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 0 keys were apparently filtered out and not found in target bucket 2019-05-22 09:46:58 | INFO | MainProcess | load_gen_task | [rest_client.fetch_bucket_stats] http://127.0.0.1:9010/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:46:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9012'} 2019-05-22 09:46:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:46:58 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9012'}, exp_pager_stime to 10 on default setting param: exp_pager_stime 10 2019-05-22 09:46:58 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9012'}, exp_pager_stime to 10, result: (35782398, 0, '') 2019-05-22 09:46:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9015'} 2019-05-22 09:46:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12030 default 2019-05-22 09:46:58 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9015'}, exp_pager_stime to 10 on default setting param: exp_pager_stime 10 2019-05-22 09:46:58 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9015'}, exp_pager_stime to 10, result: (1404406890, 0, '') 2019-05-22 09:46:58 | INFO | MainProcess | test_thread | [xdcrnewbasetests.run_expiry_pager] wait for expiry pager to run on all these nodes 2019-05-22 09:47:08 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9010'} 2019-05-22 09:47:08 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:47:08 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9010'}, exp_pager_stime to 10 on default setting param: exp_pager_stime 10 2019-05-22 09:47:08 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': u'192.168.4.39', 'password': 'asdasd', 'port': u'9010'}, exp_pager_stime to 10, result: (387600776, 0, '') 2019-05-22 09:47:09 | INFO | MainProcess | test_thread | [data_helper.direct_client] dict:{'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9011'} 2019-05-22 09:47:09 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:47:09 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9011'}, exp_pager_stime to 10 on default setting param: exp_pager_stime 10 2019-05-22 09:47:09 | INFO | MainProcess | test_thread | [cluster_helper.flushctl_set_per_node] Setting flush param on server {'username': 'Administrator', 'ip': '127.0.0.1', 'password': 'asdasd', 'port': u'9011'}, exp_pager_stime to 10, result: (2572421391, 0, '') 2019-05-22 09:47:09 | INFO | MainProcess | test_thread | [xdcrnewbasetests.run_expiry_pager] wait for expiry pager to run on all these nodes 2019-05-22 09:47:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12024 default 2019-05-22 09:47:19 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '127.0.0.1:9012',default bucket 2019-05-22 09:47:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12030 default 2019-05-22 09:47:20 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '127.0.0.1:9015',default bucket 2019-05-22 09:47:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12020 default 2019-05-22 09:47:20 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '127.0.0.1:9010',default bucket 2019-05-22 09:47:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:47:20 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '127.0.0.1:9011',default bucket 2019-05-22 09:47:20 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_dcp_queue_drain] Waiting for dcp queue to drain on cluster node: 127.0.0.1 2019-05-22 09:47:20 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9012/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:47:20 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_dcp_queue_drain] Current dcp queue size on C2 for default is 0 2019-05-22 09:47:20 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_dcp_queue_drain] Waiting for dcp queue to drain on cluster node: 127.0.0.1 2019-05-22 09:47:20 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9010/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:47:20 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_dcp_queue_drain] Current dcp queue size on C1 for default is 0 2019-05-22 09:47:20 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Waiting for Outbound mutation to be zero on cluster node: 127.0.0.1 2019-05-22 09:47:20 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Current Outbound mutations on cluster node: C2 for bucket default is 0 2019-05-22 09:47:20 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Waiting for Outbound mutation to be zero on cluster node: 127.0.0.1 2019-05-22 09:47:20 | INFO | MainProcess | test_thread | [xdcrnewbasetests.wait_for_outbound_mutations] Current Outbound mutations on cluster node: C1 for bucket default is 0 2019-05-22 09:47:21 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9012/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:47:21 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Saw: vb_active_curr_items 140000 == 140000 expected on C2, default bucket 2019-05-22 09:47:21 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9012/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:47:21 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Saw: vb_replica_curr_items 140000 == 140000 expected on C2, default bucket 2019-05-22 09:47:21 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9010/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:47:21 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Saw: vb_active_curr_items 140000 == 140000 expected on C1, default bucket 2019-05-22 09:47:21 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://127.0.0.1:9010/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:47:21 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Saw: vb_replica_curr_items 140000 == 140000 expected on C1, default bucket 2019-05-22 09:47:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:47:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12030 default 2019-05-22 09:47:22 | INFO | MainProcess | test_thread | [task.__init__] 100000 items will be verified on default bucket 2019-05-22 09:47:23 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified 2019-05-22 09:47:26 | INFO | MainProcess | load_gen_task | [task.has_next] 10000 items were verified 2019-05-22 09:47:30 | INFO | MainProcess | load_gen_task | [task.has_next] 20000 items were verified 2019-05-22 09:47:33 | INFO | MainProcess | load_gen_task | [task.has_next] 30000 items were verified 2019-05-22 09:47:36 | INFO | MainProcess | load_gen_task | [task.has_next] 40000 items were verified 2019-05-22 09:47:40 | INFO | MainProcess | load_gen_task | [task.has_next] 50000 items were verified 2019-05-22 09:47:43 | INFO | MainProcess | load_gen_task | [task.has_next] 60000 items were verified 2019-05-22 09:47:47 | INFO | MainProcess | load_gen_task | [task.has_next] 70000 items were verified 2019-05-22 09:47:50 | INFO | MainProcess | load_gen_task | [task.has_next] 80000 items were verified 2019-05-22 09:47:54 | INFO | MainProcess | load_gen_task | [task.has_next] 90000 items were verified 2019-05-22 09:47:57 | INFO | MainProcess | load_gen_task | [task.has_next] 100000 items were verified 2019-05-22 09:47:57 | INFO | MainProcess | load_gen_task | [task.has_next] 100000 items were verified in 35.4357869625 sec.the average number of ops - 2822.00582746 per second 2019-05-22 09:47:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:47:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:47:58 | INFO | MainProcess | test_thread | [task.__init__] 100000 items will be verified on default bucket 2019-05-22 09:47:58 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified 2019-05-22 09:48:02 | INFO | MainProcess | load_gen_task | [task.has_next] 10000 items were verified 2019-05-22 09:48:06 | INFO | MainProcess | load_gen_task | [task.has_next] 20000 items were verified 2019-05-22 09:48:09 | INFO | MainProcess | load_gen_task | [task.has_next] 30000 items were verified 2019-05-22 09:48:13 | INFO | MainProcess | load_gen_task | [task.has_next] 40000 items were verified 2019-05-22 09:48:16 | INFO | MainProcess | load_gen_task | [task.has_next] 50000 items were verified 2019-05-22 09:48:20 | INFO | MainProcess | load_gen_task | [task.has_next] 60000 items were verified 2019-05-22 09:48:23 | INFO | MainProcess | load_gen_task | [task.has_next] 70000 items were verified 2019-05-22 09:48:27 | INFO | MainProcess | load_gen_task | [task.has_next] 80000 items were verified 2019-05-22 09:48:30 | INFO | MainProcess | load_gen_task | [task.has_next] 90000 items were verified 2019-05-22 09:48:34 | INFO | MainProcess | load_gen_task | [task.has_next] 100000 items were verified 2019-05-22 09:48:34 | INFO | MainProcess | load_gen_task | [task.has_next] 100000 items were verified in 35.9989500046 sec.the average number of ops - 2777.85872472 per second 2019-05-22 09:48:34 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_rev_ids] Verifying RevIds for Couchbase Cluster: C2, Master Ip: 127.0.0.1 -> Couchbase Cluster: C1, Master Ip: 127.0.0.1, bucket default 2019-05-22 09:48:34 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:48:34 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12030 default 2019-05-22 09:48:34 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12024 default 2019-05-22 09:48:34 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12030 default 2019-05-22 09:48:35 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 192.168.4.39:12020 default 2019-05-22 09:48:35 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12022 default 2019-05-22 09:48:35 | INFO | MainProcess | test_thread | [task.__init__] RevID verification: in progress for default ... 2019-05-22 09:49:00 | INFO | MainProcess | load_gen_task | [task.next] 50000 items have been verified 2019-05-22 09:49:23 | INFO | MainProcess | load_gen_task | [task.next] 100000 items have been verified 2019-05-22 09:49:23 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 100000 existing items have been verified 2019-05-22 09:49:23 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 0 deleted items have been verified 2019-05-22 09:49:23 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 0 keys were apparently filtered out and not found in target bucket 2019-05-22 09:49:23 | INFO | MainProcess | load_gen_task | [rest_client.fetch_bucket_stats] http://127.0.0.1:9012/pools/default/buckets/default/stats?zoom=minute 2019-05-22 09:49:23 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9010: True content: "/home/jim/couchbase/ns_server/logs/n_10" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:23 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:23 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_10"/goxdcr.log* 2019-05-22 09:49:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:23 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:23 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 2019-05-22 09:49:23 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] Initial replication restart count on 127.0.0.1 :0, now :0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9011: True content: "/home/jim/couchbase/ns_server/logs/n_11" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_11"/goxdcr.log* 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 249 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 2019-05-22T09:38:31.571+01:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 23e43d73c537852f13795005af5e11a0/default/default. Current error(s)=r.update_err_ch : xmem_23e43d73c537852f13795005af5e11a0/default/default_127.0.0.1:12026_1 : Xmem is stuck 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] Initial replication restart count on 127.0.0.1 :0, now :249 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] Number of new replication restarts this run: 249 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] - 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] - 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] T 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] + 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] I 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] N 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] F 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] O 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] G 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] O 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] X 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] D 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] C 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] R 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] P 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] M 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] g 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] T 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] y 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] x 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] P 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] C 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] ( 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] ) 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] = 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] _ 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] _ 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] h 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] x 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] m 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] m 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] _ 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] _ 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] _ 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] X 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] m 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] m 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] k 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9012: True content: "/home/jim/couchbase/ns_server/logs/n_12" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_12"/goxdcr.log* 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] Initial replication restart count on 127.0.0.1 :0, now :0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9013: True content: "/home/jim/couchbase/ns_server/logs/n_13" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_13"/goxdcr.log* 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 221 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 2019-05-22T09:36:41.767+01:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 47ebe9bde86d837f246d6cb8539c7796/default/default. Current error(s)=r.update_err_ch : TopoChangeDet : dcp stream for vb=905 is closed by producer 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] Initial replication restart count on 127.0.0.1 :0, now :221 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] Number of new replication restarts this run: 221 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] - 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] - 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] T 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] + 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] I 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] N 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] F 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] O 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] G 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] O 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] X 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] D 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] C 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] R 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] P 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] M 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] g 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] T 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] y 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] x 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] P 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] C 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] ( 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] ) 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] = 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] _ 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] _ 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] h 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] T 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] C 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] h 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] g 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] D 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] m 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] v 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] = 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] y 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9014: True content: "/home/jim/couchbase/ns_server/logs/n_14" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_14"/goxdcr.log* 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] Initial replication restart count on 127.0.0.1 :0, now :0 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9015: True content: "/home/jim/couchbase/ns_server/logs/n_15" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "Try to fix Pipeline" "/home/jim/couchbase/ns_server/logs/n_15"/goxdcr.log* 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 588 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 2019-05-22T09:36:41.169+01:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 47ebe9bde86d837f246d6cb8539c7796/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(47ebe9bde86d837f246d6cb8539c7796/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. 2019-05-22T09:41:31.401+01:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 47ebe9bde86d837f246d6cb8539c7796/default/default. Current error(s)=r.update_err_ch : TopoChangeDet : Restarting pipeline due to source topology change... 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] Initial replication restart count on 127.0.0.1 :0, now :588 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] Number of new replication restarts this run: 588 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] - 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] - 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] T 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] + 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] I 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] N 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] F 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] O 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] G 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] O 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] X 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] D 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] C 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] R 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] P 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] M 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] g 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] T 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] y 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] x 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] P 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] C 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] ( 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] ) 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] = 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] M 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] g 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] _ 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] y 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] N 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] w 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] P 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] ( 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] ) 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] I 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] v 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] g 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] N 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] z 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] z 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] h 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] k 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] v 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] h 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] m 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] y 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] v 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] k 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] - 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] - 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] T 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] + 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] I 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] N 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] F 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] O 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] G 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] O 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] X 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] D 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] C 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] R 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] P 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] M 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] g 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] T 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] y 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] x 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] P 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 4 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] b 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 8 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 5 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 3 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 7 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 9 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 6 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] / 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] f 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] C 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] ( 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] ) 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] = 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] _ 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] _ 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] h 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] T 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] C 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] h 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] g 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] D 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] : 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] R 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] g 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] i 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] d 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] s 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] u 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] r 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] t 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] p 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] l 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] o 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] g 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] y 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] c 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] h 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] a 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] n 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] g 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] e 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] . 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_replication_restarted] 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9010: True content: "/home/jim/couchbase/ns_server/logs/n_10" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_10"/goxdcr.log* 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial panic: count on 127.0.0.1 :0, now :0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_10"/goxdcr.log* 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 127.0.0.1 :0, now :0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9011: True content: "/home/jim/couchbase/ns_server/logs/n_11" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_11"/goxdcr.log* 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial panic: count on 127.0.0.1 :0, now :0 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_11"/goxdcr.log* 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 127.0.0.1 :0, now :0 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9012: True content: "/home/jim/couchbase/ns_server/logs/n_12" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_12"/goxdcr.log* 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial panic: count on 127.0.0.1 :0, now :0 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_12"/goxdcr.log* 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 127.0.0.1 :0, now :0 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9013: True content: "/home/jim/couchbase/ns_server/logs/n_13" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_13"/goxdcr.log* 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial panic: count on 127.0.0.1 :0, now :0 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_13"/goxdcr.log* 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 127.0.0.1 :0, now :0 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9014: True content: "/home/jim/couchbase/ns_server/logs/n_14" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_14"/goxdcr.log* 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial panic: count on 127.0.0.1 :0, now :0 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_14"/goxdcr.log* 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 127.0.0.1 :0, now :0 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9015: True content: "/home/jim/couchbase/ns_server/logs/n_15" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic:" "/home/jim/couchbase/ns_server/logs/n_15"/goxdcr.log* 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial panic: count on 127.0.0.1 :0, now :0 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/home/jim/couchbase/ns_server/logs/n_15"/goxdcr.log* 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_goxdcr_log] 0 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.check_errors_in_goxdcr_logs] Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 127.0.0.1 :0, now :0 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.tearDown] ==== XDCRNewbasetests cleanup is started for test #1 swap_rebalance ==== 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] removing xdcr/nodes settings 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [rest_client.stop_replication] Deleting replication /controller/cancelXDCR/23e43d73c537852f13795005af5e11a0%2Fdefault%2Fdefault 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [rest_client.stop_replication] Replication deleted successfully 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [rest_client.remove_remote_cluster] removing remote cluster name:remote_cluster_C1-C2 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] cleanup [ip:127.0.0.1 port:9010 ssh_username:Administrator, ip:127.0.0.1 port:9011 ssh_username:Administrator] 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 127.0.0.1 2019-05-22 09:49:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ... 2019-05-22 09:49:29 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1 2019-05-22 09:49:29 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2019-05-22 09:49:29 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 127.0.0.1 existing buckets : [] 2019-05-22 09:49:29 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] sleep 2 seconds to make sure all buckets ([u'default']) were deleted completely. 2019-05-22 09:49:31 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2019-05-22 09:49:31 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'password': 'asdasd', 'ejectedNodes': u'n_11@127.0.0.1', 'user': 'Administrator', 'knownNodes': u'n_11@127.0.0.1,n_10@192.168.4.39'} 2019-05-22 09:49:31 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2019-05-22 09:49:31 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-05-22 09:49:51 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 20.04 seconds 2019-05-22 09:49:51 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2019-05-22 09:50:01 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:127.0.0.1 port:9010 ssh_username:Administrator ? [(u'n_11@127.0.0.1', 9011)] 2019-05-22 09:50:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9010 2019-05-22 09:50:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9010 is running 2019-05-22 09:50:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9011 2019-05-22 09:50:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9011 is running Cluster instance shutdown with force 2019-05-22 09:50:01 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] removing xdcr/nodes settings 2019-05-22 09:50:01 | INFO | MainProcess | test_thread | [rest_client.remove_remote_cluster] removing remote cluster name:remote_cluster_C2-C1 2019-05-22 09:50:01 | INFO | MainProcess | test_thread | [xdcrnewbasetests.cleanup_cluster] cleanup [ip:127.0.0.1 port:9012 ssh_username:Administrator, ip:127.0.0.1 port:9015 ssh_username:Administrator] 2019-05-22 09:50:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 127.0.0.1 2019-05-22 09:50:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ... 2019-05-22 09:50:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1 2019-05-22 09:50:02 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2019-05-22 09:50:02 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 127.0.0.1 existing buckets : [] 2019-05-22 09:50:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] sleep 2 seconds to make sure all buckets ([u'default']) were deleted completely. 2019-05-22 09:50:04 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2019-05-22 09:50:04 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'password': 'asdasd', 'ejectedNodes': u'n_15@127.0.0.1', 'user': 'Administrator', 'knownNodes': u'n_12@192.168.4.39,n_15@127.0.0.1'} 2019-05-22 09:50:04 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2019-05-22 09:50:04 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-05-22 09:50:24 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 20.03 seconds 2019-05-22 09:50:24 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2019-05-22 09:50:34 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:127.0.0.1 port:9012 ssh_username:Administrator ? [(u'n_15@127.0.0.1', 9015)] 2019-05-22 09:50:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9012 2019-05-22 09:50:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9012 is running 2019-05-22 09:50:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9015 2019-05-22 09:50:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9015 is running Cluster instance shutdown with force 2019-05-22 09:50:35 | INFO | MainProcess | test_thread | [xdcrnewbasetests.tearDown] ==== XDCRNewbasetests cleanup is finished for test #1 swap_rebalance === Cluster instance shutdown with force ok ---------------------------------------------------------------------- Ran 1 test in 942.054s OK summary so far suite xdcr.rebalanceXDCR.Rebalance , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jim/couchbase/testrunner/logs/testrunner-19-May-22_09-34-52/test_1 Run after suite setup for xdcr.rebalanceXDCR.Rebalance.swap_rebalance Thread Cluster_Thread was not properly terminated, will be terminated now. Thread Cluster_Thread was not properly terminated, will be terminated now.