./testrunner -i /tmp/testexec.24289.ini stop-on-failure=False,fail_on_errors=1 -t xdcr.uniXDCR.unidirectional.load_with_async_ops_with_warmup,items=100000,rdirection=unidirection,ctopology=chain,delete=C1,warm=C1-C2,GROUP=CHAIN;P1

Test Input params:
{'fail_on_errors': '1', 'GROUP': 'CHAIN;P1', 'items': '100000', 'case_number': 7, 'conf_file': 'py-xdcr-unidirectional-1.conf', 'num_nodes': 6, 'cluster_name': 'testexec.24289', 'ctopology': 'chain', 'warm': 'C1-C2', 'stop-on-failure': 'False', 'ini': '/tmp/testexec.24289.ini', 'rdirection': 'unidirection', 'logs_folder': '/data/workspace/centos-p0-xdcr-vset00-00-unidirectional-1/logs/testrunner-17-Nov-04_09-26-07/test_7', 'spec': 'py-xdcr-unidirectional-1', 'delete': 'C1'}
[2017-11-04 10:40:15,611] - [rest_client:1430] INFO - /diag/eval status on 172.23.107.130:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2017-11-04 10:40:15,612] - [remote_util:198] INFO - connecting to 172.23.107.130 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:15,709] - [remote_util:232] INFO - Connected to 172.23.107.130
[2017-11-04 10:40:16,127] - [remote_util:2822] INFO - running command.raw on 172.23.107.130: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:16,230] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:16,230] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:16,346] - [remote_util:198] INFO - connecting to 172.23.107.130 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:16,443] - [remote_util:232] INFO - Connected to 172.23.107.130
[2017-11-04 10:40:16,816] - [remote_util:2822] INFO - running command.raw on 172.23.107.130: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:16,908] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:16,909] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:16,912] - [remote_util:198] INFO - connecting to 172.23.107.134 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:17,079] - [remote_util:232] INFO - Connected to 172.23.107.134
[2017-11-04 10:40:17,598] - [remote_util:2822] INFO - running command.raw on 172.23.107.134: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:17,738] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:17,738] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:17,854] - [remote_util:198] INFO - connecting to 172.23.107.134 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:18,022] - [remote_util:232] INFO - Connected to 172.23.107.134
[2017-11-04 10:40:18,530] - [remote_util:2822] INFO - running command.raw on 172.23.107.134: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:18,662] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:18,662] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:18,778] - [remote_util:198] INFO - connecting to 172.23.107.138 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:18,875] - [remote_util:232] INFO - Connected to 172.23.107.138
[2017-11-04 10:40:19,277] - [remote_util:2822] INFO - running command.raw on 172.23.107.138: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:19,360] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:19,360] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:19,475] - [remote_util:198] INFO - connecting to 172.23.107.138 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:19,572] - [remote_util:232] INFO - Connected to 172.23.107.138
[2017-11-04 10:40:19,970] - [remote_util:2822] INFO - running command.raw on 172.23.107.138: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:20,055] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:20,055] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:20,171] - [remote_util:198] INFO - connecting to 172.23.107.20 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:20,269] - [remote_util:232] INFO - Connected to 172.23.107.20
[2017-11-04 10:40:20,839] - [remote_util:2822] INFO - running command.raw on 172.23.107.20: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:20,952] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:20,952] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:21,067] - [remote_util:198] INFO - connecting to 172.23.107.20 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:21,233] - [remote_util:232] INFO - Connected to 172.23.107.20
[2017-11-04 10:40:21,751] - [remote_util:2822] INFO - running command.raw on 172.23.107.20: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:21,787] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:21,788] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:21,903] - [remote_util:198] INFO - connecting to 172.23.107.201 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:22,039] - [remote_util:232] INFO - Connected to 172.23.107.201
[2017-11-04 10:40:22,686] - [remote_util:2822] INFO - running command.raw on 172.23.107.201: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:22,778] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:22,779] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:22,782] - [remote_util:198] INFO - connecting to 172.23.107.201 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:22,917] - [remote_util:232] INFO - Connected to 172.23.107.201
[2017-11-04 10:40:23,640] - [remote_util:2822] INFO - running command.raw on 172.23.107.201: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:23,729] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:23,730] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:23,846] - [remote_util:198] INFO - connecting to 172.23.107.205 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:23,944] - [remote_util:232] INFO - Connected to 172.23.107.205
[2017-11-04 10:40:24,443] - [remote_util:2822] INFO - running command.raw on 172.23.107.205: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:24,530] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:24,530] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:24,646] - [remote_util:198] INFO - connecting to 172.23.107.205 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:24,743] - [remote_util:232] INFO - Connected to 172.23.107.205
[2017-11-04 10:40:25,179] - [remote_util:2822] INFO - running command.raw on 172.23.107.205: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:25,201] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:25,201] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:25,315] - [xdcrnewbasetests:2700] INFO - {'172.23.107.138': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.134': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.130': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.20': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.201': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.205': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}}
[2017-11-04 10:40:25,323] - [rest_client:1430] INFO - /diag/eval status on 172.23.107.130:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2017-11-04 10:40:25,325] - [remote_util:198] INFO - connecting to 172.23.107.130 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:25,421] - [remote_util:232] INFO - Connected to 172.23.107.130
[2017-11-04 10:40:25,797] - [remote_util:2822] INFO - running command.raw on 172.23.107.130: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:25,891] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:25,891] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:26,007] - [remote_util:198] INFO - connecting to 172.23.107.134 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:26,176] - [remote_util:232] INFO - Connected to 172.23.107.134
[2017-11-04 10:40:26,671] - [remote_util:2822] INFO - running command.raw on 172.23.107.134: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:26,738] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:26,739] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:26,742] - [remote_util:198] INFO - connecting to 172.23.107.138 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:26,839] - [remote_util:232] INFO - Connected to 172.23.107.138
[2017-11-04 10:40:27,218] - [remote_util:2822] INFO - running command.raw on 172.23.107.138: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:27,301] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:27,302] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:27,417] - [remote_util:198] INFO - connecting to 172.23.107.20 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:27,548] - [remote_util:232] INFO - Connected to 172.23.107.20
[2017-11-04 10:40:28,156] - [remote_util:2822] INFO - running command.raw on 172.23.107.20: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:28,216] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:28,216] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:28,332] - [remote_util:198] INFO - connecting to 172.23.107.201 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:28,466] - [remote_util:232] INFO - Connected to 172.23.107.201
[2017-11-04 10:40:29,089] - [remote_util:2822] INFO - running command.raw on 172.23.107.201: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:29,178] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:29,178] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:29,294] - [remote_util:198] INFO - connecting to 172.23.107.205 with username:root password:couchbase ssh_key:
[2017-11-04 10:40:29,393] - [remote_util:232] INFO - Connected to 172.23.107.205
[2017-11-04 10:40:29,787] - [remote_util:2822] INFO - running command.raw on 172.23.107.205: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:40:29,874] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:40:29,875] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:40:29,989] - [xdcrnewbasetests:2718] INFO - {'172.23.107.138': 0, '172.23.107.134': 0, '172.23.107.130': 0, '172.23.107.20': 0, '172.23.107.201': 0, '172.23.107.205': 0}
[2017-11-04 10:40:29,989] - [xdcrnewbasetests:2481] INFO - ==== XDCRNewbasetests setup is started for test #7 load_with_async_ops_with_warmup ====
[2017-11-04 10:40:29,990] - [xdcrnewbasetests:1178] INFO - removing xdcr/nodes settings
[2017-11-04 10:40:30,012] - [xdcrnewbasetests:1188] INFO - cleanup [ip:172.23.107.130 port:8091 ssh_username:root, ip:172.23.107.134 port:8091 ssh_username:root]
[2017-11-04 10:40:30,019] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.107.130
[2017-11-04 10:40:30,020] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2017-11-04 10:40:32,042] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.130:8091
[2017-11-04 10:40:32,046] - [cluster_helper:80] INFO - ns_server @ 172.23.107.130:8091 is running
[2017-11-04 10:40:32,058] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.107.134
[2017-11-04 10:40:32,058] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2017-11-04 10:40:34,085] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.134:8091
[2017-11-04 10:40:34,090] - [cluster_helper:80] INFO - ns_server @ 172.23.107.134:8091 is running
[2017-11-04 10:40:34,090] - [xdcrnewbasetests:1178] INFO - removing xdcr/nodes settings
[2017-11-04 10:40:34,121] - [xdcrnewbasetests:1188] INFO - cleanup [ip:172.23.107.138 port:8091 ssh_username:root, ip:172.23.107.20 port:8091 ssh_username:root]
[2017-11-04 10:40:34,128] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.107.138
[2017-11-04 10:40:34,128] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2017-11-04 10:40:36,160] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.138:8091
[2017-11-04 10:40:36,164] - [cluster_helper:80] INFO - ns_server @ 172.23.107.138:8091 is running
[2017-11-04 10:40:36,179] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.107.20
[2017-11-04 10:40:36,179] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2017-11-04 10:40:38,243] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.20:8091
[2017-11-04 10:40:38,247] - [cluster_helper:80] INFO - ns_server @ 172.23.107.20:8091 is running
[2017-11-04 10:40:38,247] - [xdcrnewbasetests:2728] INFO - Initializing all clusters...
[2017-11-04 10:40:38,254] - [rest_client:1430] INFO - /diag/eval status on 172.23.107.130:8091: True content: [5,1] command: cluster_compat_mode:get_compat_version().
[2017-11-04 10:40:39,008] - [task:125] INFO - server: ip:172.23.107.130 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.130', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.130', 'uptime': u'4529', 'mcdMemoryReserved': 3104, 'storageTotalRam': 3880, 'hostname': u'172.23.107.130:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f34e5a62390>], 'moxi': 11211, 'port': u'8091', 'version': u'5.1.0-1368-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327681, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3425546240, 'memoryTotal': 4069449728, 'memoryQuota': 2069, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2017-11-04 10:40:39,008] - [rest_client:902] INFO - pools/default params : memoryQuota=2069
[2017-11-04 10:40:39,013] - [rest_client:936] INFO - settings/indexes params : storageMode=forestdb
[2017-11-04 10:40:39,016] - [rest_client:803] ERROR - POST http://172.23.107.130:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in 5.0 enterprise edition."}} auth: Administrator:password
[2017-11-04 10:40:39,017] - [rest_client:821] INFO - settings/web params on 172.23.107.130:8091:username=Administrator&password=password&port=8091
[2017-11-04 10:40:39,114] - [task:125] INFO - server: ip:172.23.107.134 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.134', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.134', 'uptime': u'1255', 'mcdMemoryReserved': 3104, 'storageTotalRam': 3880, 'hostname': u'172.23.107.134:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f34e5a62550>], 'moxi': 11211, 'port': u'8091', 'version': u'5.1.0-1368-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327681, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3686547456, 'memoryTotal': 4069318656, 'memoryQuota': 2328, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2017-11-04 10:40:39,114] - [rest_client:902] INFO - pools/default params : memoryQuota=2069
[2017-11-04 10:40:39,120] - [rest_client:936] INFO - settings/indexes params : storageMode=forestdb
[2017-11-04 10:40:39,125] - [rest_client:803] ERROR - POST http://172.23.107.134:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in 5.0 enterprise edition."}} auth: Administrator:password
[2017-11-04 10:40:39,125] - [rest_client:821] INFO - settings/web params on 172.23.107.134:8091:username=Administrator&password=password&port=8091
[2017-11-04 10:40:40,316] - [task:425] INFO - adding node 172.23.107.134:8091 to cluster
[2017-11-04 10:40:40,317] - [rest_client:1194] INFO - adding remote node @172.23.107.134:8091 to this cluster @172.23.107.130:8091
[2017-11-04 10:40:44,428] - [rest_client:1412] INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.107.130,ns_1@172.23.107.134'}
[2017-11-04 10:40:44,432] - [rest_client:1417] INFO - rebalance operation started
[2017-11-04 10:40:44,439] - [rest_client:1565] INFO - rebalance percentage : 0.00 %
[2017-11-04 10:40:44,439] - [task:491] INFO - Rebalance - status: running, progress: 0
[2017-11-04 10:40:54,459] - [task:491] INFO - Rebalance - status: none, progress: 100
[2017-11-04 10:40:54,464] - [task:550] INFO - rebalancing was completed with progress: 100% in 10.0321240425 sec
[2017-11-04 10:40:54,475] - [rest_client:1430] INFO - /diag/eval status on 172.23.107.138:8091: True content: [5,1] command: cluster_compat_mode:get_compat_version().
[2017-11-04 10:40:55,026] - [task:125] INFO - server: ip:172.23.107.138 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.138', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.138', 'uptime': u'4543', 'mcdMemoryReserved': 3104, 'storageTotalRam': 3880, 'hostname': u'172.23.107.138:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f34d7d1b890>], 'moxi': 11211, 'port': u'8091', 'version': u'5.1.0-1368-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327681, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3579723776, 'memoryTotal': 4069322752, 'memoryQuota': 2069, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2017-11-04 10:40:55,027] - [rest_client:902] INFO - pools/default params : memoryQuota=2069
[2017-11-04 10:40:55,032] - [rest_client:936] INFO - settings/indexes params : storageMode=forestdb
[2017-11-04 10:40:55,035] - [rest_client:803] ERROR - POST http://172.23.107.138:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in 5.0 enterprise edition."}} auth: Administrator:password
[2017-11-04 10:40:55,035] - [rest_client:821] INFO - settings/web params on 172.23.107.138:8091:username=Administrator&password=password&port=8091
[2017-11-04 10:40:55,140] - [task:125] INFO - server: ip:172.23.107.20 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.20', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.20', 'uptime': u'551', 'mcdMemoryReserved': 3104, 'storageTotalRam': 3881, 'hostname': u'172.23.107.20:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f34d7d1b0d0>], 'moxi': 11211, 'port': u'8091', 'version': u'5.1.0-1368-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327681, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3706232832, 'memoryTotal': 4069695488, 'memoryQuota': 2328, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2017-11-04 10:40:55,140] - [rest_client:902] INFO - pools/default params : memoryQuota=2069
[2017-11-04 10:40:55,147] - [rest_client:936] INFO - settings/indexes params : storageMode=forestdb
[2017-11-04 10:40:55,158] - [rest_client:803] ERROR - POST http://172.23.107.20:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in 5.0 enterprise edition."}} auth: Administrator:password
[2017-11-04 10:40:55,158] - [rest_client:821] INFO - settings/web params on 172.23.107.20:8091:username=Administrator&password=password&port=8091
[2017-11-04 10:40:56,361] - [task:425] INFO - adding node 172.23.107.20:8091 to cluster
[2017-11-04 10:40:56,361] - [rest_client:1194] INFO - adding remote node @172.23.107.20:8091 to this cluster @172.23.107.138:8091
[2017-11-04 10:41:01,883] - [rest_client:1412] INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.107.138,ns_1@172.23.107.20'}
[2017-11-04 10:41:01,888] - [rest_client:1417] INFO - rebalance operation started
[2017-11-04 10:41:01,896] - [rest_client:1565] INFO - rebalance percentage : 0.00 %
[2017-11-04 10:41:01,896] - [task:491] INFO - Rebalance - status: running, progress: 0
[2017-11-04 10:41:11,917] - [task:491] INFO - Rebalance - status: none, progress: 100
[2017-11-04 10:41:11,922] - [task:550] INFO - rebalancing was completed with progress: 100% in 10.0343170166 sec
[2017-11-04 10:41:11,932] - [rest_client:1951] INFO - Node versions in cluster [u'5.1.0-1368-enterprise', u'5.1.0-1368-enterprise']
[2017-11-04 10:41:11,942] - [rest_client:803] ERROR - DELETE http://172.23.107.130:8091/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password
[2017-11-04 10:41:11,943] - [internal_user:36] INFO - Exception while deleting user. Exception is -"User was not found."
[2017-11-04 10:41:22,052] - [rest_client:1951] INFO - Node versions in cluster [u'5.1.0-1368-enterprise', u'5.1.0-1368-enterprise']
[2017-11-04 10:41:32,078] - [rest_client:1951] INFO - Node versions in cluster [u'5.1.0-1368-enterprise', u'5.1.0-1368-enterprise']
[2017-11-04 10:41:32,085] - [rest_client:803] ERROR - DELETE http://172.23.107.138:8091/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password
[2017-11-04 10:41:32,086] - [internal_user:36] INFO - Exception while deleting user. Exception is -"User was not found."
[2017-11-04 10:41:42,221] - [rest_client:1951] INFO - Node versions in cluster [u'5.1.0-1368-enterprise', u'5.1.0-1368-enterprise']
[2017-11-04 10:41:52,555] - [rest_client:1951] INFO - Node versions in cluster [u'5.1.0-1368-enterprise', u'5.1.0-1368-enterprise']
[2017-11-04 10:41:52,556] - [rest_client:2191] INFO - http://172.23.107.130:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=sasl&replicaIndex=1&name=default&saslPassword=None&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=1569
[2017-11-04 10:41:52,564] - [rest_client:2216] INFO - 0.01 seconds to create bucket default
[2017-11-04 10:41:52,564] - [bucket_helper:342] INFO - waiting for memcached bucket : default in 172.23.107.130 to accept set ops
[2017-11-04 10:41:53,732] - [data_helper:295] INFO - creating direct client 172.23.107.130:11210 default
[2017-11-04 10:41:53,812] - [data_helper:295] INFO - creating direct client 172.23.107.134:11210 default
[2017-11-04 10:41:53,928] - [data_helper:295] INFO - creating direct client 172.23.107.130:11210 default
[2017-11-04 10:41:53,992] - [data_helper:295] INFO - creating direct client 172.23.107.134:11210 default
[2017-11-04 10:41:54,884] - [task:312] INFO - bucket 'default' was created with per node RAM quota: 1569
[2017-11-04 10:41:54,992] - [rest_client:1951] INFO - Node versions in cluster [u'5.1.0-1368-enterprise', u'5.1.0-1368-enterprise']
[2017-11-04 10:41:54,992] - [rest_client:2191] INFO - http://172.23.107.138:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=sasl&replicaIndex=1&name=default&saslPassword=None&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=1569
[2017-11-04 10:41:55,001] - [rest_client:2216] INFO - 0.01 seconds to create bucket default
[2017-11-04 10:41:55,001] - [bucket_helper:342] INFO - waiting for memcached bucket : default in 172.23.107.138 to accept set ops
[2017-11-04 10:41:56,168] - [data_helper:295] INFO - creating direct client 172.23.107.138:11210 default
[2017-11-04 10:41:56,253] - [data_helper:295] INFO - creating direct client 172.23.107.20:11210 default
[2017-11-04 10:41:56,389] - [data_helper:295] INFO - creating direct client 172.23.107.138:11210 default
[2017-11-04 10:41:56,459] - [data_helper:295] INFO - creating direct client 172.23.107.20:11210 default
[2017-11-04 10:41:57,422] - [task:312] INFO - bucket 'default' was created with per node RAM quota: 1569
[2017-11-04 10:41:57,453] - [xdcrnewbasetests:2487] INFO - ==== XDCRNewbasetests setup is finished for test #7 load_with_async_ops_with_warmup ====
[2017-11-04 10:41:57,457] - [rest_client:1084] INFO - adding remote cluster hostname:172.23.107.138:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.107.130:8091
[2017-11-04 10:41:57,477] - [rest_client:1131] INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {}
[2017-11-04 10:41:57,531] - [rest_client:1144] INFO - Replication created with id: 42dafee52670db2df4aa02d785b2d2b5/default/default
[2017-11-04 10:42:07,559] - [rest_client:2498] INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.107.130
[2017-11-04 10:42:07,610] - [data_helper:295] INFO - creating direct client 172.23.107.130:11210 default
[2017-11-04 10:42:07,687] - [data_helper:295] INFO - creating direct client 172.23.107.134:11210 default
[2017-11-04 10:42:38,958] - [xdcrnewbasetests:3463] INFO - sleep for 10 secs.  ...
[2017-11-04 10:42:48,971] - [remote_util:198] INFO - connecting to 172.23.107.134 with username:root password:couchbase ssh_key:
[2017-11-04 10:42:49,138] - [remote_util:232] INFO - Connected to 172.23.107.134
[2017-11-04 10:42:49,842] - [remote_util:2822] INFO - running command.raw on 172.23.107.134: /etc/init.d/couchbase-server stop
[2017-11-04 10:42:54,287] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:42:54,288] - [remote_util:2657] INFO - Stopping couchbase-server
[2017-11-04 10:42:54,288] - [remote_util:2657] INFO - [60G[[0;32m  OK  [0;39m]
[2017-11-04 10:42:54,288] - [remote_util:2657] INFO - 
[2017-11-04 10:42:59,405] - [remote_util:2822] INFO - running command.raw on 172.23.107.134: /etc/init.d/couchbase-server start
[2017-11-04 10:43:02,309] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:43:02,310] - [remote_util:2657] INFO - Starting couchbase-server
[2017-11-04 10:43:02,310] - [remote_util:2657] INFO - [  OK  ]
[2017-11-04 10:43:02,426] - [remote_util:198] INFO - connecting to 172.23.107.20 with username:root password:couchbase ssh_key:
[2017-11-04 10:43:02,556] - [remote_util:232] INFO - Connected to 172.23.107.20
[2017-11-04 10:43:03,345] - [remote_util:2822] INFO - running command.raw on 172.23.107.20: /etc/init.d/couchbase-server stop
[2017-11-04 10:43:06,738] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:43:06,738] - [remote_util:2657] INFO - Stopping couchbase-server
[2017-11-04 10:43:06,739] - [remote_util:2657] INFO - [60G[[0;32m  OK  [0;39m]
[2017-11-04 10:43:06,739] - [remote_util:2657] INFO - 
[2017-11-04 10:43:11,879] - [remote_util:2822] INFO - running command.raw on 172.23.107.20: /etc/init.d/couchbase-server start
[2017-11-04 10:43:14,788] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:43:14,788] - [remote_util:2657] INFO - Starting couchbase-server
[2017-11-04 10:43:14,788] - [remote_util:2657] INFO - [  OK  ]
[2017-11-04 10:43:14,790] - [xdcrnewbasetests:3463] INFO - sleep for 60 secs.  ...
[2017-11-04 10:44:14,850] - [xdcrnewbasetests:3131] INFO - Deleting keys @ C1
[2017-11-04 10:44:14,851] - [xdcrnewbasetests:1670] INFO - At bucket 'default' @ C1: operation: delete, key range 70000 - 100000
[2017-11-04 10:44:14,890] - [data_helper:295] INFO - creating direct client 172.23.107.130:11210 default
[2017-11-04 10:44:14,990] - [data_helper:295] INFO - creating direct client 172.23.107.134:11210 default
[2017-11-04 10:44:34,253] - [xdcrnewbasetests:3139] INFO - Batched deletes sent to cluster(s)
[2017-11-04 10:44:34,254] - [xdcrnewbasetests:3463] INFO - sleep for 30 secs.  ...
[2017-11-04 10:45:04,301] - [data_helper:295] INFO - creating direct client 172.23.107.134:11210 default
[2017-11-04 10:45:04,388] - [xdcrnewbasetests:325] INFO - Warmed up: 100000 items on default on ip:172.23.107.134 port:8091 ssh_username:root
[2017-11-04 10:45:14,426] - [data_helper:295] INFO - creating direct client 172.23.107.20:11210 default
[2017-11-04 10:45:14,546] - [xdcrnewbasetests:325] INFO - Warmed up: 100000 items on default on ip:172.23.107.20 port:8091 ssh_username:root
[2017-11-04 10:45:24,565] - [xdcrnewbasetests:3305] INFO - Merging keys for replication Replication C1:default -> C2:default
[2017-11-04 10:45:24,670] - [xdcrnewbasetests:3244] INFO - src_kvstore has 70000 valid and 30000 deleted keys
[2017-11-04 10:45:24,670] - [xdcrnewbasetests:3246] INFO - dest kvstore has 0 valid and 0 deleted keys
[2017-11-04 10:45:25,509] - [xdcrnewbasetests:3295] INFO - After merging: destination bucket's kv_store now has 70000 valid keys and 0 deleted keys
[2017-11-04 10:45:25,542] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.130', 'password': 'password', 'port': u'8091'}
[2017-11-04 10:45:25,542] - [data_helper:293] INFO - creating direct client 172.23.107.130:11210 default
[2017-11-04 10:45:25,593] - [cluster_helper:340] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.130', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2017-11-04 10:45:25,593] - [cluster_helper:354] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.130', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (845507116, 0, '')
[2017-11-04 10:45:25,621] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.134', 'password': 'password', 'port': u'8091'}
[2017-11-04 10:45:25,621] - [data_helper:293] INFO - creating direct client 172.23.107.134:11210 default
[2017-11-04 10:45:25,695] - [cluster_helper:340] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.134', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2017-11-04 10:45:25,696] - [cluster_helper:354] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.134', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3302684078, 0, '')
[2017-11-04 10:45:25,696] - [xdcrnewbasetests:1712] INFO - wait for expiry pager to run on all these nodes
[2017-11-04 10:45:35,738] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.138', 'password': 'password', 'port': u'8091'}
[2017-11-04 10:45:35,738] - [data_helper:293] INFO - creating direct client 172.23.107.138:11210 default
[2017-11-04 10:45:35,783] - [cluster_helper:340] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.138', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2017-11-04 10:45:35,784] - [cluster_helper:354] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.138', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3653764233, 0, '')
[2017-11-04 10:45:35,814] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.20', 'password': 'password', 'port': u'8091'}
[2017-11-04 10:45:35,814] - [data_helper:293] INFO - creating direct client 172.23.107.20:11210 default
[2017-11-04 10:45:35,907] - [cluster_helper:340] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.20', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
setting param: exp_pager_stime 10
[2017-11-04 10:45:35,908] - [cluster_helper:354] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.20', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1399745694, 0, '')
[2017-11-04 10:45:35,908] - [xdcrnewbasetests:1712] INFO - wait for expiry pager to run on all these nodes
[2017-11-04 10:45:46,263] - [data_helper:295] INFO - creating direct client 172.23.107.130:11210 default
[2017-11-04 10:45:46,317] - [task:602] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.130:8091',default bucket
[2017-11-04 10:45:46,330] - [data_helper:295] INFO - creating direct client 172.23.107.134:11210 default
[2017-11-04 10:45:46,415] - [task:602] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.134:8091',default bucket
[2017-11-04 10:45:46,726] - [data_helper:295] INFO - creating direct client 172.23.107.138:11210 default
[2017-11-04 10:45:46,782] - [task:602] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.138:8091',default bucket
[2017-11-04 10:45:46,803] - [data_helper:295] INFO - creating direct client 172.23.107.20:11210 default
[2017-11-04 10:45:46,961] - [task:602] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.20:8091',default bucket
[2017-11-04 10:45:46,984] - [xdcrnewbasetests:2362] INFO - Waiting for dcp queue to drain on cluster node: 172.23.107.130
[2017-11-04 10:45:46,988] - [rest_client:1856] INFO - http://172.23.107.130:8091/pools/default/buckets/default/stats?zoom=minute
[2017-11-04 10:45:47,026] - [xdcrnewbasetests:2372] INFO - Current dcp queue size on C1 for default is 0
[2017-11-04 10:45:47,026] - [xdcrnewbasetests:2362] INFO - Waiting for dcp queue to drain on cluster node: 172.23.107.138
[2017-11-04 10:45:47,033] - [rest_client:1856] INFO - http://172.23.107.138:8091/pools/default/buckets/default/stats?zoom=minute
[2017-11-04 10:45:47,080] - [xdcrnewbasetests:2372] INFO - Current dcp queue size on C2 for default is 0
[2017-11-04 10:45:47,080] - [xdcrnewbasetests:2392] INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.107.130
[2017-11-04 10:45:47,108] - [xdcrnewbasetests:2406] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0
[2017-11-04 10:45:47,108] - [xdcrnewbasetests:2392] INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.107.138
[2017-11-04 10:45:47,116] - [xdcrnewbasetests:2402] ERROR - Stat "replication_changes_left" not found
[2017-11-04 10:45:47,167] - [rest_client:1856] INFO - http://172.23.107.130:8091/pools/default/buckets/default/stats?zoom=minute
[2017-11-04 10:45:47,203] - [xdcrnewbasetests:2263] INFO - Saw: vb_active_curr_items 70000 == 70000 expected on C1, default bucket
[2017-11-04 10:45:47,264] - [rest_client:1856] INFO - http://172.23.107.130:8091/pools/default/buckets/default/stats?zoom=minute
[2017-11-04 10:45:47,304] - [xdcrnewbasetests:2301] INFO - Saw: vb_replica_curr_items 70000 == 70000 expected on C1, default bucket
[2017-11-04 10:45:47,352] - [rest_client:1856] INFO - http://172.23.107.138:8091/pools/default/buckets/default/stats?zoom=minute
[2017-11-04 10:45:47,416] - [xdcrnewbasetests:2263] INFO - Saw: vb_active_curr_items 70000 == 70000 expected on C2, default bucket
[2017-11-04 10:45:47,463] - [rest_client:1856] INFO - http://172.23.107.138:8091/pools/default/buckets/default/stats?zoom=minute
[2017-11-04 10:45:47,505] - [xdcrnewbasetests:2301] INFO - Saw: vb_replica_curr_items 70000 == 70000 expected on C2, default bucket
[2017-11-04 10:45:47,544] - [data_helper:295] INFO - creating direct client 172.23.107.130:11210 default
[2017-11-04 10:45:47,600] - [data_helper:295] INFO - creating direct client 172.23.107.134:11210 default
[2017-11-04 10:45:47,813] - [task:1756] INFO - 100000 items will be verified on default bucket
[2017-11-04 10:45:48,419] - [task:1765] INFO - 0 items were verified
[2017-11-04 10:45:51,452] - [task:1765] INFO - 10000 items were verified
[2017-11-04 10:45:54,513] - [task:1765] INFO - 20000 items were verified
[2017-11-04 10:45:57,601] - [task:1765] INFO - 30000 items were verified
[2017-11-04 10:46:00,727] - [task:1765] INFO - 40000 items were verified
[2017-11-04 10:46:03,792] - [task:1765] INFO - 50000 items were verified
[2017-11-04 10:46:06,796] - [task:1765] INFO - 60000 items were verified
[2017-11-04 10:46:09,861] - [task:1765] INFO - 70000 items were verified
[2017-11-04 10:46:13,047] - [task:1765] INFO - 80000 items were verified
[2017-11-04 10:46:16,311] - [task:1765] INFO - 90000 items were verified
[2017-11-04 10:46:19,533] - [task:1765] INFO - 100000 items were verified
[2017-11-04 10:46:19,533] - [task:1769] INFO - 100000 items were verified in 31.7192680836 sec.the average number of ops                - 3152.65777035 per second
[2017-11-04 10:46:19,592] - [data_helper:295] INFO - creating direct client 172.23.107.138:11210 default
[2017-11-04 10:46:19,657] - [data_helper:295] INFO - creating direct client 172.23.107.20:11210 default
[2017-11-04 10:46:19,844] - [task:1756] INFO - 70000 items will be verified on default bucket
[2017-11-04 10:46:20,012] - [task:1765] INFO - 0 items were verified
[2017-11-04 10:46:23,010] - [task:1765] INFO - 10000 items were verified
[2017-11-04 10:46:26,047] - [task:1765] INFO - 20000 items were verified
[2017-11-04 10:46:29,016] - [task:1765] INFO - 30000 items were verified
[2017-11-04 10:46:32,010] - [task:1765] INFO - 40000 items were verified
[2017-11-04 10:46:35,050] - [task:1765] INFO - 50000 items were verified
[2017-11-04 10:46:38,043] - [task:1765] INFO - 60000 items were verified
[2017-11-04 10:46:41,082] - [task:1765] INFO - 70000 items were verified
[2017-11-04 10:46:41,083] - [task:1769] INFO - 70000 items were verified in 21.2377099991 sec.the average number of ops                - 3296.02372732 per second
[2017-11-04 10:46:41,083] - [xdcrnewbasetests:3210] INFO - Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.107.130 -> Couchbase Cluster: C2, Master Ip: 172.23.107.138, bucket default
[2017-11-04 10:46:41,138] - [data_helper:295] INFO - creating direct client 172.23.107.130:11210 default
[2017-11-04 10:46:41,206] - [data_helper:295] INFO - creating direct client 172.23.107.134:11210 default
[2017-11-04 10:46:41,434] - [data_helper:295] INFO - creating direct client 172.23.107.130:11210 default
[2017-11-04 10:46:41,492] - [data_helper:295] INFO - creating direct client 172.23.107.134:11210 default
[2017-11-04 10:46:41,607] - [data_helper:295] INFO - creating direct client 172.23.107.138:11210 default
[2017-11-04 10:46:41,672] - [data_helper:295] INFO - creating direct client 172.23.107.20:11210 default
[2017-11-04 10:46:41,988] - [task:1869] INFO - RevID verification: in progress for default ...
[2017-11-04 10:47:26,761] - [task:1907] INFO - 50000 items have been verified
[2017-11-04 10:48:26,132] - [task:1907] INFO - 100000 items have been verified
[2017-11-04 10:48:26,132] - [task:1877] INFO - RevId Verification : 70000 existing items have been verified
[2017-11-04 10:48:26,132] - [task:1879] INFO - RevId Verification : 30000 deleted items have been verified
[2017-11-04 10:48:26,132] - [task:1882] INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket
[2017-11-04 10:48:26,137] - [rest_client:1856] INFO - http://172.23.107.130:8091/pools/default/buckets/default/stats?zoom=minute
[2017-11-04 10:48:26,233] - [rest_client:1430] INFO - /diag/eval status on 172.23.107.130:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2017-11-04 10:48:26,235] - [remote_util:198] INFO - connecting to 172.23.107.130 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:26,331] - [remote_util:232] INFO - Connected to 172.23.107.130
[2017-11-04 10:48:26,742] - [remote_util:2822] INFO - running command.raw on 172.23.107.130: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log*
[2017-11-04 10:48:26,835] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:26,836] - [remote_util:2822] INFO - running command.raw on 172.23.107.130: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:26,932] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:26,932] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:26,934] - [xdcrnewbasetests:3377] INFO - Initial replication restart count on 172.23.107.130 :0, now :0
[2017-11-04 10:48:26,936] - [remote_util:198] INFO - connecting to 172.23.107.134 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:27,106] - [remote_util:232] INFO - Connected to 172.23.107.134
[2017-11-04 10:48:27,649] - [remote_util:2822] INFO - running command.raw on 172.23.107.134: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log*
[2017-11-04 10:48:27,778] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:27,778] - [xdcrnewbasetests:419] INFO - ['2017-11-04T10:43:24.252-07:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 42dafee52670db2df4aa02d785b2d2b5/default/default. Current error=dial tcp 172.23.107.20:11210: getsockopt: connection refused ']
[2017-11-04 10:48:27,778] - [remote_util:2822] INFO - running command.raw on 172.23.107.134: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:27,894] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:27,894] - [xdcrnewbasetests:427] INFO - 1
[2017-11-04 10:48:28,008] - [xdcrnewbasetests:3377] INFO - Initial replication restart count on 172.23.107.134 :0, now :1
[2017-11-04 10:48:28,008] - [xdcrnewbasetests:3383] INFO - Number of new replication restarts this run: 1
[2017-11-04 10:48:28,008] - [xdcrnewbasetests:3385] INFO - 2017-11-04T10:43:24.252-07:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 42dafee52670db2df4aa02d785b2d2b5/default/default. Current error=dial tcp 172.23.107.20:11210: getsockopt: connection refused 
[2017-11-04 10:48:28,010] - [remote_util:198] INFO - connecting to 172.23.107.138 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:28,107] - [remote_util:232] INFO - Connected to 172.23.107.138
[2017-11-04 10:48:28,597] - [remote_util:2822] INFO - running command.raw on 172.23.107.138: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log*
[2017-11-04 10:48:28,682] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:28,682] - [remote_util:2822] INFO - running command.raw on 172.23.107.138: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:28,768] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:28,769] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:28,882] - [xdcrnewbasetests:3377] INFO - Initial replication restart count on 172.23.107.138 :0, now :0
[2017-11-04 10:48:28,884] - [remote_util:198] INFO - connecting to 172.23.107.20 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:29,014] - [remote_util:232] INFO - Connected to 172.23.107.20
[2017-11-04 10:48:29,640] - [remote_util:2822] INFO - running command.raw on 172.23.107.20: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log*
[2017-11-04 10:48:29,729] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:29,730] - [remote_util:2822] INFO - running command.raw on 172.23.107.20: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:29,852] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:29,852] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:29,966] - [xdcrnewbasetests:3377] INFO - Initial replication restart count on 172.23.107.20 :0, now :0
[2017-11-04 10:48:29,967] - [remote_util:198] INFO - connecting to 172.23.107.201 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:30,102] - [remote_util:232] INFO - Connected to 172.23.107.201
[2017-11-04 10:48:30,747] - [remote_util:2822] INFO - running command.raw on 172.23.107.201: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log*
[2017-11-04 10:48:30,774] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:30,774] - [remote_util:2822] INFO - running command.raw on 172.23.107.201: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:30,845] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:30,845] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:30,847] - [xdcrnewbasetests:3377] INFO - Initial replication restart count on 172.23.107.201 :0, now :0
[2017-11-04 10:48:30,848] - [remote_util:198] INFO - connecting to 172.23.107.205 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:30,947] - [remote_util:232] INFO - Connected to 172.23.107.205
[2017-11-04 10:48:31,411] - [remote_util:2822] INFO - running command.raw on 172.23.107.205: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log*
[2017-11-04 10:48:31,495] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:31,496] - [remote_util:2822] INFO - running command.raw on 172.23.107.205: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:31,581] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:31,581] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:31,695] - [xdcrnewbasetests:3377] INFO - Initial replication restart count on 172.23.107.205 :0, now :0
[2017-11-04 10:48:31,705] - [rest_client:1430] INFO - /diag/eval status on 172.23.107.130:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2017-11-04 10:48:31,706] - [remote_util:198] INFO - connecting to 172.23.107.130 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:31,803] - [remote_util:232] INFO - Connected to 172.23.107.130
[2017-11-04 10:48:32,182] - [remote_util:2822] INFO - running command.raw on 172.23.107.130: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:32,282] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:32,283] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:32,397] - [xdcrnewbasetests:3347] INFO - Initial panic: count on 172.23.107.130 :0, now :0
[2017-11-04 10:48:32,398] - [remote_util:198] INFO - connecting to 172.23.107.130 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:32,495] - [remote_util:232] INFO - Connected to 172.23.107.130
[2017-11-04 10:48:32,918] - [remote_util:2822] INFO - running command.raw on 172.23.107.130: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:33,013] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:33,013] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:33,127] - [xdcrnewbasetests:3347] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.130 :0, now :0
[2017-11-04 10:48:33,130] - [remote_util:198] INFO - connecting to 172.23.107.134 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:33,298] - [remote_util:232] INFO - Connected to 172.23.107.134
[2017-11-04 10:48:33,803] - [remote_util:2822] INFO - running command.raw on 172.23.107.134: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:33,887] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:33,887] - [xdcrnewbasetests:427] INFO - 1
[2017-11-04 10:48:34,002] - [xdcrnewbasetests:3347] INFO - Initial panic: count on 172.23.107.134 :0, now :1
[2017-11-04 10:48:34,003] - [remote_util:198] INFO - connecting to 172.23.107.134 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:34,171] - [remote_util:232] INFO - Connected to 172.23.107.134
[2017-11-04 10:48:34,704] - [rest_client:1430] INFO - /diag/eval status on 172.23.107.134:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2017-11-04 10:48:34,704] - [remote_util:2822] INFO - running command.raw on 172.23.107.134: zgrep -A 40 'panic:' "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log*
[2017-11-04 10:48:34,807] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:34,807] - [xdcrnewbasetests:3323] INFO - panic: runtime error: invalid memory address or nil pointer dereference
[2017-11-04 10:48:34,807] - [xdcrnewbasetests:3323] INFO - [signal SIGSEGV: segmentation violation code=0x1 addr=0x88 pc=0x7cef46]
[2017-11-04 10:48:34,807] - [xdcrnewbasetests:3323] INFO - 
[2017-11-04 10:48:34,808] - [xdcrnewbasetests:3323] INFO - goroutine 311 [running]:
[2017-11-04 10:48:34,808] - [xdcrnewbasetests:3323] INFO - panic(0x94d000, 0xc80430)
[2017-11-04 10:48:34,808] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/panic.go:531 +0x1cf fp=0xc420216dc8 sp=0xc420216d30
[2017-11-04 10:48:34,808] - [xdcrnewbasetests:3323] INFO - runtime.panicmem()
[2017-11-04 10:48:34,808] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/panic.go:63 +0x5e fp=0xc420216de8 sp=0xc420216dc8
[2017-11-04 10:48:34,808] - [xdcrnewbasetests:3323] INFO - runtime.sigpanic()
[2017-11-04 10:48:34,808] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/signal_unix.go:290 +0x29f fp=0xc420216e38 sp=0xc420216de8
[2017-11-04 10:48:34,808] - [xdcrnewbasetests:3323] INFO - github.com/couchbase/goxdcr/parts.(*DcpNozzle).Stop(0xc420135600, 0xc4208c9aa0, 0x58f0a3)
[2017-11-04 10:48:34,808] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/parts/dcp_nozzle.go:536 +0x166 fp=0xc420216f70 sp=0xc420216e38
[2017-11-04 10:48:34,809] - [xdcrnewbasetests:3323] INFO - github.com/couchbase/goxdcr/common.(Part).Stop-fm(0x4339f8, 0xa15430)
[2017-11-04 10:48:34,809] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/pipeline/generic_pipeline.go:305 +0x2f fp=0xc420216f98 sp=0xc420216f70
[2017-11-04 10:48:34,809] - [xdcrnewbasetests:3323] INFO - github.com/couchbase/goxdcr/simple_utils.ExecWithTimeout.func1(0xc420930d20, 0xc420890ba0)
[2017-11-04 10:48:34,809] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/simple_utils/simple_utils.go:37 +0x27 fp=0xc420216fd0 sp=0xc420216f98
[2017-11-04 10:48:34,809] - [xdcrnewbasetests:3323] INFO - runtime.goexit()
[2017-11-04 10:48:34,809] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420216fd8 sp=0xc420216fd0
[2017-11-04 10:48:34,809] - [xdcrnewbasetests:3323] INFO - created by github.com/couchbase/goxdcr/simple_utils.ExecWithTimeout
[2017-11-04 10:48:34,809] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/simple_utils/simple_utils.go:39 +0x94
[2017-11-04 10:48:34,809] - [xdcrnewbasetests:3323] INFO - 
[2017-11-04 10:48:34,809] - [xdcrnewbasetests:3323] INFO - goroutine 1 [chan receive]:
[2017-11-04 10:48:34,809] - [xdcrnewbasetests:3323] INFO - runtime.gopark(0xa15580, 0xc42017a178, 0x9ec2fc, 0xc, 0xc5d717, 0x3)
[2017-11-04 10:48:34,810] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/proc.go:271 +0x13a fp=0xc42083bc68 sp=0xc42083bc38
[2017-11-04 10:48:34,810] - [xdcrnewbasetests:3323] INFO - runtime.goparkunlock(0xc42017a178, 0x9ec2fc, 0xc, 0xc42007cf17, 0x3)
[2017-11-04 10:48:34,810] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/proc.go:277 +0x5e fp=0xc42083bca8 sp=0xc42083bc68
[2017-11-04 10:48:34,810] - [xdcrnewbasetests:3323] INFO - runtime.chanrecv(0x918de0, 0xc42017a120, 0x0, 0x1, 0x270e)
[2017-11-04 10:48:34,810] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/chan.go:513 +0x371 fp=0xc42083bd48 sp=0xc42083bca8
[2017-11-04 10:48:34,810] - [xdcrnewbasetests:3323] INFO - runtime.chanrecv1(0x918de0, 0xc42017a120, 0x0)
[2017-11-04 10:48:34,810] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/chan.go:395 +0x35 fp=0xc42083bd80 sp=0xc42083bd48
[2017-11-04 10:48:34,810] - [xdcrnewbasetests:3323] INFO - main.main()
[2017-11-04 10:48:34,810] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/main/main.go:176 +0xd17 fp=0xc42083bf88 sp=0xc42083bd80
[2017-11-04 10:48:34,810] - [xdcrnewbasetests:3323] INFO - runtime.main()
[2017-11-04 10:48:34,810] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/proc.go:185 +0x20a fp=0xc42083bfe0 sp=0xc42083bf88
[2017-11-04 10:48:34,811] - [xdcrnewbasetests:3323] INFO - runtime.goexit()
[2017-11-04 10:48:34,811] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42083bfe8 sp=0xc42083bfe0
[2017-11-04 10:48:34,811] - [xdcrnewbasetests:3323] INFO - 
[2017-11-04 10:48:34,811] - [xdcrnewbasetests:3323] INFO - goroutine 17 [syscall, locked to thread]:
[2017-11-04 10:48:34,811] - [xdcrnewbasetests:3323] INFO - runtime.goexit()
[2017-11-04 10:48:34,811] - [xdcrnewbasetests:3323] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42003cfe8 sp=0xc42003cfe0
[2017-11-04 10:48:34,811] - [xdcrnewbasetests:3323] INFO - 
[2017-11-04 10:48:34,927] - [remote_util:198] INFO - connecting to 172.23.107.134 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:35,057] - [remote_util:232] INFO - Connected to 172.23.107.134
[2017-11-04 10:48:35,553] - [remote_util:2822] INFO - running command.raw on 172.23.107.134: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:35,616] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:35,617] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:35,731] - [xdcrnewbasetests:3347] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.134 :0, now :0
[2017-11-04 10:48:35,732] - [remote_util:198] INFO - connecting to 172.23.107.138 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:35,830] - [remote_util:232] INFO - Connected to 172.23.107.138
[2017-11-04 10:48:36,238] - [remote_util:2822] INFO - running command.raw on 172.23.107.138: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:36,258] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:36,258] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:36,372] - [xdcrnewbasetests:3347] INFO - Initial panic: count on 172.23.107.138 :0, now :0
[2017-11-04 10:48:36,374] - [remote_util:198] INFO - connecting to 172.23.107.138 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:36,470] - [remote_util:232] INFO - Connected to 172.23.107.138
[2017-11-04 10:48:36,877] - [remote_util:2822] INFO - running command.raw on 172.23.107.138: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:36,962] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:36,963] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:37,077] - [xdcrnewbasetests:3347] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.138 :0, now :0
[2017-11-04 10:48:37,079] - [remote_util:198] INFO - connecting to 172.23.107.20 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:37,213] - [remote_util:232] INFO - Connected to 172.23.107.20
[2017-11-04 10:48:37,784] - [remote_util:2822] INFO - running command.raw on 172.23.107.20: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:37,836] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:37,836] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:37,838] - [xdcrnewbasetests:3347] INFO - Initial panic: count on 172.23.107.20 :0, now :0
[2017-11-04 10:48:37,839] - [remote_util:198] INFO - connecting to 172.23.107.20 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:37,973] - [remote_util:232] INFO - Connected to 172.23.107.20
[2017-11-04 10:48:38,632] - [remote_util:2822] INFO - running command.raw on 172.23.107.20: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:38,692] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:38,693] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:38,694] - [xdcrnewbasetests:3347] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.20 :0, now :0
[2017-11-04 10:48:38,696] - [remote_util:198] INFO - connecting to 172.23.107.201 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:38,827] - [remote_util:232] INFO - Connected to 172.23.107.201
[2017-11-04 10:48:39,451] - [remote_util:2822] INFO - running command.raw on 172.23.107.201: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:39,544] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:39,544] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:39,545] - [xdcrnewbasetests:3347] INFO - Initial panic: count on 172.23.107.201 :0, now :0
[2017-11-04 10:48:39,547] - [remote_util:198] INFO - connecting to 172.23.107.201 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:39,680] - [remote_util:232] INFO - Connected to 172.23.107.201
[2017-11-04 10:48:40,406] - [remote_util:2822] INFO - running command.raw on 172.23.107.201: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:40,495] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:40,495] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:40,497] - [xdcrnewbasetests:3347] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.201 :0, now :0
[2017-11-04 10:48:40,498] - [remote_util:198] INFO - connecting to 172.23.107.205 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:40,595] - [remote_util:232] INFO - Connected to 172.23.107.205
[2017-11-04 10:48:40,995] - [remote_util:2822] INFO - running command.raw on 172.23.107.205: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:41,082] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:41,082] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:41,196] - [xdcrnewbasetests:3347] INFO - Initial panic: count on 172.23.107.205 :0, now :0
[2017-11-04 10:48:41,198] - [remote_util:198] INFO - connecting to 172.23.107.205 with username:root password:couchbase ssh_key:
[2017-11-04 10:48:41,294] - [remote_util:232] INFO - Connected to 172.23.107.205
[2017-11-04 10:48:41,720] - [remote_util:2822] INFO - running command.raw on 172.23.107.205: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l
[2017-11-04 10:48:41,806] - [remote_util:2859] INFO - command executed successfully
[2017-11-04 10:48:41,807] - [xdcrnewbasetests:427] INFO - 0
[2017-11-04 10:48:41,808] - [xdcrnewbasetests:3347] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.205 :0, now :0
[2017-11-04 10:48:41,808] - [xdcrnewbasetests:3354] ERROR - ['panic: found on 172.23.107.134']
FAIL
[2017-11-04 10:48:41,820] - [rest_client:1951] INFO - Node versions in cluster [u'5.1.0-1368-enterprise', u'5.1.0-1368-enterprise']
[2017-11-04 10:48:41,840] - [rest_client:1951] INFO - Node versions in cluster [u'5.1.0-1368-enterprise', u'5.1.0-1368-enterprise']
[2017-11-04 10:48:41,850] - [xdcrnewbasetests:2539] INFO - ====  XDCRNewbasetests cleanup is started for test #7 load_with_async_ops_with_warmup ====
[2017-11-04 10:48:41,850] - [xdcrnewbasetests:1178] INFO - removing xdcr/nodes settings
[2017-11-04 10:48:41,857] - [rest_client:1167] INFO - Deleting replication /controller/cancelXDCR/42dafee52670db2df4aa02d785b2d2b5%2Fdefault%2Fdefault
[2017-11-04 10:48:41,876] - [rest_client:1115] INFO - removing remote cluster name:remote_cluster_C1-C2
[2017-11-04 10:48:41,896] - [xdcrnewbasetests:1188] INFO - cleanup [ip:172.23.107.130 port:8091 ssh_username:root, ip:172.23.107.134 port:8091 ssh_username:root]
[2017-11-04 10:48:41,913] - [bucket_helper:142] INFO - deleting existing buckets [u'default'] on 172.23.107.130
[2017-11-04 10:48:41,914] - [bucket_helper:144] INFO - remove bucket default ...
[2017-11-04 10:48:43,568] - [bucket_helper:158] INFO - deleted bucket : default from 172.23.107.130
[2017-11-04 10:48:43,568] - [bucket_helper:236] INFO - waiting for bucket deletion to complete....
[2017-11-04 10:48:43,571] - [rest_client:133] INFO - node 172.23.107.130 existing buckets : []
[2017-11-04 10:48:43,572] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2017-11-04 10:48:45,594] - [cluster_helper:254] INFO - rebalancing all nodes in order to remove nodes
[2017-11-04 10:48:45,597] - [rest_client:1412] INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.107.134', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.107.130,ns_1@172.23.107.134'}
[2017-11-04 10:48:45,606] - [rest_client:1417] INFO - rebalance operation started
[2017-11-04 10:48:45,616] - [rest_client:1565] INFO - rebalance percentage : 0.00 %
[2017-11-04 10:49:05,644] - [rest_client:1504] INFO - rebalance progress took 20.0373511314 seconds 
[2017-11-04 10:49:05,644] - [rest_client:1505] INFO - sleep for 10 seconds after rebalance...
[2017-11-04 10:49:15,668] - [cluster_helper:294] INFO - removed all the nodes from cluster associated with ip:172.23.107.130 port:8091 ssh_username:root ? [(u'ns_1@172.23.107.134', 8091)]
[2017-11-04 10:49:15,672] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.130:8091
[2017-11-04 10:49:15,675] - [cluster_helper:80] INFO - ns_server @ 172.23.107.130:8091 is running
[2017-11-04 10:49:15,686] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.107.134
[2017-11-04 10:49:15,686] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2017-11-04 10:49:17,711] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.134:8091
[2017-11-04 10:49:17,717] - [cluster_helper:80] INFO - ns_server @ 172.23.107.134:8091 is running
Cluster instance shutdown with force
[2017-11-04 10:49:17,718] - [xdcrnewbasetests:1178] INFO - removing xdcr/nodes settings
[2017-11-04 10:49:17,752] - [xdcrnewbasetests:1188] INFO - cleanup [ip:172.23.107.138 port:8091 ssh_username:root, ip:172.23.107.20 port:8091 ssh_username:root]
[2017-11-04 10:49:17,768] - [bucket_helper:142] INFO - deleting existing buckets [u'default'] on 172.23.107.138
[2017-11-04 10:49:17,769] - [bucket_helper:144] INFO - remove bucket default ...
[2017-11-04 10:49:18,908] - [bucket_helper:158] INFO - deleted bucket : default from 172.23.107.138
[2017-11-04 10:49:18,909] - [bucket_helper:236] INFO - waiting for bucket deletion to complete....
[2017-11-04 10:49:18,913] - [rest_client:133] INFO - node 172.23.107.138 existing buckets : []
[2017-11-04 10:49:18,913] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2017-11-04 10:49:20,933] - [cluster_helper:254] INFO - rebalancing all nodes in order to remove nodes
[2017-11-04 10:49:20,936] - [rest_client:1412] INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.107.20', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.107.138,ns_1@172.23.107.20'}
[2017-11-04 10:49:20,943] - [rest_client:1417] INFO - rebalance operation started
[2017-11-04 10:49:20,954] - [rest_client:1565] INFO - rebalance percentage : 0.00 %
[2017-11-04 10:49:40,985] - [rest_client:1504] INFO - rebalance progress took 20.0416419506 seconds 
[2017-11-04 10:49:40,985] - [rest_client:1505] INFO - sleep for 10 seconds after rebalance...
[2017-11-04 10:49:51,009] - [cluster_helper:294] INFO - removed all the nodes from cluster associated with ip:172.23.107.138 port:8091 ssh_username:root ? [(u'ns_1@172.23.107.20', 8091)]
[2017-11-04 10:49:51,014] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.138:8091
[2017-11-04 10:49:51,018] - [cluster_helper:80] INFO - ns_server @ 172.23.107.138:8091 is running
[2017-11-04 10:49:51,030] - [bucket_helper:142] INFO - deleting existing buckets [] on 172.23.107.20
[2017-11-04 10:49:51,030] - [bucket_helper:168] INFO - sleep 2 seconds to make sure all buckets were deleted completely.
[2017-11-04 10:49:53,058] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.20:8091
[2017-11-04 10:49:53,067] - [cluster_helper:80] INFO - ns_server @ 172.23.107.20:8091 is running
Cluster instance shutdown with force
[2017-11-04 10:49:53,067] - [xdcrnewbasetests:2544] INFO - ====  XDCRNewbasetests cleanup is finished for test #7 load_with_async_ops_with_warmup ===

======================================================================
FAIL: load_with_async_ops_with_warmup (xdcr.uniXDCR.unidirectional)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/xdcr/uniXDCR.py", line 105, in load_with_async_ops_with_warmup
    self.verify_results()
  File "pytests/xdcr/xdcrnewbasetests.py", line 3542, in verify_results
    self.fail("Errors found in logs : {0}".format(error_logger))
AssertionError: Errors found in logs : ['panic: found on 172.23.107.134']

----------------------------------------------------------------------
Ran 1 test in 577.464s

FAILED (failures=1)
load_with_async_ops_with_warmup_master (xdcr.uniXDCR.unidirectional) ... Cluster instance shutdown with force
summary so far suite xdcr.uniXDCR.unidirectional , pass 6 , fail 1
failures so far...
xdcr.uniXDCR.unidirectional.load_with_async_ops_with_warmup
testrunner logs, diags and results are available under /data/workspace/centos-p0-xdcr-vset00-00-unidirectional-1/logs/testrunner-17-Nov-04_09-26-07/test_7