[2018-11-14 16:35:02,260] - [rest_client] [140700069775104] - ERROR - POST http://172.23.104.104:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password [2018-11-14 16:35:02,260] - [rest_client] [140700069775104] - INFO - /diag/eval status on 172.23.104.104:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2018-11-14 16:35:02,265] - [remote_util] [140700069775104] - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: [2018-11-14 16:35:02,407] - [remote_util] [140700069775104] - INFO - Connected to 172.23.104.104 [2018-11-14 16:35:02,705] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:03,215] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.104.104: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:03,347] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:03,462] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:03,467] - [remote_util] [140700069775104] - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: [2018-11-14 16:35:03,611] - [remote_util] [140700069775104] - INFO - Connected to 172.23.104.104 [2018-11-14 16:35:03,901] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:04,386] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.104.104: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:04,516] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:04,631] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:04,636] - [remote_util] [140700069775104] - INFO - connecting to 172.23.107.161 with username:root password:couchbase ssh_key: [2018-11-14 16:35:04,773] - [remote_util] [140700069775104] - INFO - Connected to 172.23.107.161 [2018-11-14 16:35:05,349] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.107.161: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:05,481] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:05,596] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:05,600] - [remote_util] [140700069775104] - INFO - connecting to 172.23.107.161 with username:root password:couchbase ssh_key: [2018-11-14 16:35:05,738] - [remote_util] [140700069775104] - INFO - Connected to 172.23.107.161 [2018-11-14 16:35:06,305] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.107.161: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:06,455] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:06,570] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:06,576] - [remote_util] [140700069775104] - INFO - connecting to 172.23.108.236 with username:root password:couchbase ssh_key: [2018-11-14 16:35:06,743] - [remote_util] [140700069775104] - INFO - Connected to 172.23.108.236 [2018-11-14 16:35:06,972] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:07,479] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.108.236: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:07,610] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:07,725] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:07,729] - [remote_util] [140700069775104] - INFO - connecting to 172.23.108.236 with username:root password:couchbase ssh_key: [2018-11-14 16:35:07,872] - [remote_util] [140700069775104] - INFO - Connected to 172.23.108.236 [2018-11-14 16:35:08,129] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:08,543] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.108.236: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:08,673] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:08,788] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:08,793] - [remote_util] [140700069775104] - INFO - connecting to 172.23.122.201 with username:root password:couchbase ssh_key: [2018-11-14 16:35:09,062] - [remote_util] [140700069775104] - INFO - Connected to 172.23.122.201 [2018-11-14 16:35:09,532] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:10,072] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.122.201: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:10,254] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:10,369] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:10,373] - [remote_util] [140700069775104] - INFO - connecting to 172.23.122.201 with username:root password:couchbase ssh_key: [2018-11-14 16:35:10,649] - [remote_util] [140700069775104] - INFO - Connected to 172.23.122.201 [2018-11-14 16:35:11,059] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:11,603] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.122.201: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:11,785] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:11,900] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:11,906] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.224 with username:root password:couchbase ssh_key: [2018-11-14 16:35:12,013] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.224 [2018-11-14 16:35:12,230] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:12,687] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.224: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:12,785] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:12,900] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:12,905] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.224 with username:root password:couchbase ssh_key: [2018-11-14 16:35:13,007] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.224 [2018-11-14 16:35:13,227] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:13,607] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.224: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:13,706] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:13,820] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:13,826] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.225 with username:root password:couchbase ssh_key: [2018-11-14 16:35:13,935] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.225 [2018-11-14 16:35:14,129] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:14,531] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.225: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:14,629] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:14,743] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:14,748] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.225 with username:root password:couchbase ssh_key: [2018-11-14 16:35:14,853] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.225 [2018-11-14 16:35:15,050] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:15,487] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.225: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:15,585] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:15,700] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:15,705] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2018-11-14 16:35:15,814] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.164 [2018-11-14 16:35:16,073] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:16,444] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.164: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:16,511] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:16,625] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:16,630] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2018-11-14 16:35:16,735] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.164 [2018-11-14 16:35:17,023] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:17,429] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.164: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:17,528] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:17,642] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:17,647] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.168 with username:root password:couchbase ssh_key: [2018-11-14 16:35:17,788] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.168 [2018-11-14 16:35:17,982] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:18,430] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.168: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:18,563] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:18,678] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:18,683] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.168 with username:root password:couchbase ssh_key: [2018-11-14 16:35:18,823] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.168 [2018-11-14 16:35:18,990] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:19,415] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.168: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:19,546] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:19,661] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:19,662] - [xdcrnewbasetests] [140700069775104] - INFO - {'172.23.104.104': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.97.224': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.97.225': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.161': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.98.164': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.98.168': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.108.236': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.122.201': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}} [2018-11-14 16:35:19,677] - [rest_client] [140700069775104] - ERROR - POST http://172.23.104.104:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password [2018-11-14 16:35:19,678] - [rest_client] [140700069775104] - INFO - /diag/eval status on 172.23.104.104:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2018-11-14 16:35:19,685] - [remote_util] [140700069775104] - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: [2018-11-14 16:35:19,827] - [remote_util] [140700069775104] - INFO - Connected to 172.23.104.104 [2018-11-14 16:35:20,104] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:20,595] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.104.104: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:20,725] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:20,840] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:20,846] - [remote_util] [140700069775104] - INFO - connecting to 172.23.107.161 with username:root password:couchbase ssh_key: [2018-11-14 16:35:21,029] - [remote_util] [140700069775104] - INFO - Connected to 172.23.107.161 [2018-11-14 16:35:21,586] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.107.161: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:21,769] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:21,883] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:21,888] - [remote_util] [140700069775104] - INFO - connecting to 172.23.108.236 with username:root password:couchbase ssh_key: [2018-11-14 16:35:22,034] - [remote_util] [140700069775104] - INFO - Connected to 172.23.108.236 [2018-11-14 16:35:22,308] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:22,718] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.108.236: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:22,849] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:22,964] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:22,968] - [remote_util] [140700069775104] - INFO - connecting to 172.23.122.201 with username:root password:couchbase ssh_key: [2018-11-14 16:35:23,236] - [remote_util] [140700069775104] - INFO - Connected to 172.23.122.201 [2018-11-14 16:35:23,667] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:24,229] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.122.201: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:24,411] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:24,526] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:24,533] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.224 with username:root password:couchbase ssh_key: [2018-11-14 16:35:24,638] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.224 [2018-11-14 16:35:24,840] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:25,296] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.224: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:25,394] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:25,509] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:25,513] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.225 with username:root password:couchbase ssh_key: [2018-11-14 16:35:25,617] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.225 [2018-11-14 16:35:25,837] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:26,221] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.225: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:26,319] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:26,434] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:26,439] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2018-11-14 16:35:26,545] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.164 [2018-11-14 16:35:26,831] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:27,218] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.164: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:27,317] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:27,431] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:27,436] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.168 with username:root password:couchbase ssh_key: [2018-11-14 16:35:27,575] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.168 [2018-11-14 16:35:27,769] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:28,258] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.168: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2018-11-14 16:35:28,388] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:28,503] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:35:28,504] - [xdcrnewbasetests] [140700069775104] - INFO - {'172.23.104.104': 0, '172.23.97.224': 0, '172.23.97.225': 0, '172.23.107.161': 0, '172.23.98.164': 0, '172.23.98.168': 0, '172.23.108.236': 0, '172.23.122.201': 0} [2018-11-14 16:35:28,504] - [xdcrnewbasetests] [140700069775104] - INFO - ==== XDCRNewbasetests setup is started for test #34 load_with_ops ==== [2018-11-14 16:35:28,507] - [xdcrnewbasetests] [140700069775104] - INFO - removing xdcr/nodes settings [2018-11-14 16:35:28,544] - [xdcrnewbasetests] [140700069775104] - INFO - cleanup [ip:172.23.104.104 port:8091 ssh_username:root, ip:172.23.107.161 port:8091 ssh_username:root] [2018-11-14 16:35:28,586] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.104.104:8091 [2018-11-14 16:35:28,593] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.104.104:8091 is running [2018-11-14 16:35:28,657] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.107.161:8091 [2018-11-14 16:35:28,664] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.107.161:8091 is running [2018-11-14 16:35:28,665] - [xdcrnewbasetests] [140700069775104] - INFO - removing xdcr/nodes settings [2018-11-14 16:35:28,705] - [xdcrnewbasetests] [140700069775104] - INFO - cleanup [ip:172.23.108.236 port:8091 ssh_username:root, ip:172.23.122.201 port:8091 ssh_username:root] [2018-11-14 16:35:28,752] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.108.236:8091 [2018-11-14 16:35:28,760] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.108.236:8091 is running [2018-11-14 16:35:28,822] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.122.201:8091 [2018-11-14 16:35:28,831] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.122.201:8091 is running [2018-11-14 16:35:28,831] - [xdcrnewbasetests] [140700069775104] - INFO - removing xdcr/nodes settings [2018-11-14 16:35:28,871] - [xdcrnewbasetests] [140700069775104] - INFO - cleanup [ip:172.23.97.224 port:8091 ssh_username:root, ip:172.23.97.225 port:8091 ssh_username:root] [2018-11-14 16:35:28,912] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.97.224:8091 [2018-11-14 16:35:28,919] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.97.224:8091 is running [2018-11-14 16:35:28,957] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.97.225:8091 [2018-11-14 16:35:28,963] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.97.225:8091 is running [2018-11-14 16:35:28,963] - [xdcrnewbasetests] [140700069775104] - INFO - Initializing all clusters... [2018-11-14 16:35:28,973] - [remote_util] [140700069775104] - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: [2018-11-14 16:35:29,113] - [remote_util] [140700069775104] - INFO - Connected to 172.23.104.104 [2018-11-14 16:35:29,408] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:29,912] - [rest_client] [140700069775104] - ERROR - GET http://172.23.104.104:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2018-11-14 16:35:29,913] - [remote_util] [140700069775104] - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2018-11-14 16:35:29,913] - [xdcrnewbasetests] [140700069775104] - INFO - Running in compatibility mode, not enabled diag/eval for non-local hosts [2018-11-14 16:35:29,950] - [rest_client] [140700069775104] - ERROR - POST http://172.23.104.104:8091/diag/eval/ body: cluster_compat_mode:get_compat_version(). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password [2018-11-14 16:35:29,951] - [rest_client] [140700069775104] - INFO - /diag/eval status on 172.23.104.104:8091: False content: API is accessible from localhost only command: cluster_compat_mode:get_compat_version(). [2018-11-14 16:35:30,528] - [task] [140699840280320] - INFO - server: ip:172.23.104.104 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.104', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.104.104', 'uptime': u'20754', 'mcdMemoryReserved': 3220, 'storageTotalRam': 4025, 'hostname': u'172.23.104.104:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-1622-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 2875351040, 'memoryTotal': 4220854272, 'memoryQuota': 1749, 'mcdMemoryAllocated': 3220, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2018-11-14 16:35:30,530] - [rest_client] [140699840280320] - INFO - pools/default params : memoryQuota=2146 [2018-11-14 16:35:30,538] - [rest_client] [140699840280320] - INFO - settings/web params on 172.23.104.104:8091:username=Administrator&password=password&port=8091 [2018-11-14 16:35:30,805] - [remote_util] [140699840280320] - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: [2018-11-14 16:35:30,944] - [remote_util] [140699840280320] - INFO - Connected to 172.23.104.104 [2018-11-14 16:35:31,242] - [remote_util] [140699840280320] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:31,682] - [remote_util] [140699840280320] - INFO - running command.raw on 172.23.104.104: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2018-11-14 16:35:31,749] - [remote_util] [140699840280320] - INFO - command executed successfully [2018-11-14 16:35:31,755] - [rest_client] [140699840280320] - INFO - /diag/eval status on 172.23.104.104:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2018-11-14 16:35:31,756] - [rest_client] [140699840280320] - INFO - settings/indexes params : storageMode=forestdb [2018-11-14 16:35:31,761] - [rest_client] [140699840280320] - ERROR - POST http://172.23.104.104:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2018-11-14 16:35:31,906] - [task] [140699840280320] - INFO - server: ip:172.23.107.161 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.161', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.161', 'uptime': u'8941', 'mcdMemoryReserved': 3047, 'storageTotalRam': 3809, 'hostname': u'172.23.107.161:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-1622-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 2608644096, 'memoryTotal': 3994247168, 'memoryQuota': 2146, 'mcdMemoryAllocated': 3047, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2018-11-14 16:35:31,907] - [rest_client] [140699840280320] - INFO - pools/default params : memoryQuota=2031 [2018-11-14 16:35:31,916] - [rest_client] [140699840280320] - INFO - settings/web params on 172.23.107.161:8091:username=Administrator&password=password&port=8091 [2018-11-14 16:35:31,974] - [remote_util] [140699840280320] - INFO - connecting to 172.23.107.161 with username:root password:couchbase ssh_key: [2018-11-14 16:35:32,112] - [remote_util] [140699840280320] - INFO - Connected to 172.23.107.161 [2018-11-14 16:35:32,702] - [remote_util] [140699840280320] - INFO - running command.raw on 172.23.107.161: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2018-11-14 16:35:32,778] - [remote_util] [140699840280320] - INFO - command executed successfully [2018-11-14 16:35:32,783] - [rest_client] [140699840280320] - INFO - /diag/eval status on 172.23.107.161:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2018-11-14 16:35:32,784] - [rest_client] [140699840280320] - INFO - settings/indexes params : storageMode=forestdb [2018-11-14 16:35:32,792] - [rest_client] [140699840280320] - ERROR - POST http://172.23.107.161:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2018-11-14 16:35:33,877] - [task] [140699840280320] - INFO - adding node 172.23.107.161:8091 to cluster [2018-11-14 16:35:33,878] - [rest_client] [140699840280320] - INFO - adding remote node @172.23.107.161:8091 to this cluster @172.23.104.104:8091 [2018-11-14 16:35:36,181] - [rest_client] [140699840280320] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.104,ns_1@172.23.107.161'} [2018-11-14 16:35:36,187] - [rest_client] [140699840280320] - INFO - rebalance operation started [2018-11-14 16:35:36,193] - [rest_client] [140699840280320] - INFO - rebalance percentage : 0.00 % [2018-11-14 16:35:36,194] - [task] [140699840280320] - INFO - Rebalance - status: running, progress: 0.00% [2018-11-14 16:35:46,218] - [task] [140699840280320] - INFO - Rebalance - status: none, progress: 100.00% [2018-11-14 16:35:46,227] - [task] [140699840280320] - INFO - rebalancing was completed with progress: 100% in 10.0395739079 sec [2018-11-14 16:35:46,241] - [remote_util] [140700069775104] - INFO - connecting to 172.23.108.236 with username:root password:couchbase ssh_key: [2018-11-14 16:35:46,409] - [remote_util] [140700069775104] - INFO - Connected to 172.23.108.236 [2018-11-14 16:35:46,663] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:47,165] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.108.236: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2018-11-14 16:35:47,231] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:35:47,231] - [xdcrnewbasetests] [140700069775104] - INFO - Enabled diag/eval for non-local hosts from 172.23.108.236 [2018-11-14 16:35:47,351] - [rest_client] [140700069775104] - INFO - /diag/eval status on 172.23.108.236:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2018-11-14 16:35:47,543] - [task] [140700052989696] - INFO - server: ip:172.23.108.236 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.108.236', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.108.236', 'uptime': u'20756', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3778, 'hostname': u'172.23.108.236:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-1622-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 2637590528, 'memoryTotal': 3961716736, 'memoryQuota': 2014, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2018-11-14 16:35:47,544] - [rest_client] [140700052989696] - INFO - pools/default params : memoryQuota=2014 [2018-11-14 16:35:47,552] - [rest_client] [140700052989696] - INFO - settings/web params on 172.23.108.236:8091:username=Administrator&password=password&port=8091 [2018-11-14 16:35:47,635] - [remote_util] [140700052989696] - INFO - connecting to 172.23.108.236 with username:root password:couchbase ssh_key: [2018-11-14 16:35:47,802] - [remote_util] [140700052989696] - INFO - Connected to 172.23.108.236 [2018-11-14 16:35:48,062] - [remote_util] [140700052989696] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:48,579] - [remote_util] [140700052989696] - INFO - running command.raw on 172.23.108.236: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2018-11-14 16:35:48,646] - [remote_util] [140700052989696] - INFO - command executed successfully [2018-11-14 16:35:48,655] - [rest_client] [140700052989696] - INFO - /diag/eval status on 172.23.108.236:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2018-11-14 16:35:48,655] - [rest_client] [140700052989696] - INFO - settings/indexes params : storageMode=forestdb [2018-11-14 16:35:48,661] - [rest_client] [140700052989696] - ERROR - POST http://172.23.108.236:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2018-11-14 16:35:48,765] - [task] [140700052989696] - INFO - server: ip:172.23.122.201 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.122.201', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.122.201', 'uptime': u'8943', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3778, 'hostname': u'172.23.122.201:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-1622-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 2709278720, 'memoryTotal': 3961716736, 'memoryQuota': 1601, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2018-11-14 16:35:48,765] - [rest_client] [140700052989696] - INFO - pools/default params : memoryQuota=2014 [2018-11-14 16:35:48,776] - [rest_client] [140700052989696] - INFO - settings/web params on 172.23.122.201:8091:username=Administrator&password=password&port=8091 [2018-11-14 16:35:48,997] - [remote_util] [140700052989696] - INFO - connecting to 172.23.122.201 with username:root password:couchbase ssh_key: [2018-11-14 16:35:49,265] - [remote_util] [140700052989696] - INFO - Connected to 172.23.122.201 [2018-11-14 16:35:49,670] - [remote_util] [140700052989696] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:35:50,311] - [remote_util] [140700052989696] - INFO - running command.raw on 172.23.122.201: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2018-11-14 16:35:50,379] - [remote_util] [140700052989696] - INFO - command executed successfully [2018-11-14 16:35:50,386] - [rest_client] [140700052989696] - INFO - /diag/eval status on 172.23.122.201:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2018-11-14 16:35:50,386] - [rest_client] [140700052989696] - INFO - settings/indexes params : storageMode=forestdb [2018-11-14 16:35:50,393] - [rest_client] [140700052989696] - ERROR - POST http://172.23.122.201:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2018-11-14 16:35:51,479] - [task] [140700052989696] - INFO - adding node 172.23.122.201:8091 to cluster [2018-11-14 16:35:51,480] - [rest_client] [140700052989696] - INFO - adding remote node @172.23.122.201:8091 to this cluster @172.23.108.236:8091 [2018-11-14 16:35:58,181] - [rest_client] [140700052989696] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.108.236,ns_1@172.23.122.201'} [2018-11-14 16:35:58,193] - [rest_client] [140700052989696] - INFO - rebalance operation started [2018-11-14 16:35:58,201] - [rest_client] [140700052989696] - INFO - rebalance percentage : 0.00 % [2018-11-14 16:35:58,202] - [task] [140700052989696] - INFO - Rebalance - status: running, progress: 0.00% [2018-11-14 16:36:08,230] - [task] [140700052989696] - INFO - Rebalance - status: none, progress: 100.00% [2018-11-14 16:36:08,239] - [task] [140700052989696] - INFO - rebalancing was completed with progress: 100% in 10.0461781025 sec [2018-11-14 16:36:08,254] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.224 with username:root password:couchbase ssh_key: [2018-11-14 16:36:08,364] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.224 [2018-11-14 16:36:08,559] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:36:09,009] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.224: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2018-11-14 16:36:09,075] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:36:09,076] - [xdcrnewbasetests] [140700069775104] - INFO - Enabled diag/eval for non-local hosts from 172.23.97.224 [2018-11-14 16:36:09,196] - [rest_client] [140700069775104] - INFO - /diag/eval status on 172.23.97.224:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2018-11-14 16:36:09,567] - [task] [140699789924096] - INFO - server: ip:172.23.97.224 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.224', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.97.224', 'uptime': u'20792', 'mcdMemoryReserved': 3230, 'storageTotalRam': 4038, 'hostname': u'172.23.97.224:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-1622-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 2971660288, 'memoryTotal': 4234354688, 'memoryQuota': 2153, 'mcdMemoryAllocated': 3230, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2018-11-14 16:36:09,567] - [rest_client] [140699789924096] - INFO - pools/default params : memoryQuota=2153 [2018-11-14 16:36:09,574] - [rest_client] [140699789924096] - INFO - settings/web params on 172.23.97.224:8091:username=Administrator&password=password&port=8091 [2018-11-14 16:36:09,617] - [remote_util] [140699789924096] - INFO - connecting to 172.23.97.224 with username:root password:couchbase ssh_key: [2018-11-14 16:36:09,720] - [remote_util] [140699789924096] - INFO - Connected to 172.23.97.224 [2018-11-14 16:36:09,939] - [remote_util] [140699789924096] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:36:10,323] - [remote_util] [140699789924096] - INFO - running command.raw on 172.23.97.224: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2018-11-14 16:36:10,390] - [remote_util] [140699789924096] - INFO - command executed successfully [2018-11-14 16:36:10,396] - [rest_client] [140699789924096] - INFO - /diag/eval status on 172.23.97.224:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2018-11-14 16:36:10,397] - [rest_client] [140699789924096] - INFO - settings/indexes params : storageMode=forestdb [2018-11-14 16:36:10,402] - [rest_client] [140699789924096] - ERROR - POST http://172.23.97.224:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2018-11-14 16:36:10,494] - [task] [140699789924096] - INFO - server: ip:172.23.97.225 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.225', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.97.225', 'uptime': u'20786', 'mcdMemoryReserved': 3230, 'storageTotalRam': 4038, 'hostname': u'172.23.97.225:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-1622-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3074887680, 'memoryTotal': 4234354688, 'memoryQuota': 1757, 'mcdMemoryAllocated': 3230, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2018-11-14 16:36:10,494] - [rest_client] [140699789924096] - INFO - pools/default params : memoryQuota=2153 [2018-11-14 16:36:10,502] - [rest_client] [140699789924096] - INFO - settings/web params on 172.23.97.225:8091:username=Administrator&password=password&port=8091 [2018-11-14 16:36:10,546] - [remote_util] [140699789924096] - INFO - connecting to 172.23.97.225 with username:root password:couchbase ssh_key: [2018-11-14 16:36:10,649] - [remote_util] [140699789924096] - INFO - Connected to 172.23.97.225 [2018-11-14 16:36:10,876] - [remote_util] [140699789924096] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:36:11,358] - [remote_util] [140699789924096] - INFO - running command.raw on 172.23.97.225: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2018-11-14 16:36:11,424] - [remote_util] [140699789924096] - INFO - command executed successfully [2018-11-14 16:36:11,430] - [rest_client] [140699789924096] - INFO - /diag/eval status on 172.23.97.225:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2018-11-14 16:36:11,430] - [rest_client] [140699789924096] - INFO - settings/indexes params : storageMode=forestdb [2018-11-14 16:36:11,436] - [rest_client] [140699789924096] - ERROR - POST http://172.23.97.225:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2018-11-14 16:36:12,515] - [task] [140699789924096] - INFO - adding node 172.23.97.225:8091 to cluster [2018-11-14 16:36:12,516] - [rest_client] [140699789924096] - INFO - adding remote node @172.23.97.225:8091 to this cluster @172.23.97.224:8091 [2018-11-14 16:36:13,819] - [rest_client] [140699789924096] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.97.225,ns_1@172.23.97.224'} [2018-11-14 16:36:13,826] - [rest_client] [140699789924096] - INFO - rebalance operation started [2018-11-14 16:36:13,833] - [rest_client] [140699789924096] - INFO - rebalance percentage : 0.00 % [2018-11-14 16:36:13,833] - [task] [140699789924096] - INFO - Rebalance - status: running, progress: 0.00% [2018-11-14 16:36:23,861] - [task] [140699789924096] - INFO - Rebalance - status: none, progress: 100.00% [2018-11-14 16:36:23,869] - [task] [140699789924096] - INFO - rebalancing was completed with progress: 100% in 10.0433809757 sec [2018-11-14 16:36:23,896] - [rest_client] [140700069775104] - ERROR - DELETE http://172.23.104.104:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2018-11-14 16:36:23,897] - [internal_user] [140700069775104] - INFO - Exception while deleting user. Exception is -"User was not found." [2018-11-14 16:36:24,023] - [rest_client] [140700069775104] - ERROR - DELETE http://172.23.108.236:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2018-11-14 16:36:24,024] - [internal_user] [140700069775104] - INFO - Exception while deleting user. Exception is -"User was not found." [2018-11-14 16:36:24,181] - [rest_client] [140700069775104] - ERROR - DELETE http://172.23.97.224:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2018-11-14 16:36:24,181] - [internal_user] [140700069775104] - INFO - Exception while deleting user. Exception is -"User was not found." [2018-11-14 16:36:24,300] - [rest_client] [140699840280320] - INFO - http://172.23.104.104:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1531&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2018-11-14 16:36:24,308] - [rest_client] [140699840280320] - INFO - 0.01 seconds to create bucket default [2018-11-14 16:36:24,309] - [bucket_helper] [140699840280320] - INFO - waiting for memcached bucket : default in 172.23.104.104 to accept set ops [2018-11-14 16:36:25,467] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:36:25,560] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:36:25,758] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:36:25,858] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:36:26,576] - [task] [140699840280320] - INFO - bucket 'default' was created with per node RAM quota: 1531 [2018-11-14 16:36:27,299] - [rest_client] [140700052989696] - INFO - http://172.23.108.236:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1514&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2018-11-14 16:36:27,312] - [rest_client] [140700052989696] - INFO - 0.01 seconds to create bucket default [2018-11-14 16:36:27,312] - [bucket_helper] [140700052989696] - INFO - waiting for memcached bucket : default in 172.23.108.236 to accept set ops [2018-11-14 16:36:29,041] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:36:29,166] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:36:29,394] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:36:29,512] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:36:30,394] - [task] [140700052989696] - INFO - bucket 'default' was created with per node RAM quota: 1514 [2018-11-14 16:36:30,914] - [rest_client] [140699789924096] - INFO - http://172.23.97.224:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1653&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2018-11-14 16:36:30,923] - [rest_client] [140699789924096] - INFO - 0.01 seconds to create bucket default [2018-11-14 16:36:30,924] - [bucket_helper] [140699789924096] - INFO - waiting for memcached bucket : default in 172.23.97.224 to accept set ops [2018-11-14 16:36:32,067] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:36:32,162] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:36:32,294] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:36:32,398] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:36:32,972] - [task] [140699789924096] - INFO - bucket 'default' was created with per node RAM quota: 1653 [2018-11-14 16:36:32,998] - [xdcrnewbasetests] [140700069775104] - INFO - ==== XDCRNewbasetests setup is finished for test #34 load_with_ops ==== [2018-11-14 16:36:33,006] - [rest_client] [140700069775104] - INFO - adding remote cluster hostname:172.23.108.236:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.104.104:8091 [2018-11-14 16:36:33,037] - [rest_client] [140700069775104] - INFO - adding remote cluster hostname:172.23.104.104:8091 with username:password Administrator:password name:remote_cluster_C2-C1 to source node: 172.23.108.236:8091 [2018-11-14 16:36:33,078] - [rest_client] [140700069775104] - INFO - adding remote cluster hostname:172.23.97.224:8091 with username:password Administrator:password name:remote_cluster_C2-C3 to source node: 172.23.108.236:8091 [2018-11-14 16:36:33,111] - [rest_client] [140700069775104] - INFO - adding remote cluster hostname:172.23.108.236:8091 with username:password Administrator:password name:remote_cluster_C3-C2 to source node: 172.23.97.224:8091 [2018-11-14 16:36:33,143] - [rest_client] [140700069775104] - INFO - adding remote cluster hostname:172.23.104.104:8091 with username:password Administrator:password name:remote_cluster_C3-C1 to source node: 172.23.97.224:8091 [2018-11-14 16:36:33,175] - [rest_client] [140700069775104] - INFO - adding remote cluster hostname:172.23.97.224:8091 with username:password Administrator:password name:remote_cluster_C1-C3 to source node: 172.23.104.104:8091 [2018-11-14 16:36:33,200] - [rest_client] [140700069775104] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {'filterExpression': 'C1-key-1'} [2018-11-14 16:36:33,274] - [rest_client] [140700069775104] - INFO - Replication created with id: c0223380653196a59e4519455a6214a3/default/default [2018-11-14 16:36:43,293] - [rest_client] [140700069775104] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C3 with settings {'filterExpression': 'C1-key-1'} [2018-11-14 16:36:43,339] - [rest_client] [140700069775104] - INFO - Replication created with id: 150d541992e12ae0265b806af08c6784/default/default [2018-11-14 16:36:53,359] - [rest_client] [140700069775104] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C2-C1 with settings {} [2018-11-14 16:36:53,429] - [rest_client] [140700069775104] - INFO - Replication created with id: d16fcf8fdd1f7e163a43f34401f534fc/default/default [2018-11-14 16:37:03,448] - [rest_client] [140700069775104] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C2-C3 with settings {} [2018-11-14 16:37:03,510] - [rest_client] [140700069775104] - INFO - Replication created with id: 150d541992e12ae0265b806af08c6784/default/default [2018-11-14 16:37:13,530] - [rest_client] [140700069775104] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C3-C2 with settings {} [2018-11-14 16:37:13,577] - [rest_client] [140700069775104] - INFO - Replication created with id: c0223380653196a59e4519455a6214a3/default/default [2018-11-14 16:37:23,595] - [rest_client] [140700069775104] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C3-C1 with settings {} [2018-11-14 16:37:23,644] - [rest_client] [140700069775104] - INFO - Replication created with id: d16fcf8fdd1f7e163a43f34401f534fc/default/default [2018-11-14 16:37:33,682] - [rest_client] [140700069775104] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.104.104 [2018-11-14 16:37:33,700] - [rest_client] [140700069775104] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.104.104 [2018-11-14 16:37:33,733] - [rest_client] [140700069775104] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.108.236 [2018-11-14 16:37:33,757] - [rest_client] [140700069775104] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.108.236 [2018-11-14 16:37:33,782] - [rest_client] [140700069775104] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.97.224 [2018-11-14 16:37:33,800] - [rest_client] [140700069775104] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.97.224 [2018-11-14 16:37:33,908] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:37:34,031] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:37:34,762] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:37:34,808] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:37:34,867] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:37:34,891] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:37:34,916] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:37:34,928] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:37:34,982] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:37:34,999] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:37:37,542] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:37:37,656] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:37:38,618] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:37:38,666] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:37:38,708] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:37:38,734] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:37:38,739] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:37:38,801] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:37:38,839] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:37:38,868] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:37:41,331] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:37:41,422] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:37:42,214] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:37:42,240] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:37:42,279] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:37:42,299] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:37:42,324] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:37:42,354] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:37:42,387] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:37:42,392] - [data_helper] [140699831887616] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:37:44,285] - [xdcrnewbasetests] [140700069775104] - INFO - sleep for 10 secs. ... [2018-11-14 16:37:54,296] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C1:default -> C2:default [2018-11-14 16:37:54,368] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 10000 valid and 0 deleted keys [2018-11-14 16:37:54,368] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 10000 valid and 0 deleted keys [2018-11-14 16:37:54,417] - [xdcrnewbasetests] [140700069775104] - INFO - 1111 keys matched the filter expression C1-key-1 [2018-11-14 16:37:54,472] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 11111 valid keys and 0 deleted keys [2018-11-14 16:37:54,473] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C1:default -> C3:default [2018-11-14 16:37:54,519] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 10000 valid and 0 deleted keys [2018-11-14 16:37:54,520] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 10000 valid and 0 deleted keys [2018-11-14 16:37:54,567] - [xdcrnewbasetests] [140700069775104] - INFO - 1111 keys matched the filter expression C1-key-1 [2018-11-14 16:37:54,617] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 11111 valid keys and 0 deleted keys [2018-11-14 16:37:54,618] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C2:default -> C1:default [2018-11-14 16:37:54,665] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 11111 valid and 0 deleted keys [2018-11-14 16:37:54,665] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 10000 valid and 0 deleted keys [2018-11-14 16:37:54,930] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 20000 valid keys and 0 deleted keys [2018-11-14 16:37:54,931] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C2:default -> C3:default [2018-11-14 16:37:54,980] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 11111 valid and 0 deleted keys [2018-11-14 16:37:54,980] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 11111 valid and 0 deleted keys [2018-11-14 16:37:55,264] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2018-11-14 16:37:55,265] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C3:default -> C2:default [2018-11-14 16:37:55,340] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2018-11-14 16:37:55,340] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 11111 valid and 0 deleted keys [2018-11-14 16:37:55,800] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2018-11-14 16:37:55,801] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C3:default -> C1:default [2018-11-14 16:37:55,898] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2018-11-14 16:37:55,898] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 20000 valid and 0 deleted keys [2018-11-14 16:37:56,368] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 30000 valid keys and 0 deleted keys [2018-11-14 16:37:56,370] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C1:default -> C2:default [2018-11-14 16:37:56,506] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 30000 valid and 0 deleted keys [2018-11-14 16:37:56,507] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 21111 valid and 0 deleted keys [2018-11-14 16:37:56,647] - [xdcrnewbasetests] [140700069775104] - INFO - 1111 keys matched the filter expression C1-key-1 [2018-11-14 16:37:56,717] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2018-11-14 16:37:56,717] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C1:default -> C3:default [2018-11-14 16:37:56,838] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 30000 valid and 0 deleted keys [2018-11-14 16:37:56,838] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 21111 valid and 0 deleted keys [2018-11-14 16:37:56,973] - [xdcrnewbasetests] [140700069775104] - INFO - 1111 keys matched the filter expression C1-key-1 [2018-11-14 16:37:57,069] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2018-11-14 16:37:57,070] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C2:default -> C1:default [2018-11-14 16:37:57,248] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2018-11-14 16:37:57,248] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 30000 valid and 0 deleted keys [2018-11-14 16:37:57,746] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 30000 valid keys and 0 deleted keys [2018-11-14 16:37:57,747] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C2:default -> C3:default [2018-11-14 16:37:57,892] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2018-11-14 16:37:57,893] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 21111 valid and 0 deleted keys [2018-11-14 16:37:58,559] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2018-11-14 16:37:58,560] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C3:default -> C2:default [2018-11-14 16:37:58,648] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2018-11-14 16:37:58,648] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 21111 valid and 0 deleted keys [2018-11-14 16:37:59,051] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2018-11-14 16:37:59,051] - [xdcrnewbasetests] [140700069775104] - INFO - Merging keys for replication Replication C3:default -> C1:default [2018-11-14 16:37:59,165] - [xdcrnewbasetests] [140700069775104] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2018-11-14 16:37:59,165] - [xdcrnewbasetests] [140700069775104] - INFO - dest kvstore has 30000 valid and 0 deleted keys [2018-11-14 16:37:59,590] - [xdcrnewbasetests] [140700069775104] - INFO - After merging: destination bucket's kv_store now has 30000 valid keys and 0 deleted keys [2018-11-14 16:37:59,637] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'} [2018-11-14 16:37:59,638] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:37:59,706] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:37:59,707] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1502552488, 0, '') [2018-11-14 16:37:59,738] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'} [2018-11-14 16:37:59,739] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:37:59,831] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:37:59,832] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3017336685, 0, '') [2018-11-14 16:37:59,832] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:38:09,895] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'} [2018-11-14 16:38:09,895] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:38:09,974] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:38:09,975] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1963470990, 0, '') [2018-11-14 16:38:10,031] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'} [2018-11-14 16:38:10,031] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:38:10,206] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:38:10,207] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2795083625, 0, '') [2018-11-14 16:38:10,208] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:38:20,841] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:38:20,931] - [task] [140699840280320] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.104:8091',default bucket [2018-11-14 16:38:20,947] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:38:21,041] - [task] [140699840280320] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.161:8091',default bucket [2018-11-14 16:38:21,654] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:38:21,758] - [task] [140700052989696] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.108.236:8091',default bucket [2018-11-14 16:38:21,784] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:38:21,948] - [task] [140700052989696] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.122.201:8091',default bucket [2018-11-14 16:38:21,980] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.104.104 [2018-11-14 16:38:21,990] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:38:22,032] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C1 for default is 0 [2018-11-14 16:38:22,032] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.108.236 [2018-11-14 16:38:22,041] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:38:22,109] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C2 for default is 0 [2018-11-14 16:38:22,110] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.104.104 [2018-11-14 16:38:22,163] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0 [2018-11-14 16:38:22,163] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.108.236 [2018-11-14 16:38:22,228] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0 [2018-11-14 16:38:22,277] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:38:22,320] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 30000 == 30000 expected on C1, default bucket [2018-11-14 16:38:22,356] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:38:22,391] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 30000 == 30000 expected on C1, default bucket [2018-11-14 16:38:22,425] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:38:22,486] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C2, default bucket [2018-11-14 16:38:22,511] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:38:22,569] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C2, default bucket [2018-11-14 16:38:22,634] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:38:22,724] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:38:22,994] - [task] [140700069775104] - INFO - 30000 items will be verified on default bucket [2018-11-14 16:38:23,044] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:38:30,003] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:38:36,787] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:38:43,938] - [task] [140699831887616] - INFO - 30000 items were verified [2018-11-14 16:38:43,939] - [task] [140699831887616] - INFO - 30000 items were verified in 20.9437251091 sec.the average number of ops - 1432.40980827 per second [2018-11-14 16:38:44,037] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:38:44,200] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:38:44,462] - [task] [140700069775104] - INFO - 21111 items will be verified on default bucket [2018-11-14 16:38:44,997] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:38:51,235] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:38:58,124] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:38:58,956] - [task] [140699831887616] - INFO - 21111 items were verified in 14.4933719635 sec.the average number of ops - 1456.59663794 per second [2018-11-14 16:38:58,957] - [xdcrnewbasetests] [140700069775104] - INFO - Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.104.104 -> Couchbase Cluster: C2, Master Ip: 172.23.108.236, bucket default [2018-11-14 16:38:59,029] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:38:59,130] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:38:59,356] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:38:59,467] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:38:59,629] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:38:59,748] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:39:00,137] - [task] [140700069775104] - INFO - RevID verification: in progress for default ... [2018-11-14 16:40:02,502] - [task] [140699831887616] - INFO - RevId Verification : 30000 existing items have been verified [2018-11-14 16:40:02,503] - [task] [140699831887616] - INFO - RevId Verification : 0 deleted items have been verified [2018-11-14 16:40:02,503] - [task] [140699831887616] - INFO - RevId Verification : 8889 keys were apparently filtered out and not found in target bucket [2018-11-14 16:40:02,512] - [rest_client] [140699831887616] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:40:02,617] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'} [2018-11-14 16:40:02,617] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:40:02,686] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:40:02,687] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1225242662, 0, '') [2018-11-14 16:40:02,734] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'} [2018-11-14 16:40:02,734] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:40:02,824] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:40:02,826] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3384049445, 0, '') [2018-11-14 16:40:02,826] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:40:12,881] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'} [2018-11-14 16:40:12,882] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:40:12,951] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:40:12,952] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (807152405, 0, '') [2018-11-14 16:40:12,979] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'} [2018-11-14 16:40:12,980] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:40:13,050] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:40:13,051] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (737457452, 0, '') [2018-11-14 16:40:13,051] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:40:23,298] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:40:23,397] - [task] [140699840280320] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.104:8091',default bucket [2018-11-14 16:40:23,418] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:40:23,535] - [task] [140699840280320] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.161:8091',default bucket [2018-11-14 16:40:24,395] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:40:24,482] - [task] [140699789924096] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.224:8091',default bucket [2018-11-14 16:40:24,496] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:40:24,588] - [task] [140699789924096] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.225:8091',default bucket [2018-11-14 16:40:24,629] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.104.104 [2018-11-14 16:40:24,638] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:40:24,677] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C1 for default is 0 [2018-11-14 16:40:24,678] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.224 [2018-11-14 16:40:24,686] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:40:24,720] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C3 for default is 0 [2018-11-14 16:40:24,720] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.104.104 [2018-11-14 16:40:24,765] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0 [2018-11-14 16:40:24,765] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.224 [2018-11-14 16:40:24,804] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C3 for bucket default is 0 [2018-11-14 16:40:24,852] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:40:24,889] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 30000 == 30000 expected on C1, default bucket [2018-11-14 16:40:24,929] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:40:24,969] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 30000 == 30000 expected on C1, default bucket [2018-11-14 16:40:25,008] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:40:25,040] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C3, default bucket [2018-11-14 16:40:25,065] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:40:25,096] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C3, default bucket [2018-11-14 16:40:25,202] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:40:25,301] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:40:25,560] - [task] [140700069775104] - INFO - 30000 items will be verified on default bucket [2018-11-14 16:40:26,544] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:40:33,405] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:40:40,096] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:40:46,784] - [task] [140699831887616] - INFO - 30000 items were verified [2018-11-14 16:40:46,784] - [task] [140699831887616] - INFO - 30000 items were verified in 21.223679781 sec.the average number of ops - 1413.51534119 per second [2018-11-14 16:40:46,855] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:40:46,939] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:40:47,122] - [task] [140700069775104] - INFO - 21111 items will be verified on default bucket [2018-11-14 16:40:47,619] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:40:54,441] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:41:01,034] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:41:01,935] - [task] [140699831887616] - INFO - 21111 items were verified in 14.8127570152 sec.the average number of ops - 1425.19018578 per second [2018-11-14 16:41:01,937] - [xdcrnewbasetests] [140700069775104] - INFO - Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.104.104 -> Couchbase Cluster: C3, Master Ip: 172.23.97.224, bucket default [2018-11-14 16:41:02,011] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:41:02,105] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:41:02,351] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:41:02,513] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:41:02,675] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:41:02,772] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:41:02,981] - [task] [140700069775104] - INFO - RevID verification: in progress for default ... [2018-11-14 16:41:56,230] - [task] [140699831887616] - INFO - RevId Verification : 30000 existing items have been verified [2018-11-14 16:41:56,231] - [task] [140699831887616] - INFO - RevId Verification : 0 deleted items have been verified [2018-11-14 16:41:56,231] - [task] [140699831887616] - INFO - RevId Verification : 8889 keys were apparently filtered out and not found in target bucket [2018-11-14 16:41:56,242] - [rest_client] [140699831887616] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:41:56,364] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'} [2018-11-14 16:41:56,364] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:41:56,456] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:41:56,457] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (224761744, 0, '') [2018-11-14 16:41:56,504] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'} [2018-11-14 16:41:56,504] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:41:56,656] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:41:56,657] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (323010072, 0, '') [2018-11-14 16:41:56,658] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:42:06,706] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'} [2018-11-14 16:42:06,707] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:42:06,780] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:42:06,781] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3582813494, 0, '') [2018-11-14 16:42:06,811] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'} [2018-11-14 16:42:06,811] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:42:06,901] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:42:06,902] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1383817711, 0, '') [2018-11-14 16:42:06,902] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:42:17,428] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:42:17,568] - [task] [140700052989696] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.108.236:8091',default bucket [2018-11-14 16:42:17,594] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:42:17,767] - [task] [140700052989696] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.122.201:8091',default bucket [2018-11-14 16:42:18,781] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:42:18,862] - [task] [140699840280320] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.104:8091',default bucket [2018-11-14 16:42:18,889] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:42:18,992] - [task] [140699840280320] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.161:8091',default bucket [2018-11-14 16:42:19,021] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.108.236 [2018-11-14 16:42:19,029] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:42:19,097] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C2 for default is 0 [2018-11-14 16:42:19,097] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.104.104 [2018-11-14 16:42:19,104] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:42:19,140] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C1 for default is 0 [2018-11-14 16:42:19,140] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.108.236 [2018-11-14 16:42:19,203] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0 [2018-11-14 16:42:19,203] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.104.104 [2018-11-14 16:42:19,252] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0 [2018-11-14 16:42:19,288] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:42:19,351] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C2, default bucket [2018-11-14 16:42:19,377] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:42:19,435] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C2, default bucket [2018-11-14 16:42:19,478] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:42:19,515] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 30000 == 30000 expected on C1, default bucket [2018-11-14 16:42:19,551] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:42:19,591] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 30000 == 30000 expected on C1, default bucket [2018-11-14 16:42:19,673] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:42:19,786] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:42:20,046] - [task] [140700069775104] - INFO - 21111 items will be verified on default bucket [2018-11-14 16:42:20,771] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:42:27,676] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:42:34,243] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:42:34,993] - [task] [140699831887616] - INFO - 21111 items were verified in 14.9464890957 sec.the average number of ops - 1412.43845453 per second [2018-11-14 16:42:35,076] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:42:35,171] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:42:35,457] - [task] [140700069775104] - INFO - 30000 items will be verified on default bucket [2018-11-14 16:42:36,022] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:42:42,246] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:42:48,966] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:42:55,888] - [task] [140699831887616] - INFO - 30000 items were verified [2018-11-14 16:42:55,889] - [task] [140699831887616] - INFO - 30000 items were verified in 20.4313149452 sec.the average number of ops - 1468.33413707 per second [2018-11-14 16:42:55,889] - [xdcrnewbasetests] [140700069775104] - INFO - Verifying RevIds for Couchbase Cluster: C2, Master Ip: 172.23.108.236 -> Couchbase Cluster: C1, Master Ip: 172.23.104.104, bucket default [2018-11-14 16:42:55,975] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:42:56,095] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:42:56,381] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:42:56,499] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:42:56,698] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:42:56,787] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:42:57,018] - [task] [140700069775104] - INFO - RevID verification: in progress for default ... [2018-11-14 16:43:27,813] - [task] [140699831887616] - INFO - RevId Verification : 21111 existing items have been verified [2018-11-14 16:43:27,813] - [task] [140699831887616] - INFO - RevId Verification : 0 deleted items have been verified [2018-11-14 16:43:27,813] - [task] [140699831887616] - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket [2018-11-14 16:43:27,822] - [rest_client] [140699831887616] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:43:27,954] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'} [2018-11-14 16:43:27,954] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:43:28,079] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:43:28,080] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3634096897, 0, '') [2018-11-14 16:43:28,126] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'} [2018-11-14 16:43:28,126] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:43:28,273] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:43:28,274] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1877480205, 0, '') [2018-11-14 16:43:28,275] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:43:38,328] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'} [2018-11-14 16:43:38,329] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:43:38,396] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:43:38,397] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1317293342, 0, '') [2018-11-14 16:43:38,424] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'} [2018-11-14 16:43:38,425] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:43:38,492] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:43:38,493] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2037149050, 0, '') [2018-11-14 16:43:38,494] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:43:49,008] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:43:49,112] - [task] [140700052989696] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.108.236:8091',default bucket [2018-11-14 16:43:49,139] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:43:49,300] - [task] [140700052989696] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.122.201:8091',default bucket [2018-11-14 16:43:49,954] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:43:50,044] - [task] [140699789924096] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.224:8091',default bucket [2018-11-14 16:43:50,058] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:43:50,146] - [task] [140699789924096] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.225:8091',default bucket [2018-11-14 16:43:50,168] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.108.236 [2018-11-14 16:43:50,176] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:43:50,240] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C2 for default is 0 [2018-11-14 16:43:50,241] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.224 [2018-11-14 16:43:50,248] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:43:50,281] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C3 for default is 0 [2018-11-14 16:43:50,281] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.108.236 [2018-11-14 16:43:50,349] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0 [2018-11-14 16:43:50,349] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.224 [2018-11-14 16:43:50,382] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C3 for bucket default is 0 [2018-11-14 16:43:50,416] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:43:50,482] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C2, default bucket [2018-11-14 16:43:50,506] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:43:50,577] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C2, default bucket [2018-11-14 16:43:50,618] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:43:50,650] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C3, default bucket [2018-11-14 16:43:50,675] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:43:50,710] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C3, default bucket [2018-11-14 16:43:50,794] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:43:50,957] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:43:51,217] - [task] [140700069775104] - INFO - 21111 items will be verified on default bucket [2018-11-14 16:43:51,304] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:43:58,048] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:44:04,779] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:44:05,521] - [task] [140699831887616] - INFO - 21111 items were verified in 14.3031740189 sec.the average number of ops - 1475.96580657 per second [2018-11-14 16:44:05,590] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:44:05,674] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:44:05,874] - [task] [140700069775104] - INFO - 21111 items will be verified on default bucket [2018-11-14 16:44:06,178] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:44:13,033] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:44:19,640] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:44:20,300] - [task] [140699831887616] - INFO - 21111 items were verified in 14.4256901741 sec.the average number of ops - 1463.4304607 per second [2018-11-14 16:44:20,301] - [xdcrnewbasetests] [140700069775104] - INFO - Verifying RevIds for Couchbase Cluster: C2, Master Ip: 172.23.108.236 -> Couchbase Cluster: C3, Master Ip: 172.23.97.224, bucket default [2018-11-14 16:44:20,388] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:44:20,513] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:44:20,816] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:44:20,937] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:44:21,207] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:44:21,296] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:44:21,476] - [task] [140700069775104] - INFO - RevID verification: in progress for default ... [2018-11-14 16:44:52,825] - [task] [140699831887616] - INFO - RevId Verification : 21111 existing items have been verified [2018-11-14 16:44:52,826] - [task] [140699831887616] - INFO - RevId Verification : 0 deleted items have been verified [2018-11-14 16:44:52,826] - [task] [140699831887616] - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket [2018-11-14 16:44:52,835] - [rest_client] [140699831887616] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:44:52,987] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'} [2018-11-14 16:44:52,987] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:44:53,054] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:44:53,055] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1707240593, 0, '') [2018-11-14 16:44:53,081] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'} [2018-11-14 16:44:53,081] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:44:53,153] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:44:53,154] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (46840141, 0, '') [2018-11-14 16:44:53,155] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:45:03,217] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'} [2018-11-14 16:45:03,217] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:45:03,300] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:45:03,301] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.108.236', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3638441424, 0, '') [2018-11-14 16:45:03,347] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'} [2018-11-14 16:45:03,347] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:45:03,481] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:45:03,482] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.122.201', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2261220319, 0, '') [2018-11-14 16:45:03,482] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:45:14,373] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:45:14,434] - [task] [140699789924096] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.224:8091',default bucket [2018-11-14 16:45:14,447] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:45:14,541] - [task] [140699789924096] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.225:8091',default bucket [2018-11-14 16:45:15,510] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:45:15,618] - [task] [140700052989696] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.108.236:8091',default bucket [2018-11-14 16:45:15,642] - [data_helper] [140700052989696] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:45:15,820] - [task] [140700052989696] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.122.201:8091',default bucket [2018-11-14 16:45:15,833] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.224 [2018-11-14 16:45:15,843] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:45:15,876] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C3 for default is 0 [2018-11-14 16:45:15,876] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.108.236 [2018-11-14 16:45:15,884] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:45:15,951] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C2 for default is 0 [2018-11-14 16:45:15,951] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.224 [2018-11-14 16:45:15,988] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C3 for bucket default is 0 [2018-11-14 16:45:15,988] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.108.236 [2018-11-14 16:45:16,052] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0 [2018-11-14 16:45:16,082] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:45:16,113] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C3, default bucket [2018-11-14 16:45:16,137] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:45:16,165] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C3, default bucket [2018-11-14 16:45:16,198] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:45:16,259] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C2, default bucket [2018-11-14 16:45:16,283] - [rest_client] [140700069775104] - INFO - http://172.23.108.236:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:45:16,340] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C2, default bucket [2018-11-14 16:45:16,401] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:45:16,485] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:45:16,664] - [task] [140700069775104] - INFO - 21111 items will be verified on default bucket [2018-11-14 16:45:17,545] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:45:24,358] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:45:31,085] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:45:31,833] - [task] [140699831887616] - INFO - 21111 items were verified in 15.1678469181 sec.the average number of ops - 1391.82547135 per second [2018-11-14 16:45:31,923] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:45:32,040] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:45:32,307] - [task] [140700069775104] - INFO - 21111 items will be verified on default bucket [2018-11-14 16:45:32,843] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:45:39,563] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:45:46,100] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:45:46,923] - [task] [140699831887616] - INFO - 21111 items were verified in 14.6158680916 sec.the average number of ops - 1444.38866124 per second [2018-11-14 16:45:46,924] - [xdcrnewbasetests] [140700069775104] - INFO - Verifying RevIds for Couchbase Cluster: C3, Master Ip: 172.23.97.224 -> Couchbase Cluster: C2, Master Ip: 172.23.108.236, bucket default [2018-11-14 16:45:46,991] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:45:47,124] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:45:47,327] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:45:47,422] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:45:47,572] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.108.236:11210 default [2018-11-14 16:45:47,694] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.122.201:11210 default [2018-11-14 16:45:47,967] - [task] [140700069775104] - INFO - RevID verification: in progress for default ... [2018-11-14 16:46:16,683] - [task] [140699831887616] - INFO - RevId Verification : 21111 existing items have been verified [2018-11-14 16:46:16,683] - [task] [140699831887616] - INFO - RevId Verification : 0 deleted items have been verified [2018-11-14 16:46:16,684] - [task] [140699831887616] - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket [2018-11-14 16:46:16,691] - [rest_client] [140699831887616] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:46:16,777] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'} [2018-11-14 16:46:16,778] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:46:16,847] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:46:16,848] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.224', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1663731265, 0, '') [2018-11-14 16:46:16,876] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'} [2018-11-14 16:46:16,876] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:46:16,986] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:46:16,987] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.225', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2502037254, 0, '') [2018-11-14 16:46:16,988] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:46:27,045] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'} [2018-11-14 16:46:27,045] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:46:27,111] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:46:27,112] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.104.104', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (987897694, 0, '') [2018-11-14 16:46:27,153] - [data_helper] [140700069775104] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'} [2018-11-14 16:46:27,154] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:46:27,256] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2018-11-14 16:46:27,257] - [cluster_helper] [140700069775104] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.161', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1344068281, 0, '') [2018-11-14 16:46:27,258] - [xdcrnewbasetests] [140700069775104] - INFO - wait for expiry pager to run on all these nodes [2018-11-14 16:46:37,733] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:46:37,818] - [task] [140699789924096] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.224:8091',default bucket [2018-11-14 16:46:37,833] - [data_helper] [140699789924096] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:46:37,913] - [task] [140699789924096] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.225:8091',default bucket [2018-11-14 16:46:38,436] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:46:38,526] - [task] [140699840280320] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.104:8091',default bucket [2018-11-14 16:46:38,542] - [data_helper] [140699840280320] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:46:38,654] - [task] [140699840280320] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.161:8091',default bucket [2018-11-14 16:46:38,677] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.224 [2018-11-14 16:46:38,685] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:46:38,716] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C3 for default is 0 [2018-11-14 16:46:38,716] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.104.104 [2018-11-14 16:46:38,724] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:46:38,762] - [xdcrnewbasetests] [140700069775104] - INFO - Current dcp queue size on C1 for default is 0 [2018-11-14 16:46:38,763] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.224 [2018-11-14 16:46:38,797] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C3 for bucket default is 0 [2018-11-14 16:46:38,797] - [xdcrnewbasetests] [140700069775104] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.104.104 [2018-11-14 16:46:38,842] - [xdcrnewbasetests] [140700069775104] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0 [2018-11-14 16:46:38,875] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:46:38,905] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C3, default bucket [2018-11-14 16:46:38,930] - [rest_client] [140700069775104] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:46:38,959] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C3, default bucket [2018-11-14 16:46:39,007] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:46:39,045] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_active_curr_items 30000 == 30000 expected on C1, default bucket [2018-11-14 16:46:39,086] - [rest_client] [140700069775104] - INFO - http://172.23.104.104:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:46:39,123] - [xdcrnewbasetests] [140700069775104] - INFO - Saw: vb_replica_curr_items 30000 == 30000 expected on C1, default bucket [2018-11-14 16:46:39,189] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:46:39,276] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:46:39,503] - [task] [140700069775104] - INFO - 21111 items will be verified on default bucket [2018-11-14 16:46:39,917] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:46:46,627] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:46:52,894] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:46:53,688] - [task] [140699831887616] - INFO - 21111 items were verified in 14.1833600998 sec.the average number of ops - 1488.43412368 per second [2018-11-14 16:46:53,758] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:46:53,848] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:46:54,074] - [task] [140700069775104] - INFO - 30000 items will be verified on default bucket [2018-11-14 16:46:54,677] - [task] [140699831887616] - INFO - 0 items were verified [2018-11-14 16:47:01,343] - [task] [140699831887616] - INFO - 10000 items were verified [2018-11-14 16:47:07,946] - [task] [140699831887616] - INFO - 20000 items were verified [2018-11-14 16:47:14,805] - [task] [140699831887616] - INFO - 30000 items were verified [2018-11-14 16:47:14,805] - [task] [140699831887616] - INFO - 30000 items were verified in 20.7298381329 sec.the average number of ops - 1447.18910332 per second [2018-11-14 16:47:14,806] - [xdcrnewbasetests] [140700069775104] - INFO - Verifying RevIds for Couchbase Cluster: C3, Master Ip: 172.23.97.224 -> Couchbase Cluster: C1, Master Ip: 172.23.104.104, bucket default [2018-11-14 16:47:14,872] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:47:14,963] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:47:15,165] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.224:11210 default [2018-11-14 16:47:15,255] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.97.225:11210 default [2018-11-14 16:47:15,451] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.104.104:11210 default [2018-11-14 16:47:15,552] - [data_helper] [140700069775104] - INFO - creating direct client 172.23.107.161:11210 default [2018-11-14 16:47:15,794] - [task] [140700069775104] - INFO - RevID verification: in progress for default ... [2018-11-14 16:47:44,301] - [task] [140699831887616] - INFO - RevId Verification : 21111 existing items have been verified [2018-11-14 16:47:44,301] - [task] [140699831887616] - INFO - RevId Verification : 0 deleted items have been verified [2018-11-14 16:47:44,302] - [task] [140699831887616] - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket [2018-11-14 16:47:44,309] - [rest_client] [140699831887616] - INFO - http://172.23.97.224:8091/pools/default/buckets/default/stats?zoom=minute [2018-11-14 16:47:44,391] - [rest_client] [140700069775104] - INFO - /diag/eval status on 172.23.104.104:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2018-11-14 16:47:44,406] - [remote_util] [140700069775104] - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: [2018-11-14 16:47:44,548] - [remote_util] [140700069775104] - INFO - Connected to 172.23.104.104 [2018-11-14 16:47:44,889] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:47:45,338] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.104.104: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:45,469] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:45,584] - [xdcrnewbasetests] [140700069775104] - INFO - 20 [2018-11-14 16:47:45,584] - [xdcrnewbasetests] [140700069775104] - INFO - ['2018-11-14T13:44:15.506-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=276 is closed by producer ', '2018-11-14T13:49:06.787-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2018-11-14T16:00:26.935-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... ', '2018-11-14T16:10:37.366-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 4456ffab85ebae06d1f1547628fad6b5/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... ', '2018-11-14T16:30:30.899-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=294 is closed by producer ', '2018-11-14T16:30:30.979-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=295 is closed by producer ', '2018-11-14T16:30:39.963-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : dcp_150d541992e12ae0265b806af08c6784/default/default_172.23.104.104:11210_0:DCP upr feed has been closed. ', '2018-11-14T16:30:39.965-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=r.update_err_ch : dcp_c0223380653196a59e4519455a6214a3/default/default_172.23.104.104:11210_0:DCP upr feed has been closed. ', '2018-11-14T16:30:50.834-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:30:50.967-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:31:00.852-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:31:00.986-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:31:10.876-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:31:11.013-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:31:20.902-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:31:21.060-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:31:30.921-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:31:31.081-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:31:40.948-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:31:41.134-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. '] [2018-11-14 16:47:45,585] - [xdcrnewbasetests] [140700069775104] - INFO - Initial replication restart count on 172.23.104.104 :0, now :20 [2018-11-14 16:47:45,585] - [xdcrnewbasetests] [140700069775104] - INFO - Number of new replication restarts this run: 20 [2018-11-14 16:47:45,585] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:44:15.506-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=276 is closed by producer [2018-11-14 16:47:45,585] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:49:06.787-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2018-11-14 16:47:45,585] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:00:26.935-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... [2018-11-14 16:47:45,586] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:10:37.366-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 4456ffab85ebae06d1f1547628fad6b5/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... [2018-11-14 16:47:45,586] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:30:30.899-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=294 is closed by producer [2018-11-14 16:47:45,586] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:30:30.979-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=295 is closed by producer [2018-11-14 16:47:45,587] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:30:39.963-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : dcp_150d541992e12ae0265b806af08c6784/default/default_172.23.104.104:11210_0:DCP upr feed has been closed. [2018-11-14 16:47:45,587] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:30:39.965-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=r.update_err_ch : dcp_c0223380653196a59e4519455a6214a3/default/default_172.23.104.104:11210_0:DCP upr feed has been closed. [2018-11-14 16:47:45,587] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:30:50.834-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,587] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:30:50.967-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,587] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:31:00.852-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,588] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:31:00.986-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,588] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:31:10.876-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,588] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:31:11.013-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,588] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:31:20.902-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,589] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:31:21.060-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,589] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:31:30.921-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,589] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:31:31.081-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,589] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:31:40.948-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,589] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:31:41.134-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c0223380653196a59e4519455a6214a3/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:45,594] - [remote_util] [140700069775104] - INFO - connecting to 172.23.107.161 with username:root password:couchbase ssh_key: [2018-11-14 16:47:45,770] - [remote_util] [140700069775104] - INFO - Connected to 172.23.107.161 [2018-11-14 16:47:46,337] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.107.161: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:46,518] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:46,583] - [xdcrnewbasetests] [140700069775104] - INFO - 7 [2018-11-14 16:47:46,583] - [xdcrnewbasetests] [140700069775104] - INFO - ['2018-11-14T13:44:05.625-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=965 is closed by producer ', '2018-11-14T13:44:17.120-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=701 is closed by producer ', '2018-11-14T13:49:08.208-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2018-11-14T14:06:48.003-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=genericPipeline.context.Start : Failed to start service CheckpointManager for eac58056414b9450f2b8c09ea48801c6/default/default. err=dial tcp 172.23.122.201:11210: getsockopt: connection refused ', '2018-11-14T16:10:37.426-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 4456ffab85ebae06d1f1547628fad6b5/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... ', '2018-11-14T16:24:25.168-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/standard_bucket_1/standard_bucket_1. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=798 is closed by producer ', '2018-11-14T16:24:25.370-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/standard_bucket_1/standard_bucket_1. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=791 is closed by producer '] [2018-11-14 16:47:46,584] - [xdcrnewbasetests] [140700069775104] - INFO - Initial replication restart count on 172.23.107.161 :0, now :7 [2018-11-14 16:47:46,584] - [xdcrnewbasetests] [140700069775104] - INFO - Number of new replication restarts this run: 7 [2018-11-14 16:47:46,584] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:44:05.625-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=965 is closed by producer [2018-11-14 16:47:46,584] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:44:17.120-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=701 is closed by producer [2018-11-14 16:47:46,585] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:49:08.208-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2018-11-14 16:47:46,585] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T14:06:48.003-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=genericPipeline.context.Start : Failed to start service CheckpointManager for eac58056414b9450f2b8c09ea48801c6/default/default. err=dial tcp 172.23.122.201:11210: getsockopt: connection refused [2018-11-14 16:47:46,585] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:10:37.426-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 4456ffab85ebae06d1f1547628fad6b5/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... [2018-11-14 16:47:46,585] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:24:25.168-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/standard_bucket_1/standard_bucket_1. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=798 is closed by producer [2018-11-14 16:47:46,585] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:24:25.370-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c0223380653196a59e4519455a6214a3/standard_bucket_1/standard_bucket_1. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=791 is closed by producer [2018-11-14 16:47:46,590] - [remote_util] [140700069775104] - INFO - connecting to 172.23.108.236 with username:root password:couchbase ssh_key: [2018-11-14 16:47:46,758] - [remote_util] [140700069775104] - INFO - Connected to 172.23.108.236 [2018-11-14 16:47:47,017] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:47:47,438] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.108.236: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:47,569] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:47,683] - [xdcrnewbasetests] [140700069775104] - INFO - 18 [2018-11-14 16:47:47,684] - [xdcrnewbasetests] [140700069775104] - INFO - ['2018-11-14T13:44:35.560-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=327 is closed by producer ', '2018-11-14T13:44:46.891-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=368 is closed by producer ', '2018-11-14T13:44:58.204-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=178 is closed by producer ', '2018-11-14T15:55:37.044-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=388 is closed by producer ', '2018-11-14T15:55:37.133-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=386 is closed by producer ', '2018-11-14T15:55:49.779-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=143 is closed by producer ', '2018-11-14T15:55:49.932-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=142 is closed by producer ', '2018-11-14T16:05:47.457-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=454 is closed by producer ', '2018-11-14T16:05:59.114-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=383 is closed by producer ', '2018-11-14T16:10:50.305-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2018-11-14T16:17:09.543-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=459 is closed by producer ', '2018-11-14T16:17:21.206-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=214 is closed by producer ', '2018-11-14T16:17:32.683-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : dcp_150d541992e12ae0265b806af08c6784/default/default_172.23.108.236:11210_0:DCP upr feed has been closed. ', '2018-11-14T16:17:43.206-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:17:53.237-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:18:03.266-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:18:13.298-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:18:23.329-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. '] [2018-11-14 16:47:47,684] - [xdcrnewbasetests] [140700069775104] - INFO - Initial replication restart count on 172.23.108.236 :0, now :18 [2018-11-14 16:47:47,685] - [xdcrnewbasetests] [140700069775104] - INFO - Number of new replication restarts this run: 18 [2018-11-14 16:47:47,685] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:44:35.560-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=327 is closed by producer [2018-11-14 16:47:47,685] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:44:46.891-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=368 is closed by producer [2018-11-14 16:47:47,685] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:44:58.204-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=178 is closed by producer [2018-11-14 16:47:47,685] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T15:55:37.044-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=388 is closed by producer [2018-11-14 16:47:47,686] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T15:55:37.133-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=386 is closed by producer [2018-11-14 16:47:47,686] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T15:55:49.779-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=143 is closed by producer [2018-11-14 16:47:47,686] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T15:55:49.932-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=142 is closed by producer [2018-11-14 16:47:47,686] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:05:47.457-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=454 is closed by producer [2018-11-14 16:47:47,686] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:05:59.114-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=383 is closed by producer [2018-11-14 16:47:47,687] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:10:50.305-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2018-11-14 16:47:47,687] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:17:09.543-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=459 is closed by producer [2018-11-14 16:47:47,687] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:17:21.206-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=214 is closed by producer [2018-11-14 16:47:47,687] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:17:32.683-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : dcp_150d541992e12ae0265b806af08c6784/default/default_172.23.108.236:11210_0:DCP upr feed has been closed. [2018-11-14 16:47:47,688] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:17:43.206-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:47,688] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:17:53.237-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:47,688] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:18:03.266-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:47,688] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:18:13.298-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:47,688] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:18:23.329-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:47,693] - [remote_util] [140700069775104] - INFO - connecting to 172.23.122.201 with username:root password:couchbase ssh_key: [2018-11-14 16:47:47,960] - [remote_util] [140700069775104] - INFO - Connected to 172.23.122.201 [2018-11-14 16:47:48,375] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:47:48,938] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.122.201: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:49,121] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:49,236] - [xdcrnewbasetests] [140700069775104] - INFO - 7 [2018-11-14 16:47:49,236] - [xdcrnewbasetests] [140700069775104] - INFO - ['2018-11-14T13:44:35.813-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=946 is closed by producer ', '2018-11-14T13:44:47.843-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=854 is closed by producer ', '2018-11-14T13:49:39.799-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2018-11-14T16:00:27.204-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2018-11-14T16:00:27.223-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2018-11-14T16:05:47.692-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=943 is closed by producer ', '2018-11-14T16:10:51.155-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... '] [2018-11-14 16:47:49,236] - [xdcrnewbasetests] [140700069775104] - INFO - Initial replication restart count on 172.23.122.201 :0, now :7 [2018-11-14 16:47:49,237] - [xdcrnewbasetests] [140700069775104] - INFO - Number of new replication restarts this run: 7 [2018-11-14 16:47:49,237] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:44:35.813-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=946 is closed by producer [2018-11-14 16:47:49,237] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:44:47.843-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=854 is closed by producer [2018-11-14 16:47:49,237] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:49:39.799-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2018-11-14 16:47:49,238] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:00:27.204-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2018-11-14 16:47:49,238] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:00:27.223-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2018-11-14 16:47:49,238] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:05:47.692-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=943 is closed by producer [2018-11-14 16:47:49,238] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:10:51.155-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2018-11-14 16:47:49,242] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.224 with username:root password:couchbase ssh_key: [2018-11-14 16:47:49,347] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.224 [2018-11-14 16:47:49,565] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:47:50,002] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.224: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:50,055] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:50,169] - [xdcrnewbasetests] [140700069775104] - INFO - 1 [2018-11-14 16:47:50,170] - [xdcrnewbasetests] [140700069775104] - INFO - ['2018-11-14T16:00:27.294-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... '] [2018-11-14 16:47:50,170] - [xdcrnewbasetests] [140700069775104] - INFO - Initial replication restart count on 172.23.97.224 :0, now :1 [2018-11-14 16:47:50,170] - [xdcrnewbasetests] [140700069775104] - INFO - Number of new replication restarts this run: 1 [2018-11-14 16:47:50,170] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:00:27.294-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... [2018-11-14 16:47:50,175] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.225 with username:root password:couchbase ssh_key: [2018-11-14 16:47:50,279] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.225 [2018-11-14 16:47:50,476] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:47:50,880] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.225: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:50,929] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:51,044] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:47:51,044] - [xdcrnewbasetests] [140700069775104] - INFO - [] [2018-11-14 16:47:51,044] - [xdcrnewbasetests] [140700069775104] - INFO - Initial replication restart count on 172.23.97.225 :0, now :0 [2018-11-14 16:47:51,049] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2018-11-14 16:47:51,189] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.164 [2018-11-14 16:47:51,417] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:47:51,791] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.164: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:51,875] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:51,989] - [xdcrnewbasetests] [140700069775104] - INFO - 2 [2018-11-14 16:47:51,990] - [xdcrnewbasetests] [140700069775104] - INFO - ['2018-11-14T13:44:33.099-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(a4652e8e0ed665ac17bcbb15f794279a/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T13:49:23.198-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... '] [2018-11-14 16:47:51,990] - [xdcrnewbasetests] [140700069775104] - INFO - Initial replication restart count on 172.23.98.164 :0, now :2 [2018-11-14 16:47:51,990] - [xdcrnewbasetests] [140700069775104] - INFO - Number of new replication restarts this run: 2 [2018-11-14 16:47:51,991] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:44:33.099-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(a4652e8e0ed665ac17bcbb15f794279a/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:51,991] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:49:23.198-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline a4652e8e0ed665ac17bcbb15f794279a/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2018-11-14 16:47:51,997] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.168 with username:root password:couchbase ssh_key: [2018-11-14 16:47:52,136] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.168 [2018-11-14 16:47:52,328] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:47:52,814] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.168: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:52,913] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:53,028] - [xdcrnewbasetests] [140700069775104] - INFO - 4 [2018-11-14 16:47:53,028] - [xdcrnewbasetests] [140700069775104] - INFO - ['2018-11-14T13:44:10.963-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(eac58056414b9450f2b8c09ea48801c6/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T13:49:01.209-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2018-11-14T16:05:47.373-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2018-11-14T16:10:37.507-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... '] [2018-11-14 16:47:53,028] - [xdcrnewbasetests] [140700069775104] - INFO - Initial replication restart count on 172.23.98.168 :0, now :4 [2018-11-14 16:47:53,029] - [xdcrnewbasetests] [140700069775104] - INFO - Number of new replication restarts this run: 4 [2018-11-14 16:47:53,029] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:44:10.963-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(eac58056414b9450f2b8c09ea48801c6/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:53,029] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T13:49:01.209-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline eac58056414b9450f2b8c09ea48801c6/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2018-11-14 16:47:53,029] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:05:47.373-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(150d541992e12ae0265b806af08c6784/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2018-11-14 16:47:53,030] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T16:10:37.507-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 150d541992e12ae0265b806af08c6784/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2018-11-14 16:47:53,043] - [rest_client] [140700069775104] - INFO - /diag/eval status on 172.23.104.104:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2018-11-14 16:47:53,048] - [remote_util] [140700069775104] - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: [2018-11-14 16:47:53,187] - [remote_util] [140700069775104] - INFO - Connected to 172.23.104.104 [2018-11-14 16:47:53,486] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:47:53,973] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.104.104: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:54,103] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:54,218] - [xdcrnewbasetests] [140700069775104] - INFO - 1 [2018-11-14 16:47:54,218] - [xdcrnewbasetests] [140700069775104] - INFO - Initial panic: count on 172.23.104.104 :0, now :1 [2018-11-14 16:47:54,223] - [remote_util] [140700069775104] - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: [2018-11-14 16:47:54,364] - [remote_util] [140700069775104] - INFO - Connected to 172.23.104.104 [2018-11-14 16:47:54,697] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:47:55,202] - [rest_client] [140700069775104] - INFO - /diag/eval status on 172.23.104.104:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2018-11-14 16:47:55,203] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.104.104: zgrep -A 40 'panic:' "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:55,300] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:55,301] - [xdcrnewbasetests] [140700069775104] - INFO - panic: interface conversion: interface {} is nil, not string [2018-11-14 16:47:55,301] - [xdcrnewbasetests] [140700069775104] - INFO - [2018-11-14 16:47:55,301] - [xdcrnewbasetests] [140700069775104] - INFO - goroutine 69 [running]: [2018-11-14 16:47:55,301] - [xdcrnewbasetests] [140700069775104] - INFO - github.com/couchbase/goxdcr/metadata.ReplicationSettingsMap.CloneAndRedact(0xc4261f62a0, 0xc4261f6300) [2018-11-14 16:47:55,302] - [xdcrnewbasetests] [140700069775104] - INFO - goproj/src/github.com/couchbase/goxdcr/metadata/replication_settings.go:530 +0x40f [2018-11-14 16:47:55,302] - [xdcrnewbasetests] [140700069775104] - INFO - github.com/couchbase/goxdcr/replication_manager.(*Adminport).doCreateReplicationRequest(0xc4201946c0, 0xc422ef7000, 0x0, 0x0, 0x0) [2018-11-14 16:47:55,302] - [xdcrnewbasetests] [140700069775104] - INFO - goproj/src/github.com/couchbase/goxdcr/replication_manager/adminport.go:409 +0x298 [2018-11-14 16:47:55,302] - [xdcrnewbasetests] [140700069775104] - INFO - github.com/couchbase/goxdcr/replication_manager.(*Adminport).handleRequest(0xc4201946c0, 0xc422ef7000, 0xc4264ba280, 0xcc39e0, 0xc4264ba280) [2018-11-14 16:47:55,302] - [xdcrnewbasetests] [140700069775104] - INFO - goproj/src/github.com/couchbase/goxdcr/replication_manager/adminport.go:172 +0x1093 [2018-11-14 16:47:55,303] - [xdcrnewbasetests] [140700069775104] - INFO - github.com/couchbase/goxdcr/replication_manager.(*Adminport).processRequest(0xc4201946c0, 0xc425b05ee0, 0x1, 0x1, 0xcfafc0, 0xc425b05f10) [2018-11-14 16:47:55,303] - [xdcrnewbasetests] [140700069775104] - INFO - goproj/src/github.com/couchbase/goxdcr/replication_manager/adminport.go:136 +0xf5 [2018-11-14 16:47:55,303] - [xdcrnewbasetests] [140700069775104] - INFO - github.com/couchbase/goxdcr/replication_manager.(*Adminport).(github.com/couchbase/goxdcr/replication_manager.processRequest)-fm(0xc425b05ee0, 0x1, 0x1, 0x1, 0x1) [2018-11-14 16:47:55,303] - [xdcrnewbasetests] [140700069775104] - INFO - goproj/src/github.com/couchbase/goxdcr/replication_manager/adminport.go:69 +0x48 [2018-11-14 16:47:55,303] - [xdcrnewbasetests] [140700069775104] - INFO - github.com/couchbase/goxdcr/gen_server.(*GenServer).run(0xc4201946d8) [2018-11-14 16:47:55,304] - [xdcrnewbasetests] [140700069775104] - INFO - goproj/src/github.com/couchbase/goxdcr/gen_server/gen_server.go:104 +0x4a4 [2018-11-14 16:47:55,304] - [xdcrnewbasetests] [140700069775104] - INFO - created by github.com/couchbase/goxdcr/gen_server.(*GenServer).Start_server [2018-11-14 16:47:55,304] - [xdcrnewbasetests] [140700069775104] - INFO - goproj/src/github.com/couchbase/goxdcr/gen_server/gen_server.go:74 +0xba [2018-11-14 16:47:55,304] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.807-08:00 INFO GOXDCR.ReplMgr: GOMAXPROCS=4 [2018-11-14 16:47:55,304] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.810-08:00 INFO GOXDCR.AuditSvc: Created audit service. [2018-11-14 16:47:55,305] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.810-08:00 INFO GOXDCR.UILogSvc: Created ui log service. [2018-11-14 16:47:55,305] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.817-08:00 INFO GOXDCR.RemClusterSvc: Agent remoteCluster/zNCmdrwxKSnlN12yyBL4HglKmOHwGKdZt0KVI3xXPZ0= remote_cluster_C1-C2 started for cluster: eac58056414b9450f2b8c09ea48801c6 [2018-11-14 16:47:55,305] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.817-08:00 INFO GOXDCR.ReplSpecSvc: Cache has been initialized for ReplicationSpecService [2018-11-14 16:47:55,305] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.821-08:00 INFO GOXDCR.IntSettSvc: Internal settings not found. Using default values [2018-11-14 16:47:55,306] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.821-08:00 INFO GOXDCR.ReplMgr: pollEOF: About to start stdin polling [2018-11-14 16:47:55,306] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.822-08:00 INFO GOXDCR.ReplMgr: XDCR internal settings: map[MaxCheckpointRecordsToRead:5 DefaultHttpTimeout:180 MaxRetryForLiveUpdatePipeline:5 RetryIntervalTargetStats:1000 XmemMaxDataChanSize:10485760 NumberOfSlotsForBandwidthThrottling:10 CapiWriteTimeout:10 MaxNumOfMetakvRetries:4 CapiDataChanSizeMultiplier:1 AuditWriteTimeout:1 UprFeedDataChanLength:20000 TimeoutDcpCloseUprStreams:3 MemStatsLogInterval:120 ShortHttpTimeout:20 CapiBatchTimeout:180 MaxCheckpointRecordsToKeep:5 ReplSpecCheckInterval:15 StatsLogInterval:30 WaitTimeForLiveUpdatePipeline:2000 MaxTopologyStableCountBeforeRestart:30 XmemDefaultRespTimeout:1000 AuditReadTimeout:1 ReplicationSpecGCCnt:4 KeepAlivePeriod:30 MaxNumberOfAsyncListeners:4 TimeoutPartsStop:10 TimeoutCheckpointBeforeStop:180 RetryIntervalMetakv:500 XmemMaxReadDownTime:60 TimeoutRuntimeContextStart:30 ThresholdPercentageForEventChanSizeLogging:90 XmemMaxRetryInterval:300 XmemMaxBatchSize:50 WaitTimeBetweenMetadataChangeListeners:1000 XmemBackoffTimeNewConn:1000 TimeoutDcpCloseUprFeed:3 BypassSanInCertificateCheck:0 CapiMaxRetryBatchUpdateDocs:6 XmemSelfMonitorInterval:6 HELOTimeout:120 TimeoutRuntimeContextStop:5 MaxRetryTargetStats:6 TimeoutPartsStart:30 MaxLengthSnapshotHistory:200 MaxTopologyChangeCountBeforeRestart:30 XmemWriteTimeout:120 MaxRetryCapiService:5 RefreshRemoteClusterRefInterval:15 XmemMaxIdleCount:60 MaxWorkersForCheckpointing:5 XmemReadTimeout:120 XmemBackoffWaitTime:10 XmemMaxRetryNewConn:10 CapiRetryInterval:500 XmemMaxRetry:5 TopologyChangeCheckInterval:10 PercentageOfBytesToSendAsMin:30 CapiReadTimeout:60 UprFeedBufferSize:1048576 ThresholdForThroughSeqnoComputation:100] [2018-11-14 16:47:55,306] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.822-08:00 INFO GOXDCR.PipelineMgr: Pipeline Manager is constucted [2018-11-14 16:47:55,306] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.822-08:00 INFO GOXDCR.ReplMgr: Replication manager is initialized [2018-11-14 16:47:55,307] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.822-08:00 INFO GOXDCR.GenericSupervisor: Starting supervisor ReplicationManagerSupervisor. [2018-11-14 16:47:55,307] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.822-08:00 INFO GOXDCR.GenericSupervisor: Started supervisor ReplicationManagerSupervisor. [2018-11-14 16:47:55,307] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.822-08:00 INFO GOXDCR.ReplMgr: initReplications succeeded [2018-11-14 16:47:55,307] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.822-08:00 INFO GOXDCR.ReplMgr: upgradeRemoteClusterRefs started. [2018-11-14 16:47:55,307] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.822-08:00 INFO GOXDCR.ReplMgr: upgradeRemoteClusterRefs exited [2018-11-14 16:47:55,308] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.822-08:00 INFO GOXDCR.GlobalSettingChangeListener: Started MetakvChangeListener GlobalSettingChangeListener [2018-11-14 16:47:55,308] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.822-08:00 INFO GOXDCR.ReplMgr: checkReplicationStatus started. [2018-11-14 16:47:55,308] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:35.822-08:00 INFO GOXDCR.ReplMgr: logMemStats started. [2018-11-14 16:47:55,309] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:36.822-08:00 INFO GOXDCR.InternalSettingChangeListener: Started MetakvChangeListener InternalSettingsChangeListener [2018-11-14 16:47:55,309] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:37.823-08:00 INFO GOXDCR.RemoteClusterChangeListener: Started MetakvChangeListener RemoteClusterChangeListener [2018-11-14 16:47:55,309] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:37.825-08:00 INFO GOXDCR.RemoteClusterChangeListener: metakvCallback called on listener RemoteClusterChangeListener with path = /remoteCluster/zNCmdrwxKSnlN12yyBL4HglKmOHwGKdZt0KVI3xXPZ0= [2018-11-14 16:47:55,310] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:37.825-08:00 INFO GOXDCR.RemClusterSvc: metakvCallback called on path = /remoteCluster/zNCmdrwxKSnlN12yyBL4HglKmOHwGKdZt0KVI3xXPZ0= [2018-11-14 16:47:55,310] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:38.823-08:00 INFO GOXDCR.ReplicationSpecChangeListener: Started MetakvChangeListener ReplicationSpecChangeListener [2018-11-14 16:47:55,310] - [xdcrnewbasetests] [140700069775104] - INFO - 2018-11-14T12:50:39.823-08:00 INFO GOXDCR.AdminPort: Constructed adminport [2018-11-14 16:47:55,429] - [remote_util] [140700069775104] - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: [2018-11-14 16:47:55,569] - [remote_util] [140700069775104] - INFO - Connected to 172.23.104.104 [2018-11-14 16:47:55,839] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:47:56,353] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.104.104: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:56,483] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:56,598] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:47:56,598] - [xdcrnewbasetests] [140700069775104] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.104.104 :0, now :0 [2018-11-14 16:47:56,603] - [remote_util] [140700069775104] - INFO - connecting to 172.23.107.161 with username:root password:couchbase ssh_key: [2018-11-14 16:47:56,782] - [remote_util] [140700069775104] - INFO - Connected to 172.23.107.161 [2018-11-14 16:47:57,362] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.107.161: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:57,543] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:57,607] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:47:57,607] - [xdcrnewbasetests] [140700069775104] - INFO - Initial panic: count on 172.23.107.161 :0, now :0 [2018-11-14 16:47:57,612] - [remote_util] [140700069775104] - INFO - connecting to 172.23.107.161 with username:root password:couchbase ssh_key: [2018-11-14 16:47:57,751] - [remote_util] [140700069775104] - INFO - Connected to 172.23.107.161 [2018-11-14 16:47:58,328] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.107.161: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:58,477] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:58,541] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:47:58,542] - [xdcrnewbasetests] [140700069775104] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.161 :0, now :0 [2018-11-14 16:47:58,548] - [remote_util] [140700069775104] - INFO - connecting to 172.23.108.236 with username:root password:couchbase ssh_key: [2018-11-14 16:47:58,716] - [remote_util] [140700069775104] - INFO - Connected to 172.23.108.236 [2018-11-14 16:47:58,972] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:47:59,404] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.108.236: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:47:59,534] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:47:59,649] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:47:59,650] - [xdcrnewbasetests] [140700069775104] - INFO - Initial panic: count on 172.23.108.236 :0, now :0 [2018-11-14 16:47:59,656] - [remote_util] [140700069775104] - INFO - connecting to 172.23.108.236 with username:root password:couchbase ssh_key: [2018-11-14 16:47:59,799] - [remote_util] [140700069775104] - INFO - Connected to 172.23.108.236 [2018-11-14 16:48:00,071] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:00,491] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.108.236: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:48:00,621] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:48:00,736] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:48:00,736] - [xdcrnewbasetests] [140700069775104] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.108.236 :0, now :0 [2018-11-14 16:48:00,742] - [remote_util] [140700069775104] - INFO - connecting to 172.23.122.201 with username:root password:couchbase ssh_key: [2018-11-14 16:48:01,011] - [remote_util] [140700069775104] - INFO - Connected to 172.23.122.201 [2018-11-14 16:48:01,430] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:01,996] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.122.201: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:48:02,178] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:48:02,292] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:48:02,293] - [xdcrnewbasetests] [140700069775104] - INFO - Initial panic: count on 172.23.122.201 :0, now :0 [2018-11-14 16:48:02,297] - [remote_util] [140700069775104] - INFO - connecting to 172.23.122.201 with username:root password:couchbase ssh_key: [2018-11-14 16:48:02,538] - [remote_util] [140700069775104] - INFO - Connected to 172.23.122.201 [2018-11-14 16:48:02,933] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:03,509] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.122.201: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:48:03,642] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:48:03,757] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:48:03,757] - [xdcrnewbasetests] [140700069775104] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.122.201 :0, now :0 [2018-11-14 16:48:03,762] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.224 with username:root password:couchbase ssh_key: [2018-11-14 16:48:03,867] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.224 [2018-11-14 16:48:04,086] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:04,481] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.224: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:48:04,564] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:48:04,679] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:48:04,679] - [xdcrnewbasetests] [140700069775104] - INFO - Initial panic: count on 172.23.97.224 :0, now :0 [2018-11-14 16:48:04,683] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.224 with username:root password:couchbase ssh_key: [2018-11-14 16:48:04,786] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.224 [2018-11-14 16:48:04,981] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:05,350] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.224: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:48:05,432] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:48:05,546] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:48:05,547] - [xdcrnewbasetests] [140700069775104] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.97.224 :0, now :0 [2018-11-14 16:48:05,552] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.225 with username:root password:couchbase ssh_key: [2018-11-14 16:48:05,688] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.225 [2018-11-14 16:48:05,875] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:06,250] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.225: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:48:06,333] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:48:06,448] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:48:06,448] - [xdcrnewbasetests] [140700069775104] - INFO - Initial panic: count on 172.23.97.225 :0, now :0 [2018-11-14 16:48:06,452] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.225 with username:root password:couchbase ssh_key: [2018-11-14 16:48:06,555] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.225 [2018-11-14 16:48:06,751] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:07,146] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.225: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:48:07,228] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:48:07,343] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:48:07,343] - [xdcrnewbasetests] [140700069775104] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.97.225 :0, now :0 [2018-11-14 16:48:07,348] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2018-11-14 16:48:07,456] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.164 [2018-11-14 16:48:07,717] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:08,181] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.164: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:48:08,264] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:48:08,379] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:48:08,380] - [xdcrnewbasetests] [140700069775104] - INFO - Initial panic: count on 172.23.98.164 :0, now :0 [2018-11-14 16:48:08,386] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2018-11-14 16:48:08,525] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.164 [2018-11-14 16:48:08,785] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:09,284] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.164: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:48:09,334] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:48:09,448] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:48:09,449] - [xdcrnewbasetests] [140700069775104] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.98.164 :0, now :0 [2018-11-14 16:48:09,456] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.168 with username:root password:couchbase ssh_key: [2018-11-14 16:48:09,594] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.168 [2018-11-14 16:48:09,760] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:10,253] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.168: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:48:10,352] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:48:10,466] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:48:10,467] - [xdcrnewbasetests] [140700069775104] - INFO - Initial panic: count on 172.23.98.168 :0, now :0 [2018-11-14 16:48:10,475] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.168 with username:root password:couchbase ssh_key: [2018-11-14 16:48:10,611] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.168 [2018-11-14 16:48:10,807] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:11,299] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.168: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2018-11-14 16:48:11,398] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:48:11,512] - [xdcrnewbasetests] [140700069775104] - INFO - 0 [2018-11-14 16:48:11,513] - [xdcrnewbasetests] [140700069775104] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.98.168 :0, now :0 [2018-11-14 16:48:11,513] - [xdcrnewbasetests] [140700069775104] - ERROR - ['panic: found on 172.23.104.104'] [2018-11-14 16:48:11,514] - [xdcrnewbasetests] [140700069775104] - INFO - Collecting logs @ 172.23.104.104 [2018-11-14 16:48:11,520] - [remote_util] [140700069775104] - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: [2018-11-14 16:48:11,668] - [remote_util] [140700069775104] - INFO - Connected to 172.23.104.104 [2018-11-14 16:48:11,935] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:48:12,450] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.104.104: /opt/couchbase/bin/cbcollect_info 172.23.104.104-20181114-1648-diag.zip [2018-11-14 16:49:30,307] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:49:32,672] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.104.104: rm -f /root/172.23.104.104-20181114-1648-diag.zip [2018-11-14 16:49:32,755] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:49:32,877] - [remote_util] [140700069775104] - INFO - connecting to 172.23.104.104 with username:root password:couchbase ssh_key: [2018-11-14 16:49:33,020] - [remote_util] [140700069775104] - INFO - Connected to 172.23.104.104 [2018-11-14 16:49:33,358] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:49:33,821] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.104.104: tar -zcvf 172.23.104.104-11142018-1649-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-11-14 16:49:35,023] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:49:35,681] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.104.104: rm -f /root/172.23.104.104-11142018-1649-couch.tar.gz [2018-11-14 16:49:35,764] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:49:35,878] - [xdcrnewbasetests] [140700069775104] - INFO - Collecting logs @ 172.23.107.161 [2018-11-14 16:49:35,885] - [remote_util] [140700069775104] - INFO - connecting to 172.23.107.161 with username:root password:couchbase ssh_key: [2018-11-14 16:49:36,053] - [remote_util] [140700069775104] - INFO - Connected to 172.23.107.161 [2018-11-14 16:49:36,682] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.107.161: /opt/couchbase/bin/cbcollect_info 172.23.107.161-20181114-1649-diag.zip [2018-11-14 16:51:01,225] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:51:03,165] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.107.161: rm -f /root/172.23.107.161-20181114-1649-diag.zip [2018-11-14 16:51:03,204] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:51:03,333] - [remote_util] [140700069775104] - INFO - connecting to 172.23.107.161 with username:root password:couchbase ssh_key: [2018-11-14 16:51:03,474] - [remote_util] [140700069775104] - INFO - Connected to 172.23.107.161 [2018-11-14 16:51:04,105] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.107.161: tar -zcvf 172.23.107.161-11142018-1651-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-11-14 16:51:05,057] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:51:05,473] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.107.161: rm -f /root/172.23.107.161-11142018-1651-couch.tar.gz [2018-11-14 16:51:05,496] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:51:05,611] - [xdcrnewbasetests] [140700069775104] - INFO - Collecting logs @ 172.23.108.236 [2018-11-14 16:51:05,615] - [remote_util] [140700069775104] - INFO - connecting to 172.23.108.236 with username:root password:couchbase ssh_key: [2018-11-14 16:51:05,783] - [remote_util] [140700069775104] - INFO - Connected to 172.23.108.236 [2018-11-14 16:51:06,040] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:51:06,546] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.108.236: /opt/couchbase/bin/cbcollect_info 172.23.108.236-20181114-1651-diag.zip [2018-11-14 16:52:33,597] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:52:35,194] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.108.236: rm -f /root/172.23.108.236-20181114-1651-diag.zip [2018-11-14 16:52:35,293] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:52:35,413] - [remote_util] [140700069775104] - INFO - connecting to 172.23.108.236 with username:root password:couchbase ssh_key: [2018-11-14 16:52:35,582] - [remote_util] [140700069775104] - INFO - Connected to 172.23.108.236 [2018-11-14 16:52:35,852] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:52:36,372] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.108.236: tar -zcvf 172.23.108.236-11142018-1652-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-11-14 16:52:37,674] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:52:37,983] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.108.236: rm -f /root/172.23.108.236-11142018-1652-couch.tar.gz [2018-11-14 16:52:38,005] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:52:38,119] - [xdcrnewbasetests] [140700069775104] - INFO - Collecting logs @ 172.23.122.201 [2018-11-14 16:52:38,124] - [remote_util] [140700069775104] - INFO - connecting to 172.23.122.201 with username:root password:couchbase ssh_key: [2018-11-14 16:52:38,391] - [remote_util] [140700069775104] - INFO - Connected to 172.23.122.201 [2018-11-14 16:52:38,844] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:52:39,435] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.122.201: /opt/couchbase/bin/cbcollect_info 172.23.122.201-20181114-1652-diag.zip [2018-11-14 16:55:15,065] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:55:16,752] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.122.201: rm -f /root/172.23.122.201-20181114-1652-diag.zip [2018-11-14 16:55:16,828] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:55:16,947] - [remote_util] [140700069775104] - INFO - connecting to 172.23.122.201 with username:root password:couchbase ssh_key: [2018-11-14 16:55:17,215] - [remote_util] [140700069775104] - INFO - Connected to 172.23.122.201 [2018-11-14 16:55:17,657] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:55:18,283] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.122.201: tar -zcvf 172.23.122.201-11142018-1655-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-11-14 16:55:20,758] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:55:21,165] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.122.201: rm -f /root/172.23.122.201-11142018-1655-couch.tar.gz [2018-11-14 16:55:21,242] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:55:21,356] - [xdcrnewbasetests] [140700069775104] - INFO - Collecting logs @ 172.23.97.224 [2018-11-14 16:55:21,361] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.224 with username:root password:couchbase ssh_key: [2018-11-14 16:55:21,470] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.224 [2018-11-14 16:55:21,690] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:55:22,088] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.224: /opt/couchbase/bin/cbcollect_info 172.23.97.224-20181114-1655-diag.zip [2018-11-14 16:56:30,728] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:56:31,384] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.224: rm -f /root/172.23.97.224-20181114-1655-diag.zip [2018-11-14 16:56:31,466] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:56:31,585] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.224 with username:root password:couchbase ssh_key: [2018-11-14 16:56:31,689] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.224 [2018-11-14 16:56:31,913] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:56:32,334] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.224: tar -zcvf 172.23.97.224-11142018-1656-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-11-14 16:56:33,135] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:56:33,697] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.224: rm -f /root/172.23.97.224-11142018-1656-couch.tar.gz [2018-11-14 16:56:33,780] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:56:33,894] - [xdcrnewbasetests] [140700069775104] - INFO - Collecting logs @ 172.23.97.225 [2018-11-14 16:56:33,899] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.225 with username:root password:couchbase ssh_key: [2018-11-14 16:56:34,006] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.225 [2018-11-14 16:56:34,231] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:56:34,608] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.225: /opt/couchbase/bin/cbcollect_info 172.23.97.225-20181114-1656-diag.zip [2018-11-14 16:57:43,128] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:57:43,619] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.225: rm -f /root/172.23.97.225-20181114-1656-diag.zip [2018-11-14 16:57:43,701] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:57:43,820] - [remote_util] [140700069775104] - INFO - connecting to 172.23.97.225 with username:root password:couchbase ssh_key: [2018-11-14 16:57:43,924] - [remote_util] [140700069775104] - INFO - Connected to 172.23.97.225 [2018-11-14 16:57:44,150] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:57:44,601] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.225: tar -zcvf 172.23.97.225-11142018-1657-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-11-14 16:57:45,403] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:57:45,882] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.97.225: rm -f /root/172.23.97.225-11142018-1657-couch.tar.gz [2018-11-14 16:57:45,967] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:57:46,082] - [xdcrnewbasetests] [140700069775104] - INFO - Collecting logs @ 172.23.98.164 [2018-11-14 16:57:46,086] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2018-11-14 16:57:46,198] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.164 [2018-11-14 16:57:46,524] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:57:46,963] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.164: /opt/couchbase/bin/cbcollect_info 172.23.98.164-20181114-1657-diag.zip [2018-11-14 16:58:54,932] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:58:55,375] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.164: rm -f /root/172.23.98.164-20181114-1657-diag.zip [2018-11-14 16:58:55,458] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:58:55,577] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2018-11-14 16:58:55,683] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.164 [2018-11-14 16:58:55,943] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:58:56,361] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.164: tar -zcvf 172.23.98.164-11142018-1658-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-11-14 16:58:56,411] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:58:56,669] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.164: rm -f /root/172.23.98.164-11142018-1658-couch.tar.gz [2018-11-14 16:58:56,743] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 16:58:56,857] - [xdcrnewbasetests] [140700069775104] - INFO - Collecting logs @ 172.23.98.168 [2018-11-14 16:58:56,864] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.168 with username:root password:couchbase ssh_key: [2018-11-14 16:58:57,008] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.168 [2018-11-14 16:58:57,174] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 16:58:57,662] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.168: /opt/couchbase/bin/cbcollect_info 172.23.98.168-20181114-1658-diag.zip [2018-11-14 17:00:11,050] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 17:00:11,542] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.168: rm -f /root/172.23.98.168-20181114-1658-diag.zip [2018-11-14 17:00:11,625] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 17:00:11,744] - [remote_util] [140700069775104] - INFO - connecting to 172.23.98.168 with username:root password:couchbase ssh_key: [2018-11-14 17:00:11,882] - [remote_util] [140700069775104] - INFO - Connected to 172.23.98.168 [2018-11-14 17:00:12,112] - [remote_util] [140700069775104] - INFO - It could be other operating system. Go to check at other location [2018-11-14 17:00:12,552] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.168: tar -zcvf 172.23.98.168-11142018-170-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-11-14 17:00:12,602] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 17:00:12,805] - [remote_util] [140700069775104] - INFO - running command.raw on 172.23.98.168: rm -f /root/172.23.98.168-11142018-170-couch.tar.gz [2018-11-14 17:00:12,888] - [remote_util] [140700069775104] - INFO - command executed successfully [2018-11-14 17:00:13,084] - [xdcrnewbasetests] [140700069775104] - INFO - ==== XDCRNewbasetests cleanup is started for test #34 load_with_ops ==== [2018-11-14 17:00:13,085] - [xdcrnewbasetests] [140700069775104] - INFO - removing xdcr/nodes settings [2018-11-14 17:00:13,101] - [rest_client] [140700069775104] - INFO - Deleting replication /controller/cancelXDCR/150d541992e12ae0265b806af08c6784%2Fdefault%2Fdefault [2018-11-14 17:00:13,114] - [rest_client] [140700069775104] - INFO - Replication deleted successfully [2018-11-14 17:00:13,114] - [rest_client] [140700069775104] - INFO - Deleting replication /controller/cancelXDCR/c0223380653196a59e4519455a6214a3%2Fdefault%2Fdefault [2018-11-14 17:00:13,123] - [rest_client] [140700069775104] - INFO - Replication deleted successfully [2018-11-14 17:00:13,132] - [rest_client] [140700069775104] - INFO - removing remote cluster name:remote_cluster_C1-C2 [2018-11-14 17:00:13,142] - [rest_client] [140700069775104] - INFO - removing remote cluster name:remote_cluster_C1-C3 [2018-11-14 17:00:13,173] - [xdcrnewbasetests] [140700069775104] - INFO - cleanup [ip:172.23.104.104 port:8091 ssh_username:root, ip:172.23.107.161 port:8091 ssh_username:root] [2018-11-14 17:00:13,200] - [bucket_helper] [140700069775104] - INFO - deleting existing buckets [u'default'] on 172.23.104.104 [2018-11-14 17:00:13,200] - [bucket_helper] [140700069775104] - INFO - remove bucket default ... [2018-11-14 17:00:14,472] - [bucket_helper] [140700069775104] - INFO - deleted bucket : default from 172.23.104.104 [2018-11-14 17:00:14,473] - [bucket_helper] [140700069775104] - INFO - waiting for bucket deletion to complete.... [2018-11-14 17:00:14,480] - [rest_client] [140700069775104] - INFO - node 172.23.104.104 existing buckets : [] [2018-11-14 17:00:14,480] - [bucket_helper] [140700069775104] - INFO - sleep 2 seconds to make sure all buckets ([u'default']) were deleted completely. [2018-11-14 17:00:16,512] - [cluster_helper] [140700069775104] - INFO - rebalancing all nodes in order to remove nodes [2018-11-14 17:00:16,518] - [rest_client] [140700069775104] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.107.161', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.104,ns_1@172.23.107.161'} [2018-11-14 17:00:16,525] - [rest_client] [140700069775104] - INFO - rebalance operation started [2018-11-14 17:00:16,533] - [rest_client] [140700069775104] - INFO - rebalance percentage : 0.00 % [2018-11-14 17:00:36,567] - [rest_client] [140700069775104] - INFO - rebalance progress took 20.04 seconds [2018-11-14 17:00:36,568] - [rest_client] [140700069775104] - INFO - sleep for 10 seconds after rebalance... [2018-11-14 17:00:46,593] - [cluster_helper] [140700069775104] - INFO - removed all the nodes from cluster associated with ip:172.23.104.104 port:8091 ssh_username:root ? [(u'ns_1@172.23.107.161', 8091)] [2018-11-14 17:00:46,600] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.104.104:8091 [2018-11-14 17:00:46,606] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.104.104:8091 is running [2018-11-14 17:00:46,654] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.107.161:8091 [2018-11-14 17:00:46,662] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.107.161:8091 is running [2018-11-14 17:00:46,662] - [xdcrnewbasetests] [140700069775104] - INFO - removing xdcr/nodes settings [2018-11-14 17:00:46,679] - [rest_client] [140700069775104] - INFO - Deleting replication /controller/cancelXDCR/150d541992e12ae0265b806af08c6784%2Fdefault%2Fdefault [2018-11-14 17:00:46,693] - [rest_client] [140700069775104] - INFO - Replication deleted successfully [2018-11-14 17:00:46,704] - [rest_client] [140700069775104] - INFO - removing remote cluster name:remote_cluster_C2-C1 [2018-11-14 17:00:46,720] - [rest_client] [140700069775104] - INFO - removing remote cluster name:remote_cluster_C2-C3 [2018-11-14 17:00:46,754] - [xdcrnewbasetests] [140700069775104] - INFO - cleanup [ip:172.23.108.236 port:8091 ssh_username:root, ip:172.23.122.201 port:8091 ssh_username:root] [2018-11-14 17:00:46,790] - [bucket_helper] [140700069775104] - INFO - deleting existing buckets [u'default'] on 172.23.108.236 [2018-11-14 17:00:46,791] - [bucket_helper] [140700069775104] - INFO - remove bucket default ... [2018-11-14 17:00:48,616] - [bucket_helper] [140700069775104] - INFO - deleted bucket : default from 172.23.108.236 [2018-11-14 17:00:48,617] - [bucket_helper] [140700069775104] - INFO - waiting for bucket deletion to complete.... [2018-11-14 17:00:48,623] - [rest_client] [140700069775104] - INFO - node 172.23.108.236 existing buckets : [] [2018-11-14 17:00:48,623] - [bucket_helper] [140700069775104] - INFO - sleep 2 seconds to make sure all buckets ([u'default']) were deleted completely. [2018-11-14 17:00:50,655] - [cluster_helper] [140700069775104] - INFO - rebalancing all nodes in order to remove nodes [2018-11-14 17:00:50,661] - [rest_client] [140700069775104] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.122.201', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.108.236,ns_1@172.23.122.201'} [2018-11-14 17:00:50,669] - [rest_client] [140700069775104] - INFO - rebalance operation started [2018-11-14 17:00:50,675] - [rest_client] [140700069775104] - INFO - rebalance percentage : 0.00 % [2018-11-14 17:01:10,709] - [rest_client] [140700069775104] - INFO - rebalance progress took 20.04 seconds [2018-11-14 17:01:10,709] - [rest_client] [140700069775104] - INFO - sleep for 10 seconds after rebalance... [2018-11-14 17:01:20,738] - [cluster_helper] [140700069775104] - INFO - removed all the nodes from cluster associated with ip:172.23.108.236 port:8091 ssh_username:root ? [(u'ns_1@172.23.122.201', 8091)] [2018-11-14 17:01:20,747] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.108.236:8091 [2018-11-14 17:01:20,754] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.108.236:8091 is running [2018-11-14 17:01:20,815] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.122.201:8091 [2018-11-14 17:01:20,824] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.122.201:8091 is running [2018-11-14 17:01:20,824] - [xdcrnewbasetests] [140700069775104] - INFO - removing xdcr/nodes settings [2018-11-14 17:01:20,844] - [rest_client] [140700069775104] - INFO - removing remote cluster name:remote_cluster_C3-C2 [2018-11-14 17:01:20,854] - [rest_client] [140700069775104] - INFO - removing remote cluster name:remote_cluster_C3-C1 [2018-11-14 17:01:20,894] - [xdcrnewbasetests] [140700069775104] - INFO - cleanup [ip:172.23.97.224 port:8091 ssh_username:root, ip:172.23.97.225 port:8091 ssh_username:root] [2018-11-14 17:01:20,923] - [bucket_helper] [140700069775104] - INFO - deleting existing buckets [u'default'] on 172.23.97.224 [2018-11-14 17:01:20,924] - [bucket_helper] [140700069775104] - INFO - remove bucket default ... [2018-11-14 17:01:21,862] - [bucket_helper] [140700069775104] - INFO - deleted bucket : default from 172.23.97.224 [2018-11-14 17:01:21,862] - [bucket_helper] [140700069775104] - INFO - waiting for bucket deletion to complete.... [2018-11-14 17:01:21,868] - [rest_client] [140700069775104] - INFO - node 172.23.97.224 existing buckets : [] [2018-11-14 17:01:21,868] - [bucket_helper] [140700069775104] - INFO - sleep 2 seconds to make sure all buckets ([u'default']) were deleted completely. [2018-11-14 17:01:23,900] - [cluster_helper] [140700069775104] - INFO - rebalancing all nodes in order to remove nodes [2018-11-14 17:01:23,905] - [rest_client] [140700069775104] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.97.225', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.97.225,ns_1@172.23.97.224'} [2018-11-14 17:01:23,911] - [rest_client] [140700069775104] - INFO - rebalance operation started [2018-11-14 17:01:23,921] - [rest_client] [140700069775104] - INFO - rebalance percentage : 0.00 % [2018-11-14 17:01:43,953] - [rest_client] [140700069775104] - INFO - rebalance progress took 20.04 seconds [2018-11-14 17:01:43,953] - [rest_client] [140700069775104] - INFO - sleep for 10 seconds after rebalance... [2018-11-14 17:01:53,976] - [cluster_helper] [140700069775104] - INFO - removed all the nodes from cluster associated with ip:172.23.97.224 port:8091 ssh_username:root ? [(u'ns_1@172.23.97.225', 8091)] [2018-11-14 17:01:53,983] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.97.224:8091 [2018-11-14 17:01:53,989] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.97.224:8091 is running [2018-11-14 17:01:54,030] - [cluster_helper] [140700069775104] - INFO - waiting for ns_server @ 172.23.97.225:8091 [2018-11-14 17:01:54,036] - [cluster_helper] [140700069775104] - INFO - ns_server @ 172.23.97.225:8091 is running [2018-11-14 17:01:54,036] - [xdcrnewbasetests] [140700069775104] - INFO - ==== XDCRNewbasetests cleanup is finished for test #34 load_with_ops ===