[2019-01-23 15:08:15,999] - [rest_client] [139642968725248] - ERROR - POST http://172.23.97.83:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password [2019-01-23 15:08:15,999] - [rest_client] [139642968725248] - INFO - /diag/eval status on 172.23.97.83:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-01-23 15:08:16,001] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.83 with username:root password:couchbase ssh_key: [2019-01-23 15:08:16,098] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.83 [2019-01-23 15:08:16,313] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:16,698] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: zgrep "panic:" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:16,796] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:16,796] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:16,912] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.83 with username:root password:couchbase ssh_key: [2019-01-23 15:08:16,992] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.83 [2019-01-23 15:08:17,207] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:17,600] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:17,698] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:17,698] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:17,814] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.84 with username:root password:couchbase ssh_key: [2019-01-23 15:08:17,894] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.84 [2019-01-23 15:08:18,166] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:18,637] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.84: zgrep "panic:" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:18,735] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:18,735] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:18,850] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.84 with username:root password:couchbase ssh_key: [2019-01-23 15:08:18,931] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.84 [2019-01-23 15:08:19,167] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:19,618] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.84: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:19,716] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:19,716] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:19,832] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.85 with username:root password:couchbase ssh_key: [2019-01-23 15:08:19,929] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.85 [2019-01-23 15:08:20,140] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:20,560] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: zgrep "panic:" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:20,658] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:20,658] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:20,773] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.85 with username:root password:couchbase ssh_key: [2019-01-23 15:08:20,854] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.85 [2019-01-23 15:08:21,122] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:21,525] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:21,622] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:21,622] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:21,738] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.86 with username:root password:couchbase ssh_key: [2019-01-23 15:08:21,835] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.86 [2019-01-23 15:08:22,073] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:22,466] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.86: zgrep "panic:" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:22,563] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:22,563] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:22,678] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.86 with username:root password:couchbase ssh_key: [2019-01-23 15:08:22,759] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.86 [2019-01-23 15:08:22,998] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:23,392] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.86: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:23,489] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:23,489] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:23,606] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.87 with username:root password:couchbase ssh_key: [2019-01-23 15:08:23,703] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.87 [2019-01-23 15:08:23,905] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:24,308] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: zgrep "panic:" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:24,406] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:24,406] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:24,522] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.87 with username:root password:couchbase ssh_key: [2019-01-23 15:08:24,602] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.87 [2019-01-23 15:08:24,844] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:25,243] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:25,341] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:25,341] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:25,457] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.88 with username:root password:couchbase ssh_key: [2019-01-23 15:08:25,555] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.88 [2019-01-23 15:08:25,826] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:26,220] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.88: zgrep "panic:" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:26,318] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:26,318] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:26,433] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.88 with username:root password:couchbase ssh_key: [2019-01-23 15:08:26,514] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.88 [2019-01-23 15:08:26,761] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:27,198] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.88: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:27,295] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:27,296] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:27,414] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key: [2019-01-23 15:08:27,512] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.11 [2019-01-23 15:08:27,749] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:28,163] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.11: zgrep "panic:" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:28,260] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:28,260] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:28,375] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key: [2019-01-23 15:08:28,456] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.11 [2019-01-23 15:08:28,696] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:29,082] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.11: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:29,179] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:29,179] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:29,294] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2019-01-23 15:08:29,391] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.164 [2019-01-23 15:08:29,662] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:30,053] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.164: zgrep "panic:" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:30,150] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:30,150] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:30,266] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2019-01-23 15:08:30,346] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.164 [2019-01-23 15:08:30,589] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:30,980] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.164: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:31,078] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:31,078] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:31,192] - [xdcrnewbasetests] [139642968725248] - INFO - {'172.23.98.11': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.97.88': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.98.164': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.97.85': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.97.84': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.97.87': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.97.86': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.97.83': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}} [2019-01-23 15:08:31,200] - [rest_client] [139642968725248] - ERROR - POST http://172.23.97.83:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password [2019-01-23 15:08:31,200] - [rest_client] [139642968725248] - INFO - /diag/eval status on 172.23.97.83:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-01-23 15:08:31,202] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.83 with username:root password:couchbase ssh_key: [2019-01-23 15:08:31,299] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.83 [2019-01-23 15:08:31,537] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:31,931] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:32,029] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:32,029] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:32,144] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.84 with username:root password:couchbase ssh_key: [2019-01-23 15:08:32,242] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.84 [2019-01-23 15:08:32,462] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:32,852] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.84: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:32,949] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:32,949] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:33,065] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.85 with username:root password:couchbase ssh_key: [2019-01-23 15:08:33,163] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.85 [2019-01-23 15:08:33,401] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:33,800] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:33,898] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:33,898] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:34,013] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.86 with username:root password:couchbase ssh_key: [2019-01-23 15:08:34,111] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.86 [2019-01-23 15:08:34,312] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:34,702] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.86: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:34,799] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:34,800] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:34,915] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.87 with username:root password:couchbase ssh_key: [2019-01-23 15:08:35,013] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.87 [2019-01-23 15:08:35,210] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:35,599] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:35,697] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:35,697] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:35,812] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.88 with username:root password:couchbase ssh_key: [2019-01-23 15:08:35,910] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.88 [2019-01-23 15:08:36,156] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:36,548] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.88: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:36,645] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:36,645] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:36,761] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key: [2019-01-23 15:08:36,858] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.11 [2019-01-23 15:08:37,079] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:37,462] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.11: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:37,560] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:37,560] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:37,675] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2019-01-23 15:08:37,773] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.164 [2019-01-23 15:08:38,044] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:38,435] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.164: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* | wc -l [2019-01-23 15:08:38,500] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:38,500] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:08:38,614] - [xdcrnewbasetests] [139642968725248] - INFO - {'172.23.98.11': 0, '172.23.97.88': 0, '172.23.98.164': 0, '172.23.97.85': 0, '172.23.97.84': 0, '172.23.97.87': 0, '172.23.97.86': 0, '172.23.97.83': 0} [2019-01-23 15:08:38,615] - [xdcrnewbasetests] [139642968725248] - INFO - ==== XDCRNewbasetests setup is started for test #34 load_with_ops ==== [2019-01-23 15:08:38,616] - [xdcrnewbasetests] [139642968725248] - INFO - removing xdcr/nodes settings [2019-01-23 15:08:38,643] - [xdcrnewbasetests] [139642968725248] - INFO - cleanup [ip:172.23.97.83 port:8091 ssh_username:root, ip:172.23.97.84 port:8091 ssh_username:root] [2019-01-23 15:08:38,667] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.83:8091 [2019-01-23 15:08:38,671] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.83:8091 is running [2019-01-23 15:08:38,697] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.84:8091 [2019-01-23 15:08:38,701] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.84:8091 is running [2019-01-23 15:08:38,701] - [xdcrnewbasetests] [139642968725248] - INFO - removing xdcr/nodes settings [2019-01-23 15:08:38,723] - [xdcrnewbasetests] [139642968725248] - INFO - cleanup [ip:172.23.97.85 port:8091 ssh_username:root, ip:172.23.97.86 port:8091 ssh_username:root] [2019-01-23 15:08:38,750] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.85:8091 [2019-01-23 15:08:38,753] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.85:8091 is running [2019-01-23 15:08:38,778] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.86:8091 [2019-01-23 15:08:38,782] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.86:8091 is running [2019-01-23 15:08:38,782] - [xdcrnewbasetests] [139642968725248] - INFO - removing xdcr/nodes settings [2019-01-23 15:08:38,804] - [xdcrnewbasetests] [139642968725248] - INFO - cleanup [ip:172.23.97.87 port:8091 ssh_username:root, ip:172.23.97.88 port:8091 ssh_username:root] [2019-01-23 15:08:38,827] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.87:8091 [2019-01-23 15:08:38,833] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.87:8091 is running [2019-01-23 15:08:38,859] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.88:8091 [2019-01-23 15:08:38,862] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.88:8091 is running [2019-01-23 15:08:38,862] - [xdcrnewbasetests] [139642968725248] - INFO - Initializing all clusters... [2019-01-23 15:08:38,868] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.83 with username:root password:couchbase ssh_key: [2019-01-23 15:08:38,965] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.83 [2019-01-23 15:08:39,204] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:39,606] - [rest_client] [139642968725248] - ERROR - GET http://172.23.97.83:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-01-23 15:08:39,606] - [remote_util] [139642968725248] - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2019-01-23 15:08:39,606] - [xdcrnewbasetests] [139642968725248] - INFO - Running in compatibility mode, not enabled diag/eval for non-local hosts [2019-01-23 15:08:39,610] - [rest_client] [139642968725248] - ERROR - POST http://172.23.97.83:8091/diag/eval/ body: cluster_compat_mode:get_compat_version(). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password [2019-01-23 15:08:39,610] - [rest_client] [139642968725248] - INFO - /diag/eval status on 172.23.97.83:8091: False content: API is accessible from localhost only command: cluster_compat_mode:get_compat_version(). [2019-01-23 15:08:39,623] - [task] [139634797430528] - INFO - server: ip:172.23.97.83 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.83', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.97.83', 'uptime': u'12740', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.97.83:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.1-2037-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3064913920, 'memoryTotal': 3961200640, 'memoryQuota': 1600, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-01-23 15:08:39,623] - [rest_client] [139634797430528] - INFO - pools/default params : memoryQuota=2014 [2019-01-23 15:08:39,627] - [rest_client] [139634797430528] - INFO - settings/web params on 172.23.97.83:8091:username=Administrator&password=password&port=8091 [2019-01-23 15:08:39,764] - [remote_util] [139634797430528] - INFO - connecting to 172.23.97.83 with username:root password:couchbase ssh_key: [2019-01-23 15:08:39,861] - [remote_util] [139634797430528] - INFO - Connected to 172.23.97.83 [2019-01-23 15:08:40,074] - [remote_util] [139634797430528] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:40,483] - [remote_util] [139634797430528] - INFO - running command.raw on 172.23.97.83: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-01-23 15:08:40,534] - [remote_util] [139634797430528] - INFO - command executed successfully [2019-01-23 15:08:40,540] - [rest_client] [139634797430528] - INFO - /diag/eval status on 172.23.97.83:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2019-01-23 15:08:40,540] - [rest_client] [139634797430528] - INFO - settings/indexes params : storageMode=forestdb [2019-01-23 15:08:40,544] - [rest_client] [139634797430528] - ERROR - POST http://172.23.97.83:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2019-01-23 15:08:40,560] - [task] [139634797430528] - INFO - server: ip:172.23.97.84 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.84', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.97.84', 'uptime': u'4869', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.97.84:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.1-2037-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3060981760, 'memoryTotal': 3961200640, 'memoryQuota': 2014, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-01-23 15:08:40,560] - [rest_client] [139634797430528] - INFO - pools/default params : memoryQuota=2014 [2019-01-23 15:08:40,565] - [rest_client] [139634797430528] - INFO - settings/web params on 172.23.97.84:8091:username=Administrator&password=password&port=8091 [2019-01-23 15:08:40,586] - [remote_util] [139634797430528] - INFO - connecting to 172.23.97.84 with username:root password:couchbase ssh_key: [2019-01-23 15:08:40,683] - [remote_util] [139634797430528] - INFO - Connected to 172.23.97.84 [2019-01-23 15:08:40,921] - [remote_util] [139634797430528] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:41,331] - [remote_util] [139634797430528] - INFO - running command.raw on 172.23.97.84: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-01-23 15:08:41,381] - [remote_util] [139634797430528] - INFO - command executed successfully [2019-01-23 15:08:41,384] - [rest_client] [139634797430528] - INFO - /diag/eval status on 172.23.97.84:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2019-01-23 15:08:41,385] - [rest_client] [139634797430528] - INFO - settings/indexes params : storageMode=forestdb [2019-01-23 15:08:41,388] - [rest_client] [139634797430528] - ERROR - POST http://172.23.97.84:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2019-01-23 15:08:42,402] - [task] [139634797430528] - INFO - adding node 172.23.97.84:8091 to cluster [2019-01-23 15:08:42,402] - [rest_client] [139634797430528] - INFO - adding remote node @172.23.97.84:8091 to this cluster @172.23.97.83:8091 [2019-01-23 15:08:44,250] - [rest_client] [139634797430528] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.97.84,ns_1@172.23.97.83'} [2019-01-23 15:08:44,255] - [rest_client] [139634797430528] - INFO - rebalance operation started [2019-01-23 15:08:44,258] - [rest_client] [139634797430528] - INFO - rebalance percentage : 0.00 % [2019-01-23 15:08:44,258] - [task] [139634797430528] - INFO - Rebalance - status: running, progress: 0.00% [2019-01-23 15:08:54,278] - [task] [139634797430528] - INFO - Rebalance - status: none, progress: 100.00% [2019-01-23 15:08:54,282] - [task] [139634797430528] - INFO - rebalancing was completed with progress: 100% in 10.0274820328 sec [2019-01-23 15:08:54,289] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.85 with username:root password:couchbase ssh_key: [2019-01-23 15:08:54,386] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.85 [2019-01-23 15:08:54,623] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:54,989] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-01-23 15:08:55,038] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:08:55,038] - [xdcrnewbasetests] [139642968725248] - INFO - Enabled diag/eval for non-local hosts from 172.23.97.85 [2019-01-23 15:08:55,042] - [rest_client] [139642968725248] - INFO - /diag/eval status on 172.23.97.85:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2019-01-23 15:08:55,641] - [task] [139634881074944] - INFO - server: ip:172.23.97.85 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.85', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.97.85', 'uptime': u'12803', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.97.85:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.1-2037-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3031547904, 'memoryTotal': 3961200640, 'memoryQuota': 2014, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-01-23 15:08:55,642] - [rest_client] [139634881074944] - INFO - pools/default params : memoryQuota=2014 [2019-01-23 15:08:55,645] - [rest_client] [139634881074944] - INFO - settings/web params on 172.23.97.85:8091:username=Administrator&password=password&port=8091 [2019-01-23 15:08:55,670] - [remote_util] [139634881074944] - INFO - connecting to 172.23.97.85 with username:root password:couchbase ssh_key: [2019-01-23 15:08:55,767] - [remote_util] [139634881074944] - INFO - Connected to 172.23.97.85 [2019-01-23 15:08:56,006] - [remote_util] [139634881074944] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:56,395] - [remote_util] [139634881074944] - INFO - running command.raw on 172.23.97.85: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-01-23 15:08:56,445] - [remote_util] [139634881074944] - INFO - command executed successfully [2019-01-23 15:08:56,451] - [rest_client] [139634881074944] - INFO - /diag/eval status on 172.23.97.85:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2019-01-23 15:08:56,451] - [rest_client] [139634881074944] - INFO - settings/indexes params : storageMode=forestdb [2019-01-23 15:08:56,455] - [rest_client] [139634881074944] - ERROR - POST http://172.23.97.85:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2019-01-23 15:08:56,478] - [task] [139634881074944] - INFO - server: ip:172.23.97.86 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.86', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.97.86', 'uptime': u'4877', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.97.86:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.1-2037-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3199774720, 'memoryTotal': 3961200640, 'memoryQuota': 1600, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-01-23 15:08:56,478] - [rest_client] [139634881074944] - INFO - pools/default params : memoryQuota=2014 [2019-01-23 15:08:56,482] - [rest_client] [139634881074944] - INFO - settings/web params on 172.23.97.86:8091:username=Administrator&password=password&port=8091 [2019-01-23 15:08:56,504] - [remote_util] [139634881074944] - INFO - connecting to 172.23.97.86 with username:root password:couchbase ssh_key: [2019-01-23 15:08:56,621] - [remote_util] [139634881074944] - INFO - Connected to 172.23.97.86 [2019-01-23 15:08:56,829] - [remote_util] [139634881074944] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:08:57,254] - [remote_util] [139634881074944] - INFO - running command.raw on 172.23.97.86: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-01-23 15:08:57,319] - [remote_util] [139634881074944] - INFO - command executed successfully [2019-01-23 15:08:57,322] - [rest_client] [139634881074944] - INFO - /diag/eval status on 172.23.97.86:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2019-01-23 15:08:57,322] - [rest_client] [139634881074944] - INFO - settings/indexes params : storageMode=forestdb [2019-01-23 15:08:57,325] - [rest_client] [139634881074944] - ERROR - POST http://172.23.97.86:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2019-01-23 15:08:58,333] - [task] [139634881074944] - INFO - adding node 172.23.97.86:8091 to cluster [2019-01-23 15:08:58,333] - [rest_client] [139634881074944] - INFO - adding remote node @172.23.97.86:8091 to this cluster @172.23.97.85:8091 [2019-01-23 15:08:59,920] - [rest_client] [139634881074944] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.97.85,ns_1@172.23.97.86'} [2019-01-23 15:08:59,925] - [rest_client] [139634881074944] - INFO - rebalance operation started [2019-01-23 15:08:59,929] - [rest_client] [139634881074944] - INFO - rebalance percentage : 0.00 % [2019-01-23 15:08:59,929] - [task] [139634881074944] - INFO - Rebalance - status: running, progress: 0.00% [2019-01-23 15:09:09,949] - [task] [139634881074944] - INFO - Rebalance - status: none, progress: 100.00% [2019-01-23 15:09:09,953] - [task] [139634881074944] - INFO - rebalancing was completed with progress: 100% in 10.0286560059 sec [2019-01-23 15:09:09,960] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.87 with username:root password:couchbase ssh_key: [2019-01-23 15:09:10,058] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.87 [2019-01-23 15:09:10,258] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:09:10,637] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-01-23 15:09:10,702] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:09:10,702] - [xdcrnewbasetests] [139642968725248] - INFO - Enabled diag/eval for non-local hosts from 172.23.97.87 [2019-01-23 15:09:10,706] - [rest_client] [139642968725248] - INFO - /diag/eval status on 172.23.97.87:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2019-01-23 15:09:11,658] - [task] [139635082422016] - INFO - server: ip:172.23.97.87 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.87', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.97.87', 'uptime': u'12824', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.97.87:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.1-2037-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3153096704, 'memoryTotal': 3961200640, 'memoryQuota': 2014, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-01-23 15:09:11,658] - [rest_client] [139635082422016] - INFO - pools/default params : memoryQuota=2014 [2019-01-23 15:09:11,662] - [rest_client] [139635082422016] - INFO - settings/web params on 172.23.97.87:8091:username=Administrator&password=password&port=8091 [2019-01-23 15:09:11,683] - [remote_util] [139635082422016] - INFO - connecting to 172.23.97.87 with username:root password:couchbase ssh_key: [2019-01-23 15:09:11,779] - [remote_util] [139635082422016] - INFO - Connected to 172.23.97.87 [2019-01-23 15:09:11,975] - [remote_util] [139635082422016] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:09:12,446] - [remote_util] [139635082422016] - INFO - running command.raw on 172.23.97.87: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-01-23 15:09:12,511] - [remote_util] [139635082422016] - INFO - command executed successfully [2019-01-23 15:09:12,514] - [rest_client] [139635082422016] - INFO - /diag/eval status on 172.23.97.87:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2019-01-23 15:09:12,514] - [rest_client] [139635082422016] - INFO - settings/indexes params : storageMode=forestdb [2019-01-23 15:09:12,518] - [rest_client] [139635082422016] - ERROR - POST http://172.23.97.87:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2019-01-23 15:09:12,534] - [task] [139635082422016] - INFO - server: ip:172.23.97.88 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.88', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.97.88', 'uptime': u'12780', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.97.88:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.1-2037-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3267211264, 'memoryTotal': 3961200640, 'memoryQuota': 1600, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-01-23 15:09:12,534] - [rest_client] [139635082422016] - INFO - pools/default params : memoryQuota=2014 [2019-01-23 15:09:12,538] - [rest_client] [139635082422016] - INFO - settings/web params on 172.23.97.88:8091:username=Administrator&password=password&port=8091 [2019-01-23 15:09:12,561] - [remote_util] [139635082422016] - INFO - connecting to 172.23.97.88 with username:root password:couchbase ssh_key: [2019-01-23 15:09:12,657] - [remote_util] [139635082422016] - INFO - Connected to 172.23.97.88 [2019-01-23 15:09:12,926] - [remote_util] [139635082422016] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:09:13,363] - [remote_util] [139635082422016] - INFO - running command.raw on 172.23.97.88: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-01-23 15:09:13,428] - [remote_util] [139635082422016] - INFO - command executed successfully [2019-01-23 15:09:13,431] - [rest_client] [139635082422016] - INFO - /diag/eval status on 172.23.97.88:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2019-01-23 15:09:13,431] - [rest_client] [139635082422016] - INFO - settings/indexes params : storageMode=forestdb [2019-01-23 15:09:13,434] - [rest_client] [139635082422016] - ERROR - POST http://172.23.97.88:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2019-01-23 15:09:14,442] - [task] [139635082422016] - INFO - adding node 172.23.97.88:8091 to cluster [2019-01-23 15:09:14,442] - [rest_client] [139635082422016] - INFO - adding remote node @172.23.97.88:8091 to this cluster @172.23.97.87:8091 [2019-01-23 15:09:16,287] - [rest_client] [139635082422016] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.97.87,ns_1@172.23.97.88'} [2019-01-23 15:09:16,292] - [rest_client] [139635082422016] - INFO - rebalance operation started [2019-01-23 15:09:16,297] - [rest_client] [139635082422016] - INFO - rebalance percentage : 0.00 % [2019-01-23 15:09:16,297] - [task] [139635082422016] - INFO - Rebalance - status: running, progress: 0.00% [2019-01-23 15:09:26,317] - [task] [139635082422016] - INFO - Rebalance - status: none, progress: 100.00% [2019-01-23 15:09:26,323] - [task] [139635082422016] - INFO - rebalancing was completed with progress: 100% in 10.0308899879 sec [2019-01-23 15:09:26,343] - [rest_client] [139642968725248] - ERROR - DELETE http://172.23.97.83:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2019-01-23 15:09:26,343] - [internal_user] [139642968725248] - INFO - Exception while deleting user. Exception is -"User was not found." [2019-01-23 15:09:26,394] - [rest_client] [139642968725248] - ERROR - DELETE http://172.23.97.85:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2019-01-23 15:09:26,394] - [internal_user] [139642968725248] - INFO - Exception while deleting user. Exception is -"User was not found." [2019-01-23 15:09:26,450] - [rest_client] [139642968725248] - ERROR - DELETE http://172.23.97.87:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2019-01-23 15:09:26,450] - [internal_user] [139642968725248] - INFO - Exception while deleting user. Exception is -"User was not found." [2019-01-23 15:09:27,340] - [rest_client] [139634797430528] - INFO - http://172.23.97.83:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1514&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2019-01-23 15:09:27,346] - [rest_client] [139634797430528] - INFO - 0.01 seconds to create bucket default [2019-01-23 15:09:27,346] - [bucket_helper] [139634797430528] - INFO - waiting for memcached bucket : default in 172.23.97.83 to accept set ops [2019-01-23 15:09:28,462] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:09:28,556] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:09:28,635] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:09:28,687] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:09:29,551] - [task] [139634797430528] - INFO - bucket 'default' was created with per node RAM quota: 1514 [2019-01-23 15:09:29,996] - [rest_client] [139634881074944] - INFO - http://172.23.97.85:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1514&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2019-01-23 15:09:30,011] - [rest_client] [139634881074944] - INFO - 0.01 seconds to create bucket default [2019-01-23 15:09:30,011] - [bucket_helper] [139634881074944] - INFO - waiting for memcached bucket : default in 172.23.97.85 to accept set ops [2019-01-23 15:09:31,113] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:09:31,170] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:09:31,279] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:09:31,331] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:09:31,803] - [task] [139634881074944] - INFO - bucket 'default' was created with per node RAM quota: 1514 [2019-01-23 15:09:32,354] - [rest_client] [139635082422016] - INFO - http://172.23.97.87:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1514&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2019-01-23 15:09:32,359] - [rest_client] [139635082422016] - INFO - 0.01 seconds to create bucket default [2019-01-23 15:09:32,360] - [bucket_helper] [139635082422016] - INFO - waiting for memcached bucket : default in 172.23.97.87 to accept set ops [2019-01-23 15:09:33,993] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:09:34,046] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:09:34,118] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:09:34,191] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:09:34,509] - [task] [139635082422016] - INFO - bucket 'default' was created with per node RAM quota: 1514 [2019-01-23 15:09:34,514] - [xdcrnewbasetests] [139642968725248] - INFO - ==== XDCRNewbasetests setup is finished for test #34 load_with_ops ==== [2019-01-23 15:09:34,519] - [rest_client] [139642968725248] - INFO - adding remote cluster hostname:172.23.97.85:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.97.83:8091 [2019-01-23 15:09:34,540] - [rest_client] [139642968725248] - INFO - adding remote cluster hostname:172.23.97.83:8091 with username:password Administrator:password name:remote_cluster_C2-C1 to source node: 172.23.97.85:8091 [2019-01-23 15:09:34,562] - [rest_client] [139642968725248] - INFO - adding remote cluster hostname:172.23.97.87:8091 with username:password Administrator:password name:remote_cluster_C2-C3 to source node: 172.23.97.85:8091 [2019-01-23 15:09:34,583] - [rest_client] [139642968725248] - INFO - adding remote cluster hostname:172.23.97.85:8091 with username:password Administrator:password name:remote_cluster_C3-C2 to source node: 172.23.97.87:8091 [2019-01-23 15:09:34,608] - [rest_client] [139642968725248] - INFO - adding remote cluster hostname:172.23.97.83:8091 with username:password Administrator:password name:remote_cluster_C3-C1 to source node: 172.23.97.87:8091 [2019-01-23 15:09:34,624] - [rest_client] [139642968725248] - INFO - adding remote cluster hostname:172.23.97.87:8091 with username:password Administrator:password name:remote_cluster_C1-C3 to source node: 172.23.97.83:8091 [2019-01-23 15:09:34,644] - [rest_client] [139642968725248] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {'filterExpression': 'C1-key-1'} [2019-01-23 15:09:34,683] - [rest_client] [139642968725248] - INFO - Replication created with id: 5e01cdb750b65f73af652fdbc2fb9264/default/default [2019-01-23 15:09:44,690] - [rest_client] [139642968725248] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C3 with settings {'filterExpression': 'C1-key-1'} [2019-01-23 15:09:44,726] - [rest_client] [139642968725248] - INFO - Replication created with id: f7f021251634dfe770fccec884116b2b/default/default [2019-01-23 15:09:54,741] - [rest_client] [139642968725248] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C2-C1 with settings {} [2019-01-23 15:09:54,780] - [rest_client] [139642968725248] - INFO - Replication created with id: b5eda3772e0990c9e0b5dc05a6ce75c7/default/default [2019-01-23 15:10:04,795] - [rest_client] [139642968725248] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C2-C3 with settings {} [2019-01-23 15:10:04,827] - [rest_client] [139642968725248] - INFO - Replication created with id: f7f021251634dfe770fccec884116b2b/default/default [2019-01-23 15:10:14,844] - [rest_client] [139642968725248] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C3-C2 with settings {} [2019-01-23 15:10:14,883] - [rest_client] [139642968725248] - INFO - Replication created with id: 5e01cdb750b65f73af652fdbc2fb9264/default/default [2019-01-23 15:10:24,898] - [rest_client] [139642968725248] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C3-C1 with settings {} [2019-01-23 15:10:24,946] - [rest_client] [139642968725248] - INFO - Replication created with id: b5eda3772e0990c9e0b5dc05a6ce75c7/default/default [2019-01-23 15:10:34,973] - [rest_client] [139642968725248] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.97.83 [2019-01-23 15:10:34,989] - [rest_client] [139642968725248] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.97.83 [2019-01-23 15:10:35,007] - [rest_client] [139642968725248] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.97.85 [2019-01-23 15:10:35,020] - [rest_client] [139642968725248] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.97.85 [2019-01-23 15:10:35,041] - [rest_client] [139642968725248] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.97.87 [2019-01-23 15:10:35,060] - [rest_client] [139642968725248] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.97.87 [2019-01-23 15:10:35,101] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:10:35,151] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:10:35,693] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:10:35,709] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:10:35,731] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:10:35,758] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:10:35,760] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:10:35,768] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:10:35,799] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:10:35,813] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:10:36,488] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:10:36,540] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:10:36,985] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:10:37,001] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:10:37,033] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:10:37,038] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:10:37,040] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:10:37,055] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:10:37,088] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:10:37,088] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:10:37,876] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:10:37,947] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:10:38,654] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:10:38,669] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:10:38,718] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:10:38,718] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:10:38,738] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:10:38,740] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:10:38,871] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:10:38,887] - [data_helper] [139634402973440] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:10:39,557] - [xdcrnewbasetests] [139642968725248] - INFO - sleep for 10 secs. ... [2019-01-23 15:10:49,568] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C1:default -> C2:default [2019-01-23 15:10:49,592] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 10000 valid and 0 deleted keys [2019-01-23 15:10:49,593] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 10000 valid and 0 deleted keys [2019-01-23 15:10:49,603] - [xdcrnewbasetests] [139642968725248] - INFO - 1111 keys matched the filter expression C1-key-1 [2019-01-23 15:10:49,622] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 11111 valid keys and 0 deleted keys [2019-01-23 15:10:49,622] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C1:default -> C3:default [2019-01-23 15:10:49,640] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 10000 valid and 0 deleted keys [2019-01-23 15:10:49,640] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 10000 valid and 0 deleted keys [2019-01-23 15:10:49,654] - [xdcrnewbasetests] [139642968725248] - INFO - 1111 keys matched the filter expression C1-key-1 [2019-01-23 15:10:49,673] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 11111 valid keys and 0 deleted keys [2019-01-23 15:10:49,673] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C2:default -> C1:default [2019-01-23 15:10:49,690] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 11111 valid and 0 deleted keys [2019-01-23 15:10:49,690] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 10000 valid and 0 deleted keys [2019-01-23 15:10:49,776] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 20000 valid keys and 0 deleted keys [2019-01-23 15:10:49,776] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C2:default -> C3:default [2019-01-23 15:10:49,793] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 11111 valid and 0 deleted keys [2019-01-23 15:10:49,793] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 11111 valid and 0 deleted keys [2019-01-23 15:10:49,883] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2019-01-23 15:10:49,883] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C3:default -> C2:default [2019-01-23 15:10:49,906] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2019-01-23 15:10:49,906] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 11111 valid and 0 deleted keys [2019-01-23 15:10:50,050] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2019-01-23 15:10:50,051] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C3:default -> C1:default [2019-01-23 15:10:50,082] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2019-01-23 15:10:50,082] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 20000 valid and 0 deleted keys [2019-01-23 15:10:50,230] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 30000 valid keys and 0 deleted keys [2019-01-23 15:10:50,230] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C1:default -> C2:default [2019-01-23 15:10:50,278] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 30000 valid and 0 deleted keys [2019-01-23 15:10:50,278] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 21111 valid and 0 deleted keys [2019-01-23 15:10:50,310] - [xdcrnewbasetests] [139642968725248] - INFO - 1111 keys matched the filter expression C1-key-1 [2019-01-23 15:10:50,334] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2019-01-23 15:10:50,334] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C1:default -> C3:default [2019-01-23 15:10:50,383] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 30000 valid and 0 deleted keys [2019-01-23 15:10:50,383] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 21111 valid and 0 deleted keys [2019-01-23 15:10:50,419] - [xdcrnewbasetests] [139642968725248] - INFO - 1111 keys matched the filter expression C1-key-1 [2019-01-23 15:10:50,442] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2019-01-23 15:10:50,442] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C2:default -> C1:default [2019-01-23 15:10:50,486] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2019-01-23 15:10:50,486] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 30000 valid and 0 deleted keys [2019-01-23 15:10:50,633] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 30000 valid keys and 0 deleted keys [2019-01-23 15:10:50,633] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C2:default -> C3:default [2019-01-23 15:10:50,673] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2019-01-23 15:10:50,673] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 21111 valid and 0 deleted keys [2019-01-23 15:10:50,813] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2019-01-23 15:10:50,814] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C3:default -> C2:default [2019-01-23 15:10:50,845] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2019-01-23 15:10:50,845] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 21111 valid and 0 deleted keys [2019-01-23 15:10:50,992] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 21111 valid keys and 0 deleted keys [2019-01-23 15:10:50,992] - [xdcrnewbasetests] [139642968725248] - INFO - Merging keys for replication Replication C3:default -> C1:default [2019-01-23 15:10:51,033] - [xdcrnewbasetests] [139642968725248] - INFO - src_kvstore has 21111 valid and 0 deleted keys [2019-01-23 15:10:51,033] - [xdcrnewbasetests] [139642968725248] - INFO - dest kvstore has 30000 valid and 0 deleted keys [2019-01-23 15:10:51,191] - [xdcrnewbasetests] [139642968725248] - INFO - After merging: destination bucket's kv_store now has 30000 valid keys and 0 deleted keys [2019-01-23 15:10:51,218] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'} [2019-01-23 15:10:51,218] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:10:51,315] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:10:51,316] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2010243047, 0, '') [2019-01-23 15:10:51,333] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'} [2019-01-23 15:10:51,333] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:10:51,372] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:10:51,372] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2538315210, 0, '') [2019-01-23 15:10:51,372] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:11:01,409] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'} [2019-01-23 15:11:01,409] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:11:01,448] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:11:01,449] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (836961323, 0, '') [2019-01-23 15:11:01,466] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'} [2019-01-23 15:11:01,466] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:11:01,503] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:11:01,504] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3313790014, 0, '') [2019-01-23 15:11:01,504] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:11:11,690] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:11:11,741] - [task] [139634797430528] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.83:8091',default bucket [2019-01-23 15:11:11,758] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:11:11,804] - [task] [139634797430528] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.84:8091',default bucket [2019-01-23 15:11:11,939] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:11:11,987] - [task] [139634881074944] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.85:8091',default bucket [2019-01-23 15:11:11,996] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:11:12,042] - [task] [139634881074944] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.86:8091',default bucket [2019-01-23 15:11:12,043] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.83 [2019-01-23 15:11:12,048] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:11:12,076] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C1 for default is 0 [2019-01-23 15:11:12,077] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.85 [2019-01-23 15:11:12,082] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:11:12,109] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C2 for default is 0 [2019-01-23 15:11:12,109] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.83 [2019-01-23 15:11:12,140] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0 [2019-01-23 15:11:12,140] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.85 [2019-01-23 15:11:12,172] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0 [2019-01-23 15:11:12,197] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:11:12,221] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 30000 == 30000 expected on C1, default bucket [2019-01-23 15:11:12,232] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:11:12,257] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 30000 == 30000 expected on C1, default bucket [2019-01-23 15:11:12,272] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:11:12,296] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C2, default bucket [2019-01-23 15:11:12,303] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:11:12,325] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C2, default bucket [2019-01-23 15:11:12,364] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:11:12,458] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:11:12,588] - [task] [139642968725248] - INFO - 30000 items will be verified on default bucket [2019-01-23 15:11:12,806] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:11:15,167] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:11:17,486] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:11:19,948] - [task] [139634402973440] - INFO - 30000 items were verified [2019-01-23 15:11:19,948] - [task] [139634402973440] - INFO - 30000 items were verified in 7.36010408401 sec.the average number of ops - 4076.02878578 per second [2019-01-23 15:11:19,989] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:11:20,039] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:11:20,145] - [task] [139642968725248] - INFO - 21111 items will be verified on default bucket [2019-01-23 15:11:21,051] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:11:23,520] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:11:25,963] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:11:26,255] - [task] [139634402973440] - INFO - 21111 items were verified in 6.10954189301 sec.the average number of ops - 3455.41334444 per second [2019-01-23 15:11:26,255] - [xdcrnewbasetests] [139642968725248] - INFO - Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.97.83 -> Couchbase Cluster: C2, Master Ip: 172.23.97.85, bucket default [2019-01-23 15:11:26,297] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:11:26,348] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:11:26,480] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:11:26,533] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:11:26,659] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:11:26,711] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:11:26,800] - [task] [139642968725248] - INFO - RevID verification: in progress for default ... [2019-01-23 15:12:08,306] - [task] [139634402973440] - INFO - RevId Verification : 30000 existing items have been verified [2019-01-23 15:12:08,306] - [task] [139634402973440] - INFO - RevId Verification : 0 deleted items have been verified [2019-01-23 15:12:08,306] - [task] [139634402973440] - INFO - RevId Verification : 8889 keys were apparently filtered out and not found in target bucket [2019-01-23 15:12:08,312] - [rest_client] [139634402973440] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:12:08,394] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'} [2019-01-23 15:12:08,394] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:12:08,434] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:12:08,434] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (203943586, 0, '') [2019-01-23 15:12:08,453] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'} [2019-01-23 15:12:08,453] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:12:08,496] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:12:08,496] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3235324569, 0, '') [2019-01-23 15:12:08,497] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:12:18,534] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'} [2019-01-23 15:12:18,534] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:12:18,577] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:12:18,578] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2156199912, 0, '') [2019-01-23 15:12:18,595] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'} [2019-01-23 15:12:18,595] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:12:18,633] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:12:18,634] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3390333570, 0, '') [2019-01-23 15:12:18,635] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:12:28,936] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:12:28,986] - [task] [139634797430528] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.83:8091',default bucket [2019-01-23 15:12:28,996] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:12:29,077] - [task] [139634797430528] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.84:8091',default bucket [2019-01-23 15:12:29,729] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:12:29,781] - [task] [139635082422016] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.87:8091',default bucket [2019-01-23 15:12:29,791] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:12:29,836] - [task] [139635082422016] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.88:8091',default bucket [2019-01-23 15:12:29,845] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.83 [2019-01-23 15:12:29,850] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:12:29,879] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C1 for default is 0 [2019-01-23 15:12:29,880] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.87 [2019-01-23 15:12:29,885] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:12:29,916] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C3 for default is 0 [2019-01-23 15:12:29,916] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.83 [2019-01-23 15:12:29,945] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0 [2019-01-23 15:12:29,946] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.87 [2019-01-23 15:12:29,982] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C3 for bucket default is 0 [2019-01-23 15:12:30,005] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:12:30,031] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 30000 == 30000 expected on C1, default bucket [2019-01-23 15:12:30,047] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:12:30,071] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 30000 == 30000 expected on C1, default bucket [2019-01-23 15:12:30,094] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:12:30,120] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C3, default bucket [2019-01-23 15:12:30,128] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:12:30,156] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C3, default bucket [2019-01-23 15:12:30,195] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:12:30,246] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:12:30,373] - [task] [139642968725248] - INFO - 30000 items will be verified on default bucket [2019-01-23 15:12:31,081] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:12:33,525] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:12:36,039] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:12:38,429] - [task] [139634402973440] - INFO - 30000 items were verified [2019-01-23 15:12:38,429] - [task] [139634402973440] - INFO - 30000 items were verified in 8.05475902557 sec.the average number of ops - 3724.50576768 per second [2019-01-23 15:12:38,471] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:12:38,527] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:12:38,632] - [task] [139642968725248] - INFO - 21111 items will be verified on default bucket [2019-01-23 15:12:38,848] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:12:41,405] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:12:43,893] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:12:44,166] - [task] [139634402973440] - INFO - 21111 items were verified in 5.5331120491 sec.the average number of ops - 3815.39223052 per second [2019-01-23 15:12:44,166] - [xdcrnewbasetests] [139642968725248] - INFO - Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.97.83 -> Couchbase Cluster: C3, Master Ip: 172.23.97.87, bucket default [2019-01-23 15:12:44,214] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:12:44,332] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:12:44,460] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:12:44,517] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:12:44,590] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:12:44,646] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:12:44,754] - [task] [139642968725248] - INFO - RevID verification: in progress for default ... [2019-01-23 15:13:26,695] - [task] [139634402973440] - INFO - RevId Verification : 30000 existing items have been verified [2019-01-23 15:13:26,695] - [task] [139634402973440] - INFO - RevId Verification : 0 deleted items have been verified [2019-01-23 15:13:26,695] - [task] [139634402973440] - INFO - RevId Verification : 8889 keys were apparently filtered out and not found in target bucket [2019-01-23 15:13:26,701] - [rest_client] [139634402973440] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:13:26,806] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'} [2019-01-23 15:13:26,806] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:13:26,847] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:13:26,848] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (886257487, 0, '') [2019-01-23 15:13:26,866] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'} [2019-01-23 15:13:26,866] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:13:26,954] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:13:26,955] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3680830329, 0, '') [2019-01-23 15:13:26,955] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:13:36,991] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'} [2019-01-23 15:13:36,991] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:13:37,028] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:13:37,029] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2553004709, 0, '') [2019-01-23 15:13:37,046] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'} [2019-01-23 15:13:37,047] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:13:37,088] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:13:37,089] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2171863575, 0, '') [2019-01-23 15:13:37,089] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:13:47,247] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:13:47,299] - [task] [139634881074944] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.85:8091',default bucket [2019-01-23 15:13:47,307] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:13:47,350] - [task] [139634881074944] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.86:8091',default bucket [2019-01-23 15:13:48,208] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:13:48,253] - [task] [139634797430528] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.83:8091',default bucket [2019-01-23 15:13:48,262] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:13:48,310] - [task] [139634797430528] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.84:8091',default bucket [2019-01-23 15:13:48,326] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.85 [2019-01-23 15:13:48,330] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:13:48,356] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C2 for default is 0 [2019-01-23 15:13:48,356] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.83 [2019-01-23 15:13:48,361] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:13:48,388] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C1 for default is 0 [2019-01-23 15:13:48,388] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.85 [2019-01-23 15:13:48,417] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0 [2019-01-23 15:13:48,417] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.83 [2019-01-23 15:13:48,452] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0 [2019-01-23 15:13:48,471] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:13:48,497] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C2, default bucket [2019-01-23 15:13:48,505] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:13:48,532] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C2, default bucket [2019-01-23 15:13:48,555] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:13:48,578] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 30000 == 30000 expected on C1, default bucket [2019-01-23 15:13:48,598] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:13:48,621] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 30000 == 30000 expected on C1, default bucket [2019-01-23 15:13:48,657] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:13:48,706] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:13:48,862] - [task] [139642968725248] - INFO - 21111 items will be verified on default bucket [2019-01-23 15:13:49,353] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:13:51,783] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:13:54,236] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:13:54,516] - [task] [139634402973440] - INFO - 21111 items were verified in 5.65347504616 sec.the average number of ops - 3734.16284081 per second [2019-01-23 15:13:54,558] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:13:54,611] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:13:54,747] - [task] [139642968725248] - INFO - 30000 items will be verified on default bucket [2019-01-23 15:13:55,319] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:13:57,801] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:14:00,249] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:14:02,700] - [task] [139634402973440] - INFO - 30000 items were verified [2019-01-23 15:14:02,701] - [task] [139634402973440] - INFO - 30000 items were verified in 7.95337486267 sec.the average number of ops - 3771.98309164 per second [2019-01-23 15:14:02,701] - [xdcrnewbasetests] [139642968725248] - INFO - Verifying RevIds for Couchbase Cluster: C2, Master Ip: 172.23.97.85 -> Couchbase Cluster: C1, Master Ip: 172.23.97.83, bucket default [2019-01-23 15:14:02,746] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:14:02,794] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:14:02,925] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:14:02,975] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:14:03,104] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:14:03,159] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:14:03,262] - [task] [139642968725248] - INFO - RevID verification: in progress for default ... [2019-01-23 15:14:28,444] - [task] [139634402973440] - INFO - RevId Verification : 21111 existing items have been verified [2019-01-23 15:14:28,445] - [task] [139634402973440] - INFO - RevId Verification : 0 deleted items have been verified [2019-01-23 15:14:28,445] - [task] [139634402973440] - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket [2019-01-23 15:14:28,449] - [rest_client] [139634402973440] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:14:28,518] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'} [2019-01-23 15:14:28,518] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:14:28,556] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:14:28,557] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2628004648, 0, '') [2019-01-23 15:14:28,576] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'} [2019-01-23 15:14:28,576] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:14:28,616] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:14:28,616] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1443181950, 0, '') [2019-01-23 15:14:28,616] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:14:38,655] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'} [2019-01-23 15:14:38,655] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:14:38,696] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:14:38,697] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1593584680, 0, '') [2019-01-23 15:14:38,714] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'} [2019-01-23 15:14:38,714] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:14:38,751] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:14:38,752] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (366651055, 0, '') [2019-01-23 15:14:38,752] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:14:49,462] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:14:49,510] - [task] [139634881074944] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.85:8091',default bucket [2019-01-23 15:14:49,519] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:14:49,562] - [task] [139634881074944] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.86:8091',default bucket [2019-01-23 15:14:50,079] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:14:50,168] - [task] [139635082422016] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.87:8091',default bucket [2019-01-23 15:14:50,177] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:14:50,224] - [task] [139635082422016] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.88:8091',default bucket [2019-01-23 15:14:50,239] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.85 [2019-01-23 15:14:50,244] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:14:50,268] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C2 for default is 0 [2019-01-23 15:14:50,268] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.87 [2019-01-23 15:14:50,273] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:14:50,298] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C3 for default is 0 [2019-01-23 15:14:50,299] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.85 [2019-01-23 15:14:50,336] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0 [2019-01-23 15:14:50,336] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.87 [2019-01-23 15:14:50,366] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C3 for bucket default is 0 [2019-01-23 15:14:50,383] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:14:50,409] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C2, default bucket [2019-01-23 15:14:50,416] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:14:50,437] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C2, default bucket [2019-01-23 15:14:50,457] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:14:50,479] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C3, default bucket [2019-01-23 15:14:50,489] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:14:50,513] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C3, default bucket [2019-01-23 15:14:50,551] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:14:50,598] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:14:50,720] - [task] [139642968725248] - INFO - 21111 items will be verified on default bucket [2019-01-23 15:14:51,565] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:14:54,026] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:14:56,559] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:14:56,842] - [task] [139634402973440] - INFO - 21111 items were verified in 6.12110304832 sec.the average number of ops - 3448.88765051 per second [2019-01-23 15:14:56,882] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:14:56,935] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:14:57,051] - [task] [139642968725248] - INFO - 21111 items will be verified on default bucket [2019-01-23 15:14:57,233] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:14:59,715] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:15:02,170] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:15:02,414] - [task] [139634402973440] - INFO - 21111 items were verified in 5.3623380661 sec.the average number of ops - 3936.90140108 per second [2019-01-23 15:15:02,415] - [xdcrnewbasetests] [139642968725248] - INFO - Verifying RevIds for Couchbase Cluster: C2, Master Ip: 172.23.97.85 -> Couchbase Cluster: C3, Master Ip: 172.23.97.87, bucket default [2019-01-23 15:15:02,462] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:15:02,511] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:15:02,702] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:15:02,758] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:15:02,841] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:15:02,898] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:15:02,997] - [task] [139642968725248] - INFO - RevID verification: in progress for default ... [2019-01-23 15:15:26,960] - [task] [139634402973440] - INFO - RevId Verification : 21111 existing items have been verified [2019-01-23 15:15:26,960] - [task] [139634402973440] - INFO - RevId Verification : 0 deleted items have been verified [2019-01-23 15:15:26,961] - [task] [139634402973440] - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket [2019-01-23 15:15:26,965] - [rest_client] [139634402973440] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:15:27,053] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'} [2019-01-23 15:15:27,053] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:15:27,097] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:15:27,098] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (651966730, 0, '') [2019-01-23 15:15:27,114] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'} [2019-01-23 15:15:27,114] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:15:27,154] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:15:27,154] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (846107136, 0, '') [2019-01-23 15:15:27,154] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:15:37,190] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'} [2019-01-23 15:15:37,190] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:15:37,288] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:15:37,289] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.85', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2372370103, 0, '') [2019-01-23 15:15:37,306] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'} [2019-01-23 15:15:37,306] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:15:37,343] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:15:37,344] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.86', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3818992479, 0, '') [2019-01-23 15:15:37,344] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:15:48,337] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:15:48,381] - [task] [139635082422016] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.87:8091',default bucket [2019-01-23 15:15:48,390] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:15:48,439] - [task] [139635082422016] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.88:8091',default bucket [2019-01-23 15:15:48,686] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:15:48,733] - [task] [139634881074944] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.85:8091',default bucket [2019-01-23 15:15:48,742] - [data_helper] [139634881074944] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:15:48,786] - [task] [139634881074944] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.86:8091',default bucket [2019-01-23 15:15:48,800] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.87 [2019-01-23 15:15:48,806] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:15:48,835] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C3 for default is 0 [2019-01-23 15:15:48,835] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.85 [2019-01-23 15:15:48,840] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:15:48,865] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C2 for default is 0 [2019-01-23 15:15:48,865] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.87 [2019-01-23 15:15:48,897] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C3 for bucket default is 0 [2019-01-23 15:15:48,898] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.85 [2019-01-23 15:15:48,930] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0 [2019-01-23 15:15:48,946] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:15:48,970] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C3, default bucket [2019-01-23 15:15:48,978] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:15:49,000] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C3, default bucket [2019-01-23 15:15:49,019] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:15:49,042] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C2, default bucket [2019-01-23 15:15:49,050] - [rest_client] [139642968725248] - INFO - http://172.23.97.85:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:15:49,072] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C2, default bucket [2019-01-23 15:15:49,111] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:15:49,159] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:15:49,262] - [task] [139642968725248] - INFO - 21111 items will be verified on default bucket [2019-01-23 15:15:49,441] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:15:51,968] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:15:54,446] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:15:54,718] - [task] [139634402973440] - INFO - 21111 items were verified in 5.4560239315 sec.the average number of ops - 3869.3003646 per second [2019-01-23 15:15:54,763] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:15:54,879] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:15:55,002] - [task] [139642968725248] - INFO - 21111 items will be verified on default bucket [2019-01-23 15:15:55,833] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:15:58,291] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:16:00,790] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:16:01,077] - [task] [139634402973440] - INFO - 21111 items were verified in 6.07519102097 sec.the average number of ops - 3474.95191726 per second [2019-01-23 15:16:01,078] - [xdcrnewbasetests] [139642968725248] - INFO - Verifying RevIds for Couchbase Cluster: C3, Master Ip: 172.23.97.87 -> Couchbase Cluster: C2, Master Ip: 172.23.97.85, bucket default [2019-01-23 15:16:01,116] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:16:01,169] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:16:01,300] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:16:01,355] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:16:01,430] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.85:11210 default [2019-01-23 15:16:01,546] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.86:11210 default [2019-01-23 15:16:01,633] - [task] [139642968725248] - INFO - RevID verification: in progress for default ... [2019-01-23 15:16:25,148] - [task] [139634402973440] - INFO - RevId Verification : 21111 existing items have been verified [2019-01-23 15:16:25,148] - [task] [139634402973440] - INFO - RevId Verification : 0 deleted items have been verified [2019-01-23 15:16:25,148] - [task] [139634402973440] - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket [2019-01-23 15:16:25,153] - [rest_client] [139634402973440] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:16:25,234] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'} [2019-01-23 15:16:25,234] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:16:25,273] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:16:25,274] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.87', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1913334837, 0, '') [2019-01-23 15:16:25,291] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'} [2019-01-23 15:16:25,291] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:16:25,331] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:16:25,332] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.88', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (667239272, 0, '') [2019-01-23 15:16:25,332] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:16:35,367] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'} [2019-01-23 15:16:35,367] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:16:35,405] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:16:35,406] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.83', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1278255308, 0, '') [2019-01-23 15:16:35,427] - [data_helper] [139642968725248] - INFO - dict:{'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'} [2019-01-23 15:16:35,428] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:16:35,465] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default [2019-01-23 15:16:35,466] - [cluster_helper] [139642968725248] - INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.97.84', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2382801827, 0, '') [2019-01-23 15:16:35,466] - [xdcrnewbasetests] [139642968725248] - INFO - wait for expiry pager to run on all these nodes [2019-01-23 15:16:45,546] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:16:45,593] - [task] [139635082422016] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.87:8091',default bucket [2019-01-23 15:16:45,603] - [data_helper] [139635082422016] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:16:45,651] - [task] [139635082422016] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.88:8091',default bucket [2019-01-23 15:16:46,549] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:16:46,598] - [task] [139634797430528] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.83:8091',default bucket [2019-01-23 15:16:46,606] - [data_helper] [139634797430528] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:16:46,698] - [task] [139634797430528] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.84:8091',default bucket [2019-01-23 15:16:46,730] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.87 [2019-01-23 15:16:46,736] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:16:46,763] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C3 for default is 0 [2019-01-23 15:16:46,763] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for dcp queue to drain on cluster node: 172.23.97.83 [2019-01-23 15:16:46,768] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:16:46,801] - [xdcrnewbasetests] [139642968725248] - INFO - Current dcp queue size on C1 for default is 0 [2019-01-23 15:16:46,801] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.87 [2019-01-23 15:16:46,836] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C3 for bucket default is 0 [2019-01-23 15:16:46,837] - [xdcrnewbasetests] [139642968725248] - INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.97.83 [2019-01-23 15:16:46,870] - [xdcrnewbasetests] [139642968725248] - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0 [2019-01-23 15:16:46,888] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:16:46,916] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 21111 == 21111 expected on C3, default bucket [2019-01-23 15:16:46,927] - [rest_client] [139642968725248] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:16:46,953] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 21111 == 21111 expected on C3, default bucket [2019-01-23 15:16:46,977] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:16:47,003] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_active_curr_items 30000 == 30000 expected on C1, default bucket [2019-01-23 15:16:47,015] - [rest_client] [139642968725248] - INFO - http://172.23.97.83:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:16:47,038] - [xdcrnewbasetests] [139642968725248] - INFO - Saw: vb_replica_curr_items 30000 == 30000 expected on C1, default bucket [2019-01-23 15:16:47,077] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:16:47,125] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:16:47,241] - [task] [139642968725248] - INFO - 21111 items will be verified on default bucket [2019-01-23 15:16:47,683] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:16:50,179] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:16:52,597] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:16:52,885] - [task] [139634402973440] - INFO - 21111 items were verified in 5.64432215691 sec.the average number of ops - 3740.21819559 per second [2019-01-23 15:16:52,926] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:16:52,980] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:16:53,107] - [task] [139642968725248] - INFO - 30000 items will be verified on default bucket [2019-01-23 15:16:53,706] - [task] [139634402973440] - INFO - 0 items were verified [2019-01-23 15:16:56,220] - [task] [139634402973440] - INFO - 10000 items were verified [2019-01-23 15:16:58,741] - [task] [139634402973440] - INFO - 20000 items were verified [2019-01-23 15:17:00,906] - [task] [139634402973440] - INFO - 30000 items were verified [2019-01-23 15:17:00,906] - [task] [139634402973440] - INFO - 30000 items were verified in 7.7985060215 sec.the average number of ops - 3846.88231289 per second [2019-01-23 15:17:00,906] - [xdcrnewbasetests] [139642968725248] - INFO - Verifying RevIds for Couchbase Cluster: C3, Master Ip: 172.23.97.87 -> Couchbase Cluster: C1, Master Ip: 172.23.97.83, bucket default [2019-01-23 15:17:00,948] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:17:01,005] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:17:01,176] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.87:11210 default [2019-01-23 15:17:01,225] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.88:11210 default [2019-01-23 15:17:01,303] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.83:11210 default [2019-01-23 15:17:01,351] - [data_helper] [139642968725248] - INFO - creating direct client 172.23.97.84:11210 default [2019-01-23 15:17:01,446] - [task] [139642968725248] - INFO - RevID verification: in progress for default ... [2019-01-23 15:17:26,669] - [task] [139634402973440] - INFO - RevId Verification : 21111 existing items have been verified [2019-01-23 15:17:26,669] - [task] [139634402973440] - INFO - RevId Verification : 0 deleted items have been verified [2019-01-23 15:17:26,670] - [task] [139634402973440] - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket [2019-01-23 15:17:26,674] - [rest_client] [139634402973440] - INFO - http://172.23.97.87:8091/pools/default/buckets/default/stats?zoom=minute [2019-01-23 15:17:26,733] - [rest_client] [139642968725248] - INFO - /diag/eval status on 172.23.97.83:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-01-23 15:17:26,734] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.83 with username:root password:couchbase ssh_key: [2019-01-23 15:17:26,832] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.83 [2019-01-23 15:17:27,083] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:27,485] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2019-01-23 15:17:27,583] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:27,583] - [xdcrnewbasetests] [139642968725248] - INFO - ['2019-01-23T13:41:51.643-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2019-01-23T14:47:56.993-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 62e1bb55e9499a55fc4373729acd8272/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... ', '2019-01-23T15:04:07.654-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=57 is closed by producer ', '2019-01-23T15:04:07.712-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=54 is closed by producer ', '2019-01-23T15:04:19.551-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:04:19.608-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:04:29.571-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:04:29.630-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:04:39.595-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:04:39.649-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:04:49.616-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:04:49.667-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:04:59.639-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:04:59.688-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:05:09.661-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:05:09.708-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:05:19.683-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T15:05:19.726-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. '] [2019-01-23 15:17:27,583] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:27,681] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:27,681] - [xdcrnewbasetests] [139642968725248] - INFO - 18 [2019-01-23 15:17:27,795] - [xdcrnewbasetests] [139642968725248] - INFO - Initial replication restart count on 172.23.97.83 :0, now :18 [2019-01-23 15:17:27,795] - [xdcrnewbasetests] [139642968725248] - INFO - Number of new replication restarts this run: 18 [2019-01-23 15:17:27,795] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:41:51.643-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2019-01-23 15:17:27,796] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:47:56.993-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 62e1bb55e9499a55fc4373729acd8272/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... [2019-01-23 15:17:27,796] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:07.654-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=57 is closed by producer [2019-01-23 15:17:27,796] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:07.712-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=54 is closed by producer [2019-01-23 15:17:27,796] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:19.551-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,796] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:19.608-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,796] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:29.571-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,796] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:29.630-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,796] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:39.595-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,796] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:39.649-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,796] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:49.616-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,796] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:49.667-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,797] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:59.639-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,797] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:04:59.688-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,797] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:05:09.661-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,797] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:05:09.708-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,797] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:05:19.683-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(5e01cdb750b65f73af652fdbc2fb9264/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,797] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T15:05:19.726-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:27,798] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.84 with username:root password:couchbase ssh_key: [2019-01-23 15:17:27,895] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.84 [2019-01-23 15:17:28,162] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:28,568] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.84: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2019-01-23 15:17:28,666] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:28,666] - [xdcrnewbasetests] [139642968725248] - INFO - ['2019-01-23T13:41:51.647-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2019-01-23T13:47:48.556-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=genericPipeline.context.Start : Failed to start service CheckpointManager for d8398a9e06f32debccecfcbeccc300a2/default/default. err=dial tcp 172.23.97.86:11210: getsockopt: connection refused ', '2019-01-23T14:47:57.004-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 62e1bb55e9499a55fc4373729acd8272/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... ', '2019-01-23T14:59:01.748-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/standard_bucket_1/standard_bucket_1. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=607 is closed by producer ', '2019-01-23T14:59:01.799-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/standard_bucket_1/standard_bucket_1. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=606 is closed by producer '] [2019-01-23 15:17:28,666] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.84: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:28,796] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:28,796] - [xdcrnewbasetests] [139642968725248] - INFO - 5 [2019-01-23 15:17:28,910] - [xdcrnewbasetests] [139642968725248] - INFO - Initial replication restart count on 172.23.97.84 :0, now :5 [2019-01-23 15:17:28,911] - [xdcrnewbasetests] [139642968725248] - INFO - Number of new replication restarts this run: 5 [2019-01-23 15:17:28,911] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:41:51.647-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2019-01-23 15:17:28,912] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:47:48.556-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=genericPipeline.context.Start : Failed to start service CheckpointManager for d8398a9e06f32debccecfcbeccc300a2/default/default. err=dial tcp 172.23.97.86:11210: getsockopt: connection refused [2019-01-23 15:17:28,912] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:47:57.004-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 62e1bb55e9499a55fc4373729acd8272/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... [2019-01-23 15:17:28,912] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:59:01.748-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline 5e01cdb750b65f73af652fdbc2fb9264/standard_bucket_1/standard_bucket_1. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=607 is closed by producer [2019-01-23 15:17:28,912] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:59:01.799-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/standard_bucket_1/standard_bucket_1. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=606 is closed by producer [2019-01-23 15:17:28,915] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.85 with username:root password:couchbase ssh_key: [2019-01-23 15:17:29,013] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.85 [2019-01-23 15:17:29,254] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:29,665] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2019-01-23 15:17:29,762] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:29,762] - [xdcrnewbasetests] [139642968725248] - INFO - ['2019-01-23T13:37:21.694-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=408 is closed by producer ', '2019-01-23T13:42:12.885-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2019-01-23T14:35:35.686-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=266 is closed by producer ', '2019-01-23T14:35:35.746-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=264 is closed by producer ', '2019-01-23T14:43:07.079-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=465 is closed by producer ', '2019-01-23T14:47:58.186-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2019-01-23T14:52:43.220-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=165 is closed by producer ', '2019-01-23T14:52:46.596-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : dcp_f7f021251634dfe770fccec884116b2b/default/default_172.23.97.85:11210_1:DCP upr feed has been closed. ', '2019-01-23T14:52:57.454-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T14:53:07.474-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T14:53:17.493-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T14:53:27.513-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T14:53:37.533-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T14:53:47.553-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. '] [2019-01-23 15:17:29,762] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:29,860] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:29,860] - [xdcrnewbasetests] [139642968725248] - INFO - 14 [2019-01-23 15:17:29,974] - [xdcrnewbasetests] [139642968725248] - INFO - Initial replication restart count on 172.23.97.85 :0, now :14 [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - Number of new replication restarts this run: 14 [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:37:21.694-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=408 is closed by producer [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:42:12.885-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:35:35.686-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=266 is closed by producer [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:35:35.746-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=264 is closed by producer [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:43:07.079-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=465 is closed by producer [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:47:58.186-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:52:43.220-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:dcp stream for vb=165 is closed by producer [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:52:46.596-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : dcp_f7f021251634dfe770fccec884116b2b/default/default_172.23.97.85:11210_1:DCP upr feed has been closed. [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:52:57.454-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:53:07.474-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:29,975] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:53:17.493-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:29,976] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:53:27.513-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:29,976] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:53:37.533-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:29,976] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:53:47.553-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:29,977] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.86 with username:root password:couchbase ssh_key: [2019-01-23 15:17:30,075] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.86 [2019-01-23 15:17:30,318] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:30,731] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.86: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2019-01-23 15:17:30,828] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:30,828] - [xdcrnewbasetests] [139642968725248] - INFO - ['2019-01-23T13:41:51.711-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... ', '2019-01-23T14:40:25.719-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2019-01-23T14:47:57.056-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... '] [2019-01-23 15:17:30,828] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.86: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:30,893] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:30,894] - [xdcrnewbasetests] [139642968725248] - INFO - 3 [2019-01-23 15:17:31,008] - [xdcrnewbasetests] [139642968725248] - INFO - Initial replication restart count on 172.23.97.86 :0, now :3 [2019-01-23 15:17:31,008] - [xdcrnewbasetests] [139642968725248] - INFO - Number of new replication restarts this run: 3 [2019-01-23 15:17:31,008] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:41:51.711-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... [2019-01-23 15:17:31,008] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:40:25.719-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2019-01-23 15:17:31,008] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:47:57.056-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2019-01-23 15:17:31,009] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.87 with username:root password:couchbase ssh_key: [2019-01-23 15:17:31,106] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.87 [2019-01-23 15:17:31,346] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:31,741] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2019-01-23 15:17:31,823] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:31,823] - [xdcrnewbasetests] [139642968725248] - INFO - ['2019-01-23T14:40:25.764-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... '] [2019-01-23 15:17:31,823] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:31,922] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:31,922] - [xdcrnewbasetests] [139642968725248] - INFO - 1 [2019-01-23 15:17:32,036] - [xdcrnewbasetests] [139642968725248] - INFO - Initial replication restart count on 172.23.97.87 :0, now :1 [2019-01-23 15:17:32,036] - [xdcrnewbasetests] [139642968725248] - INFO - Number of new replication restarts this run: 1 [2019-01-23 15:17:32,036] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:40:25.764-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to target topology change... [2019-01-23 15:17:32,038] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.88 with username:root password:couchbase ssh_key: [2019-01-23 15:17:32,135] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.88 [2019-01-23 15:17:32,405] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:32,803] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.88: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2019-01-23 15:17:32,884] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:32,884] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.88: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:32,966] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:32,966] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:33,080] - [xdcrnewbasetests] [139642968725248] - INFO - Initial replication restart count on 172.23.97.88 :0, now :0 [2019-01-23 15:17:33,082] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key: [2019-01-23 15:17:33,180] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.11 [2019-01-23 15:17:33,395] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:33,815] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.11: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2019-01-23 15:17:33,896] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:33,896] - [xdcrnewbasetests] [139642968725248] - INFO - ['2019-01-23T13:37:23.618-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c95c313a111930261326c5f3571db949/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T13:42:13.750-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... '] [2019-01-23 15:17:33,896] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.11: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:33,977] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:33,978] - [xdcrnewbasetests] [139642968725248] - INFO - 2 [2019-01-23 15:17:34,092] - [xdcrnewbasetests] [139642968725248] - INFO - Initial replication restart count on 172.23.98.11 :0, now :2 [2019-01-23 15:17:34,092] - [xdcrnewbasetests] [139642968725248] - INFO - Number of new replication restarts this run: 2 [2019-01-23 15:17:34,092] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:37:23.618-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(c95c313a111930261326c5f3571db949/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:34,092] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:42:13.750-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline c95c313a111930261326c5f3571db949/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2019-01-23 15:17:34,094] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2019-01-23 15:17:34,191] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.164 [2019-01-23 15:17:34,436] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:34,818] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.164: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2019-01-23 15:17:34,901] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:34,901] - [xdcrnewbasetests] [139642968725248] - INFO - ['2019-01-23T13:37:01.502-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(d8398a9e06f32debccecfcbeccc300a2/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T13:42:01.601-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... ', '2019-01-23T14:43:12.619-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. ', '2019-01-23T14:48:02.744-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... '] [2019-01-23 15:17:34,901] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.164: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:34,983] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:34,983] - [xdcrnewbasetests] [139642968725248] - INFO - 4 [2019-01-23 15:17:35,097] - [xdcrnewbasetests] [139642968725248] - INFO - Initial replication restart count on 172.23.98.164 :0, now :4 [2019-01-23 15:17:35,097] - [xdcrnewbasetests] [139642968725248] - INFO - Number of new replication restarts this run: 4 [2019-01-23 15:17:35,097] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:37:01.502-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(d8398a9e06f32debccecfcbeccc300a2/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:35,097] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:42:01.601-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline d8398a9e06f32debccecfcbeccc300a2/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2019-01-23 15:17:35,097] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:43:12.619-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=pipelineMgr.pipeline_factory.NewPipeline(f7f021251634dfe770fccec884116b2b/default/default) : Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets. [2019-01-23 15:17:35,097] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T14:48:02.744-08:00 INFO GOXDCR.PipelineMgr: Try to fix Pipeline f7f021251634dfe770fccec884116b2b/default/default. Current error(s)=r.update_err_ch : TopoChangeDet:Restarting pipeline due to source topology change... [2019-01-23 15:17:35,105] - [rest_client] [139642968725248] - INFO - /diag/eval status on 172.23.97.83:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-01-23 15:17:35,107] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.83 with username:root password:couchbase ssh_key: [2019-01-23 15:17:35,204] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.83 [2019-01-23 15:17:35,442] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:35,849] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:35,979] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:35,979] - [xdcrnewbasetests] [139642968725248] - INFO - 1 [2019-01-23 15:17:36,093] - [xdcrnewbasetests] [139642968725248] - INFO - Initial panic: count on 172.23.97.83 :0, now :1 [2019-01-23 15:17:36,095] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.83 with username:root password:couchbase ssh_key: [2019-01-23 15:17:36,176] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.83 [2019-01-23 15:17:36,396] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:36,807] - [rest_client] [139642968725248] - INFO - /diag/eval status on 172.23.97.83:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-01-23 15:17:36,807] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: zgrep -A 40 'panic:' "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* [2019-01-23 15:17:36,920] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:36,921] - [xdcrnewbasetests] [139642968725248] - INFO - panic: interface conversion: interface {} is nil, not string [2019-01-23 15:17:36,921] - [xdcrnewbasetests] [139642968725248] - INFO - [2019-01-23 15:17:36,921] - [xdcrnewbasetests] [139642968725248] - INFO - goroutine 75 [running]: [2019-01-23 15:17:36,921] - [xdcrnewbasetests] [139642968725248] - INFO - github.com/couchbase/goxdcr/metadata.ReplicationSettingsMap.CloneAndRedact(0xc42622b4d0, 0xc42622b530) [2019-01-23 15:17:36,921] - [xdcrnewbasetests] [139642968725248] - INFO - goproj/src/github.com/couchbase/goxdcr/metadata/replication_settings.go:530 +0x40f [2019-01-23 15:17:36,921] - [xdcrnewbasetests] [139642968725248] - INFO - github.com/couchbase/goxdcr/replication_manager.(*Adminport).doCreateReplicationRequest(0xc42012ccf0, 0xc421accc00, 0x0, 0x0, 0x0) [2019-01-23 15:17:36,921] - [xdcrnewbasetests] [139642968725248] - INFO - goproj/src/github.com/couchbase/goxdcr/replication_manager/adminport.go:409 +0x298 [2019-01-23 15:17:36,921] - [xdcrnewbasetests] [139642968725248] - INFO - github.com/couchbase/goxdcr/replication_manager.(*Adminport).handleRequest(0xc42012ccf0, 0xc421accc00, 0xc4278f31e0, 0xcc89a0, 0xc4278f31e0) [2019-01-23 15:17:36,921] - [xdcrnewbasetests] [139642968725248] - INFO - goproj/src/github.com/couchbase/goxdcr/replication_manager/adminport.go:172 +0x1093 [2019-01-23 15:17:36,921] - [xdcrnewbasetests] [139642968725248] - INFO - github.com/couchbase/goxdcr/replication_manager.(*Adminport).processRequest(0xc42012ccf0, 0xc427cda000, 0x1, 0x1, 0xd00000, 0xc427cda010) [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - goproj/src/github.com/couchbase/goxdcr/replication_manager/adminport.go:136 +0xf5 [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - github.com/couchbase/goxdcr/replication_manager.(*Adminport).(github.com/couchbase/goxdcr/replication_manager.processRequest)-fm(0xc427cda000, 0x1, 0x1, 0x1, 0x1) [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - goproj/src/github.com/couchbase/goxdcr/replication_manager/adminport.go:69 +0x48 [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - github.com/couchbase/goxdcr/gen_server.(*GenServer).run(0xc42012cd08) [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - goproj/src/github.com/couchbase/goxdcr/gen_server/gen_server.go:104 +0x4a4 [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - created by github.com/couchbase/goxdcr/gen_server.(*GenServer).Start_server [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - goproj/src/github.com/couchbase/goxdcr/gen_server/gen_server.go:74 +0xba [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - [goport(/opt/couchbase/bin/goxdcr)] 2019/01/23 13:00:55 child process exited with status 2 [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.577-08:00 INFO GOXDCR.ReplMgr: GOMAXPROCS=4 [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.579-08:00 INFO GOXDCR.AuditSvc: Created audit service. [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.579-08:00 INFO GOXDCR.UILogSvc: Created ui log service. [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.582-08:00 INFO GOXDCR.RemClusterSvc: Agent remoteCluster/EYaZs6-162MdFYreZjYF7oZiodxlOS5bVuWB2tsBiKw= remote_cluster_C1-C2 started for cluster: d8398a9e06f32debccecfcbeccc300a2 [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.582-08:00 INFO GOXDCR.ReplSpecSvc: Cache has been initialized for ReplicationSpecService [2019-01-23 15:17:36,922] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.585-08:00 INFO GOXDCR.IntSettSvc: Internal settings not found. Using default values [2019-01-23 15:17:36,923] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.585-08:00 INFO GOXDCR.ReplMgr: pollEOF: About to start stdin polling [2019-01-23 15:17:36,923] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.589-08:00 INFO GOXDCR.ReplMgr: XDCR internal settings: map[TimeoutDcpCloseUprStreams:3 BypassSanInCertificateCheck:0 DefaultHttpTimeout:180 CapiDataChanSizeMultiplier:1 TimeoutCheckpointBeforeStop:180 XmemWriteTimeout:120 MaxCheckpointRecordsToRead:5 TimeoutRuntimeContextStop:5 XmemMaxRetry:5 MaxLengthSnapshotHistory:200 XmemMaxRetryNewConn:10 AuditWriteTimeout:1 ReplSpecCheckInterval:15 MaxWorkersForCheckpointing:5 XmemMaxReadDownTime:60 ShortHttpTimeout:20 UprFeedBufferSize:1048576 MemStatsLogInterval:120 XmemBackoffWaitTime:10 TimeoutPartsStart:30 CapiRetryInterval:500 MaxTopologyStableCountBeforeRestart:30 XmemMaxIdleCount:60 RetryIntervalMetakv:500 MaxCheckpointRecordsToKeep:5 ThresholdForThroughSeqnoComputation:100 WaitTimeBetweenMetadataChangeListeners:1000 CapiMaxRetryBatchUpdateDocs:6 RetryIntervalTargetStats:1000 MaxNumberOfAsyncListeners:4 MaxRetryCapiService:5 HELOTimeout:120 TimeoutPartsStop:10 MaxTopologyChangeCountBeforeRestart:30 XmemDefaultRespTimeout:1000 TimeoutDcpCloseUprFeed:3 StatsLogInterval:30 WaitTimeForLiveUpdatePipeline:2000 PercentageOfBytesToSendAsMin:30 CapiWriteTimeout:10 AuditReadTimeout:1 MaxNumOfMetakvRetries:4 TopologyChangeCheckInterval:10 ReplicationSpecGCCnt:4 XmemMaxDataChanSize:10485760 UprFeedDataChanLength:20000 KeepAlivePeriod:30 RefreshRemoteClusterRefInterval:15 CapiBatchTimeout:180 MaxRetryTargetStats:6 XmemSelfMonitorInterval:6 TimeoutRuntimeContextStart:30 XmemMaxRetryInterval:300 XmemReadTimeout:120 MaxRetryForLiveUpdatePipeline:5 CapiReadTimeout:60 NumberOfSlotsForBandwidthThrottling:10 ThresholdPercentageForEventChanSizeLogging:90 XmemBackoffTimeNewConn:1000 XmemMaxBatchSize:50] [2019-01-23 15:17:36,923] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.GenericSupervisor: Adding child PipelineMasterSupervisor to supervisor ReplicationManagerSupervisor [2019-01-23 15:17:36,923] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.PipelineMgr: Pipeline Manager is constucted [2019-01-23 15:17:36,923] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.ReplMgr: Replication manager is initialized [2019-01-23 15:17:36,923] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.GenericSupervisor: Starting supervisor PipelineMasterSupervisor. [2019-01-23 15:17:36,923] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.GenericSupervisor: Started supervisor PipelineMasterSupervisor. [2019-01-23 15:17:36,923] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.ReplMgr: Master supervisor has started [2019-01-23 15:17:36,923] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.GenericSupervisor: Starting supervisor ReplicationManagerSupervisor. [2019-01-23 15:17:36,923] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.GenericSupervisor: Started supervisor ReplicationManagerSupervisor. [2019-01-23 15:17:36,923] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.ReplMgr: initReplications succeeded [2019-01-23 15:17:36,924] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.ReplMgr: upgradeRemoteClusterRefs started. [2019-01-23 15:17:36,924] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.ReplMgr: upgradeRemoteClusterRefs exited [2019-01-23 15:17:36,924] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.GlobalSettingChangeListener: Started MetakvChangeListener GlobalSettingChangeListener [2019-01-23 15:17:36,925] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.ReplMgr: checkReplicationStatus started. [2019-01-23 15:17:36,926] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:55.590-08:00 INFO GOXDCR.ReplMgr: logMemStats started. [2019-01-23 15:17:36,926] - [xdcrnewbasetests] [139642968725248] - INFO - 2019-01-23T13:00:56.590-08:00 INFO GOXDCR.InternalSettingChangeListener: Started MetakvChangeListener InternalSettingsChangeListener [2019-01-23 15:17:37,041] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.83 with username:root password:couchbase ssh_key: [2019-01-23 15:17:37,122] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.83 [2019-01-23 15:17:37,392] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:37,804] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:37,901] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:37,902] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:38,016] - [xdcrnewbasetests] [139642968725248] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.97.83 :0, now :0 [2019-01-23 15:17:38,017] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.84 with username:root password:couchbase ssh_key: [2019-01-23 15:17:38,115] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.84 [2019-01-23 15:17:38,387] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:38,835] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.84: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:38,965] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:38,965] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:39,080] - [xdcrnewbasetests] [139642968725248] - INFO - Initial panic: count on 172.23.97.84 :0, now :0 [2019-01-23 15:17:39,081] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.84 with username:root password:couchbase ssh_key: [2019-01-23 15:17:39,162] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.84 [2019-01-23 15:17:39,436] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:39,859] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.84: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:39,990] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:39,990] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:40,104] - [xdcrnewbasetests] [139642968725248] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.97.84 :0, now :0 [2019-01-23 15:17:40,106] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.85 with username:root password:couchbase ssh_key: [2019-01-23 15:17:40,203] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.85 [2019-01-23 15:17:40,474] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:40,871] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:40,969] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:40,969] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:41,083] - [xdcrnewbasetests] [139642968725248] - INFO - Initial panic: count on 172.23.97.85 :0, now :0 [2019-01-23 15:17:41,084] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.85 with username:root password:couchbase ssh_key: [2019-01-23 15:17:41,165] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.85 [2019-01-23 15:17:41,439] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:41,840] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:41,938] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:41,938] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:42,052] - [xdcrnewbasetests] [139642968725248] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.97.85 :0, now :0 [2019-01-23 15:17:42,053] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.86 with username:root password:couchbase ssh_key: [2019-01-23 15:17:42,134] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.86 [2019-01-23 15:17:42,381] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:42,757] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.86: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:42,854] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:42,855] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:42,969] - [xdcrnewbasetests] [139642968725248] - INFO - Initial panic: count on 172.23.97.86 :0, now :0 [2019-01-23 15:17:42,970] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.86 with username:root password:couchbase ssh_key: [2019-01-23 15:17:43,052] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.86 [2019-01-23 15:17:43,265] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:43,661] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.86: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:43,759] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:43,759] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:43,873] - [xdcrnewbasetests] [139642968725248] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.97.86 :0, now :0 [2019-01-23 15:17:43,875] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.87 with username:root password:couchbase ssh_key: [2019-01-23 15:17:43,972] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.87 [2019-01-23 15:17:44,186] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:44,583] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:44,664] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:44,664] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:44,778] - [xdcrnewbasetests] [139642968725248] - INFO - Initial panic: count on 172.23.97.87 :0, now :0 [2019-01-23 15:17:44,780] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.87 with username:root password:couchbase ssh_key: [2019-01-23 15:17:44,877] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.87 [2019-01-23 15:17:45,125] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:45,511] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:45,592] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:45,592] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:45,707] - [xdcrnewbasetests] [139642968725248] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.97.87 :0, now :0 [2019-01-23 15:17:45,708] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.88 with username:root password:couchbase ssh_key: [2019-01-23 15:17:45,806] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.88 [2019-01-23 15:17:46,077] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:46,483] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.88: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:46,565] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:46,565] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:46,679] - [xdcrnewbasetests] [139642968725248] - INFO - Initial panic: count on 172.23.97.88 :0, now :0 [2019-01-23 15:17:46,680] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.88 with username:root password:couchbase ssh_key: [2019-01-23 15:17:46,761] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.88 [2019-01-23 15:17:47,032] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:47,431] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.88: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:47,512] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:47,512] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:47,627] - [xdcrnewbasetests] [139642968725248] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.97.88 :0, now :0 [2019-01-23 15:17:47,628] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key: [2019-01-23 15:17:47,710] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.11 [2019-01-23 15:17:47,983] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:48,401] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.11: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:48,482] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:48,483] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:48,597] - [xdcrnewbasetests] [139642968725248] - INFO - Initial panic: count on 172.23.98.11 :0, now :0 [2019-01-23 15:17:48,598] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key: [2019-01-23 15:17:48,679] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.11 [2019-01-23 15:17:48,917] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:49,336] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.11: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:49,417] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:49,417] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:49,531] - [xdcrnewbasetests] [139642968725248] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.98.11 :0, now :0 [2019-01-23 15:17:49,533] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2019-01-23 15:17:49,630] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.164 [2019-01-23 15:17:49,877] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:50,269] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.164: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:50,350] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:50,350] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:50,465] - [xdcrnewbasetests] [139642968725248] - INFO - Initial panic: count on 172.23.98.164 :0, now :0 [2019-01-23 15:17:50,466] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2019-01-23 15:17:50,547] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.164 [2019-01-23 15:17:50,792] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:51,180] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.164: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2019-01-23 15:17:51,261] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:17:51,261] - [xdcrnewbasetests] [139642968725248] - INFO - 0 [2019-01-23 15:17:51,375] - [xdcrnewbasetests] [139642968725248] - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.98.164 :0, now :0 [2019-01-23 15:17:51,375] - [xdcrnewbasetests] [139642968725248] - ERROR - ['panic: found on 172.23.97.83'] [2019-01-23 15:17:51,375] - [xdcrnewbasetests] [139642968725248] - INFO - Collecting logs @ 172.23.97.83 [2019-01-23 15:17:51,377] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.83 with username:root password:couchbase ssh_key: [2019-01-23 15:17:51,474] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.83 [2019-01-23 15:17:51,689] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:17:52,103] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: /opt/couchbase/bin/cbcollect_info 172.23.97.83-20190123-1517-diag.zip [2019-01-23 15:19:21,120] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:19:21,682] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: rm -f /root/172.23.97.83-20190123-1517-diag.zip [2019-01-23 15:19:21,703] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:19:21,818] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.83 with username:root password:couchbase ssh_key: [2019-01-23 15:19:21,948] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.83 [2019-01-23 15:19:22,217] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:19:22,646] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: tar -zcvf 172.23.97.83-1232019-1519-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-01-23 15:19:23,294] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:19:23,525] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.83: rm -f /root/172.23.97.83-1232019-1519-couch.tar.gz [2019-01-23 15:19:23,546] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:19:23,660] - [xdcrnewbasetests] [139642968725248] - INFO - Collecting logs @ 172.23.97.84 [2019-01-23 15:19:23,662] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.84 with username:root password:couchbase ssh_key: [2019-01-23 15:19:23,760] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.84 [2019-01-23 15:19:24,011] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:19:24,419] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.84: /opt/couchbase/bin/cbcollect_info 172.23.97.84-20190123-1519-diag.zip [2019-01-23 15:20:52,880] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:20:53,429] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.84: rm -f /root/172.23.97.84-20190123-1519-diag.zip [2019-01-23 15:20:53,449] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:20:53,565] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.84 with username:root password:couchbase ssh_key: [2019-01-23 15:20:53,663] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.84 [2019-01-23 15:20:53,945] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:20:54,357] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.84: tar -zcvf 172.23.97.84-1232019-1520-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-01-23 15:20:55,057] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:20:55,209] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.84: rm -f /root/172.23.97.84-1232019-1520-couch.tar.gz [2019-01-23 15:20:55,230] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:20:55,344] - [xdcrnewbasetests] [139642968725248] - INFO - Collecting logs @ 172.23.97.85 [2019-01-23 15:20:55,346] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.85 with username:root password:couchbase ssh_key: [2019-01-23 15:20:55,443] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.85 [2019-01-23 15:20:55,691] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:20:56,100] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: /opt/couchbase/bin/cbcollect_info 172.23.97.85-20190123-1520-diag.zip [2019-01-23 15:22:24,596] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:22:25,019] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: rm -f /root/172.23.97.85-20190123-1520-diag.zip [2019-01-23 15:22:25,040] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:22:25,155] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.85 with username:root password:couchbase ssh_key: [2019-01-23 15:22:25,252] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.85 [2019-01-23 15:22:25,526] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:22:25,972] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: tar -zcvf 172.23.97.85-1232019-1522-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-01-23 15:22:26,470] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:22:26,633] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.85: rm -f /root/172.23.97.85-1232019-1522-couch.tar.gz [2019-01-23 15:22:26,651] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:22:26,765] - [xdcrnewbasetests] [139642968725248] - INFO - Collecting logs @ 172.23.97.86 [2019-01-23 15:22:26,767] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.86 with username:root password:couchbase ssh_key: [2019-01-23 15:22:26,864] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.86 [2019-01-23 15:22:27,134] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:22:27,510] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.86: /opt/couchbase/bin/cbcollect_info 172.23.97.86-20190123-1522-diag.zip [2019-01-23 15:23:56,795] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:23:57,288] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.86: rm -f /root/172.23.97.86-20190123-1522-diag.zip [2019-01-23 15:23:57,308] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:23:57,424] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.86 with username:root password:couchbase ssh_key: [2019-01-23 15:23:57,505] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.86 [2019-01-23 15:23:57,776] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:23:58,203] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.86: tar -zcvf 172.23.97.86-1232019-1523-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-01-23 15:23:58,703] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:23:58,854] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.86: rm -f /root/172.23.97.86-1232019-1523-couch.tar.gz [2019-01-23 15:23:58,875] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:23:58,989] - [xdcrnewbasetests] [139642968725248] - INFO - Collecting logs @ 172.23.97.87 [2019-01-23 15:23:58,991] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.87 with username:root password:couchbase ssh_key: [2019-01-23 15:23:59,088] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.87 [2019-01-23 15:23:59,335] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:23:59,741] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: /opt/couchbase/bin/cbcollect_info 172.23.97.87-20190123-1523-diag.zip [2019-01-23 15:25:29,811] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:25:30,136] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: rm -f /root/172.23.97.87-20190123-1523-diag.zip [2019-01-23 15:25:30,157] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:25:30,272] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.87 with username:root password:couchbase ssh_key: [2019-01-23 15:25:30,354] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.87 [2019-01-23 15:25:30,624] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:25:31,043] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: tar -zcvf 172.23.97.87-1232019-1525-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-01-23 15:25:31,541] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:25:31,730] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.87: rm -f /root/172.23.97.87-1232019-1525-couch.tar.gz [2019-01-23 15:25:31,751] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:25:31,865] - [xdcrnewbasetests] [139642968725248] - INFO - Collecting logs @ 172.23.97.88 [2019-01-23 15:25:31,866] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.88 with username:root password:couchbase ssh_key: [2019-01-23 15:25:31,964] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.88 [2019-01-23 15:25:32,237] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:25:32,601] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.88: /opt/couchbase/bin/cbcollect_info 172.23.97.88-20190123-1525-diag.zip [2019-01-23 15:26:57,457] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:26:57,683] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.88: rm -f /root/172.23.97.88-20190123-1525-diag.zip [2019-01-23 15:26:57,701] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:26:57,817] - [remote_util] [139642968725248] - INFO - connecting to 172.23.97.88 with username:root password:couchbase ssh_key: [2019-01-23 15:26:57,914] - [remote_util] [139642968725248] - INFO - Connected to 172.23.97.88 [2019-01-23 15:26:58,187] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:26:58,594] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.88: tar -zcvf 172.23.97.88-1232019-1526-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-01-23 15:26:59,142] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:26:59,296] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.97.88: rm -f /root/172.23.97.88-1232019-1526-couch.tar.gz [2019-01-23 15:26:59,317] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:26:59,431] - [xdcrnewbasetests] [139642968725248] - INFO - Collecting logs @ 172.23.98.11 [2019-01-23 15:26:59,433] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key: [2019-01-23 15:26:59,530] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.11 [2019-01-23 15:26:59,769] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:27:00,188] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.11: /opt/couchbase/bin/cbcollect_info 172.23.98.11-20190123-1526-diag.zip [2019-01-23 15:28:09,876] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:28:10,043] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.11: rm -f /root/172.23.98.11-20190123-1526-diag.zip [2019-01-23 15:28:10,062] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:28:10,177] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key: [2019-01-23 15:28:10,258] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.11 [2019-01-23 15:28:10,528] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:28:10,975] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.11: tar -zcvf 172.23.98.11-1232019-1528-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-01-23 15:28:11,024] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:28:11,141] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.11: rm -f /root/172.23.98.11-1232019-1528-couch.tar.gz [2019-01-23 15:28:11,159] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:28:11,274] - [xdcrnewbasetests] [139642968725248] - INFO - Collecting logs @ 172.23.98.164 [2019-01-23 15:28:11,276] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2019-01-23 15:28:11,373] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.164 [2019-01-23 15:28:11,642] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:28:12,034] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.164: /opt/couchbase/bin/cbcollect_info 172.23.98.164-20190123-1528-diag.zip [2019-01-23 15:29:20,535] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:29:20,725] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.164: rm -f /root/172.23.98.164-20190123-1528-diag.zip [2019-01-23 15:29:20,742] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:29:20,858] - [remote_util] [139642968725248] - INFO - connecting to 172.23.98.164 with username:root password:couchbase ssh_key: [2019-01-23 15:29:20,939] - [remote_util] [139642968725248] - INFO - Connected to 172.23.98.164 [2019-01-23 15:29:21,209] - [remote_util] [139642968725248] - INFO - It could be other operating system. Go to check at other location [2019-01-23 15:29:21,596] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.164: tar -zcvf 172.23.98.164-1232019-1529-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-01-23 15:29:21,676] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:29:21,794] - [remote_util] [139642968725248] - INFO - running command.raw on 172.23.98.164: rm -f /root/172.23.98.164-1232019-1529-couch.tar.gz [2019-01-23 15:29:21,867] - [remote_util] [139642968725248] - INFO - command executed successfully [2019-01-23 15:29:22,036] - [xdcrnewbasetests] [139642968725248] - INFO - ==== XDCRNewbasetests cleanup is started for test #34 load_with_ops ==== [2019-01-23 15:29:22,036] - [xdcrnewbasetests] [139642968725248] - INFO - removing xdcr/nodes settings [2019-01-23 15:29:22,046] - [rest_client] [139642968725248] - INFO - Deleting replication /controller/cancelXDCR/5e01cdb750b65f73af652fdbc2fb9264%2Fdefault%2Fdefault [2019-01-23 15:29:22,056] - [rest_client] [139642968725248] - INFO - Replication deleted successfully [2019-01-23 15:29:22,056] - [rest_client] [139642968725248] - INFO - Deleting replication /controller/cancelXDCR/f7f021251634dfe770fccec884116b2b%2Fdefault%2Fdefault [2019-01-23 15:29:22,063] - [rest_client] [139642968725248] - INFO - Replication deleted successfully [2019-01-23 15:29:22,068] - [rest_client] [139642968725248] - INFO - removing remote cluster name:remote_cluster_C1-C2 [2019-01-23 15:29:22,075] - [rest_client] [139642968725248] - INFO - removing remote cluster name:remote_cluster_C1-C3 [2019-01-23 15:29:22,090] - [xdcrnewbasetests] [139642968725248] - INFO - cleanup [ip:172.23.97.83 port:8091 ssh_username:root, ip:172.23.97.84 port:8091 ssh_username:root] [2019-01-23 15:29:22,104] - [bucket_helper] [139642968725248] - INFO - deleting existing buckets [u'default'] on 172.23.97.83 [2019-01-23 15:29:22,104] - [bucket_helper] [139642968725248] - INFO - remove bucket default ... [2019-01-23 15:29:23,178] - [bucket_helper] [139642968725248] - INFO - deleted bucket : default from 172.23.97.83 [2019-01-23 15:29:23,178] - [bucket_helper] [139642968725248] - INFO - waiting for bucket deletion to complete.... [2019-01-23 15:29:23,182] - [rest_client] [139642968725248] - INFO - node 172.23.97.83 existing buckets : [] [2019-01-23 15:29:23,182] - [bucket_helper] [139642968725248] - INFO - sleep 2 seconds to make sure all buckets ([u'default']) were deleted completely. [2019-01-23 15:29:25,201] - [cluster_helper] [139642968725248] - INFO - rebalancing all nodes in order to remove nodes [2019-01-23 15:29:25,204] - [rest_client] [139642968725248] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.97.84', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.97.84,ns_1@172.23.97.83'} [2019-01-23 15:29:25,208] - [rest_client] [139642968725248] - INFO - rebalance operation started [2019-01-23 15:29:25,212] - [rest_client] [139642968725248] - INFO - rebalance percentage : 0.00 % [2019-01-23 15:29:45,240] - [rest_client] [139642968725248] - INFO - rebalance progress took 20.03 seconds [2019-01-23 15:29:45,240] - [rest_client] [139642968725248] - INFO - sleep for 10 seconds after rebalance... [2019-01-23 15:29:55,260] - [cluster_helper] [139642968725248] - INFO - removed all the nodes from cluster associated with ip:172.23.97.83 port:8091 ssh_username:root ? [(u'ns_1@172.23.97.84', 8091)] [2019-01-23 15:29:55,264] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.83:8091 [2019-01-23 15:29:55,268] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.83:8091 is running [2019-01-23 15:29:55,290] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.84:8091 [2019-01-23 15:29:55,294] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.84:8091 is running [2019-01-23 15:29:55,294] - [xdcrnewbasetests] [139642968725248] - INFO - removing xdcr/nodes settings [2019-01-23 15:29:55,303] - [rest_client] [139642968725248] - INFO - Deleting replication /controller/cancelXDCR/f7f021251634dfe770fccec884116b2b%2Fdefault%2Fdefault [2019-01-23 15:29:55,311] - [rest_client] [139642968725248] - INFO - Replication deleted successfully [2019-01-23 15:29:55,321] - [rest_client] [139642968725248] - INFO - removing remote cluster name:remote_cluster_C2-C1 [2019-01-23 15:29:55,327] - [rest_client] [139642968725248] - INFO - removing remote cluster name:remote_cluster_C2-C3 [2019-01-23 15:29:55,345] - [xdcrnewbasetests] [139642968725248] - INFO - cleanup [ip:172.23.97.85 port:8091 ssh_username:root, ip:172.23.97.86 port:8091 ssh_username:root] [2019-01-23 15:29:55,359] - [bucket_helper] [139642968725248] - INFO - deleting existing buckets [u'default'] on 172.23.97.85 [2019-01-23 15:29:55,359] - [bucket_helper] [139642968725248] - INFO - remove bucket default ... [2019-01-23 15:29:56,798] - [bucket_helper] [139642968725248] - INFO - deleted bucket : default from 172.23.97.85 [2019-01-23 15:29:56,798] - [bucket_helper] [139642968725248] - INFO - waiting for bucket deletion to complete.... [2019-01-23 15:29:56,802] - [rest_client] [139642968725248] - INFO - node 172.23.97.85 existing buckets : [] [2019-01-23 15:29:56,802] - [bucket_helper] [139642968725248] - INFO - sleep 2 seconds to make sure all buckets ([u'default']) were deleted completely. [2019-01-23 15:29:58,821] - [cluster_helper] [139642968725248] - INFO - rebalancing all nodes in order to remove nodes [2019-01-23 15:29:58,824] - [rest_client] [139642968725248] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.97.86', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.97.85,ns_1@172.23.97.86'} [2019-01-23 15:29:58,829] - [rest_client] [139642968725248] - INFO - rebalance operation started [2019-01-23 15:29:58,834] - [rest_client] [139642968725248] - INFO - rebalance percentage : 0.00 % [2019-01-23 15:30:18,868] - [rest_client] [139642968725248] - INFO - rebalance progress took 20.04 seconds [2019-01-23 15:30:18,868] - [rest_client] [139642968725248] - INFO - sleep for 10 seconds after rebalance... [2019-01-23 15:30:28,890] - [cluster_helper] [139642968725248] - INFO - removed all the nodes from cluster associated with ip:172.23.97.85 port:8091 ssh_username:root ? [(u'ns_1@172.23.97.86', 8091)] [2019-01-23 15:30:28,895] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.85:8091 [2019-01-23 15:30:28,899] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.85:8091 is running [2019-01-23 15:30:28,922] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.86:8091 [2019-01-23 15:30:28,925] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.86:8091 is running [2019-01-23 15:30:28,925] - [xdcrnewbasetests] [139642968725248] - INFO - removing xdcr/nodes settings [2019-01-23 15:30:28,938] - [rest_client] [139642968725248] - INFO - removing remote cluster name:remote_cluster_C3-C1 [2019-01-23 15:30:28,945] - [rest_client] [139642968725248] - INFO - removing remote cluster name:remote_cluster_C3-C2 [2019-01-23 15:30:28,967] - [xdcrnewbasetests] [139642968725248] - INFO - cleanup [ip:172.23.97.87 port:8091 ssh_username:root, ip:172.23.97.88 port:8091 ssh_username:root] [2019-01-23 15:30:28,986] - [bucket_helper] [139642968725248] - INFO - deleting existing buckets [u'default'] on 172.23.97.87 [2019-01-23 15:30:28,988] - [bucket_helper] [139642968725248] - INFO - remove bucket default ... [2019-01-23 15:30:30,130] - [bucket_helper] [139642968725248] - INFO - deleted bucket : default from 172.23.97.87 [2019-01-23 15:30:30,131] - [bucket_helper] [139642968725248] - INFO - waiting for bucket deletion to complete.... [2019-01-23 15:30:30,135] - [rest_client] [139642968725248] - INFO - node 172.23.97.87 existing buckets : [] [2019-01-23 15:30:30,135] - [bucket_helper] [139642968725248] - INFO - sleep 2 seconds to make sure all buckets ([u'default']) were deleted completely. [2019-01-23 15:30:32,152] - [cluster_helper] [139642968725248] - INFO - rebalancing all nodes in order to remove nodes [2019-01-23 15:30:32,154] - [rest_client] [139642968725248] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.97.88', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.97.87,ns_1@172.23.97.88'} [2019-01-23 15:30:32,159] - [rest_client] [139642968725248] - INFO - rebalance operation started [2019-01-23 15:30:32,162] - [rest_client] [139642968725248] - INFO - rebalance percentage : 0.00 % [2019-01-23 15:30:52,190] - [rest_client] [139642968725248] - INFO - rebalance progress took 20.03 seconds [2019-01-23 15:30:52,190] - [rest_client] [139642968725248] - INFO - sleep for 10 seconds after rebalance... [2019-01-23 15:31:02,208] - [cluster_helper] [139642968725248] - INFO - removed all the nodes from cluster associated with ip:172.23.97.87 port:8091 ssh_username:root ? [(u'ns_1@172.23.97.88', 8091)] [2019-01-23 15:31:02,213] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.87:8091 [2019-01-23 15:31:02,217] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.87:8091 is running [2019-01-23 15:31:02,240] - [cluster_helper] [139642968725248] - INFO - waiting for ns_server @ 172.23.97.88:8091 [2019-01-23 15:31:02,244] - [cluster_helper] [139642968725248] - INFO - ns_server @ 172.23.97.88:8091 is running [2019-01-23 15:31:02,244] - [xdcrnewbasetests] [139642968725248] - INFO - ==== XDCRNewbasetests cleanup is finished for test #34 load_with_ops ===