[2019-04-26 20:52:31,032] - [rest_client] [140043376469760] - ERROR - POST http://172.23.104.201:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password [2019-04-26 20:52:31,034] - [rest_client] [140043376469760] - INFO - /diag/eval status on 172.23.104.201:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-04-26 20:52:31,037] - [remote_util] [140043376469760] - INFO - connecting to 172.23.104.201 with username:root [2019-04-26 20:52:31,205] - [remote_util] [140043376469760] - INFO - Connected to 172.23.104.201 [2019-04-26 20:52:31,657] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:32,115] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.201: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:32,267] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:32,332] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:32,338] - [remote_util] [140043376469760] - INFO - connecting to 172.23.104.201 with username:root [2019-04-26 20:52:32,506] - [remote_util] [140043376469760] - INFO - Connected to 172.23.104.201 [2019-04-26 20:52:32,962] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:33,416] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.201: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:33,568] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:33,633] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:33,637] - [remote_util] [140043376469760] - INFO - connecting to 172.23.104.202 with username:root [2019-04-26 20:52:33,805] - [remote_util] [140043376469760] - INFO - Connected to 172.23.104.202 [2019-04-26 20:52:34,251] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:34,693] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.202: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:34,793] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:34,908] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:34,911] - [remote_util] [140043376469760] - INFO - connecting to 172.23.104.202 with username:root [2019-04-26 20:52:35,046] - [remote_util] [140043376469760] - INFO - Connected to 172.23.104.202 [2019-04-26 20:52:35,487] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:35,929] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.202: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:36,080] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:36,144] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:36,149] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.44 with username:root [2019-04-26 20:52:36,285] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.44 [2019-04-26 20:52:36,588] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:37,026] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.44: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:37,159] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:37,273] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:37,276] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.44 with username:root [2019-04-26 20:52:37,412] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.44 [2019-04-26 20:52:37,700] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:38,156] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.44: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:38,287] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:38,402] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:38,405] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.45 with username:root [2019-04-26 20:52:38,541] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.45 [2019-04-26 20:52:38,871] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:39,387] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.45: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:39,519] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:39,635] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:39,640] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.45 with username:root [2019-04-26 20:52:39,777] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.45 [2019-04-26 20:52:40,077] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:40,581] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.45: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:40,714] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:40,829] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:40,833] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.46 with username:root [2019-04-26 20:52:40,969] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.46 [2019-04-26 20:52:41,298] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:41,829] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.46: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:41,928] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:42,043] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:42,047] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.46 with username:root [2019-04-26 20:52:42,182] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.46 [2019-04-26 20:52:42,498] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:43,017] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.46: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:43,149] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:43,263] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:43,267] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.47 with username:root [2019-04-26 20:52:43,403] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.47 [2019-04-26 20:52:43,700] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:44,211] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.47: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:44,344] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:44,459] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:44,463] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.47 with username:root [2019-04-26 20:52:44,599] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.47 [2019-04-26 20:52:44,883] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:45,324] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.47: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:45,455] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:45,570] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:45,571] - [xdcrnewbasetests] [140043376469760] - INFO - {'172.23.105.46': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.47': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.44': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.45': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.104.201': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.104.202': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}} [2019-04-26 20:52:45,591] - [rest_client] [140043376469760] - ERROR - POST http://172.23.104.201:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password [2019-04-26 20:52:45,592] - [rest_client] [140043376469760] - INFO - /diag/eval status on 172.23.104.201:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-04-26 20:52:45,595] - [remote_util] [140043376469760] - INFO - connecting to 172.23.104.201 with username:root [2019-04-26 20:52:45,761] - [remote_util] [140043376469760] - INFO - Connected to 172.23.104.201 [2019-04-26 20:52:46,230] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:46,721] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.201: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:46,873] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:46,989] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:46,993] - [remote_util] [140043376469760] - INFO - connecting to 172.23.104.202 with username:root [2019-04-26 20:52:47,161] - [remote_util] [140043376469760] - INFO - Connected to 172.23.104.202 [2019-04-26 20:52:47,634] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:48,067] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.202: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:48,219] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:48,284] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:48,289] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.44 with username:root [2019-04-26 20:52:48,425] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.44 [2019-04-26 20:52:48,724] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:49,227] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.44: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:49,359] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:49,475] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:49,477] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.45 with username:root [2019-04-26 20:52:49,613] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.45 [2019-04-26 20:52:49,934] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:50,370] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.45: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:50,501] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:50,616] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:50,619] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.46 with username:root [2019-04-26 20:52:50,755] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.46 [2019-04-26 20:52:51,054] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:51,560] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.46: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:51,692] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:51,807] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:51,811] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.47 with username:root [2019-04-26 20:52:51,947] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.47 [2019-04-26 20:52:52,237] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:52,680] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.47: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2019-04-26 20:52:52,813] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 20:52:52,928] - [xdcrnewbasetests] [140043376469760] - INFO - 0 [2019-04-26 20:52:52,929] - [xdcrnewbasetests] [140043376469760] - INFO - {'172.23.105.46': 0, '172.23.105.47': 0, '172.23.105.44': 0, '172.23.105.45': 0, '172.23.104.201': 0, '172.23.104.202': 0} [2019-04-26 20:52:52,930] - [xdcrnewbasetests] [140043376469760] - INFO - ==== XDCRNewbasetests setup is started for test #9 swap_rebalance ==== [2019-04-26 20:52:52,932] - [xdcrnewbasetests] [140043376469760] - INFO - removing xdcr/nodes settings [2019-04-26 20:52:53,004] - [xdcrnewbasetests] [140043376469760] - INFO - cleanup [ip:172.23.104.201 port:8091 ssh_username:root, ip:172.23.104.202 port:8091 ssh_username:root] [2019-04-26 20:52:53,077] - [cluster_helper] [140043376469760] - INFO - waiting for ns_server @ 172.23.104.201:8091 [2019-04-26 20:52:53,089] - [cluster_helper] [140043376469760] - INFO - ns_server @ 172.23.104.201:8091 is running [2019-04-26 20:52:53,167] - [cluster_helper] [140043376469760] - INFO - waiting for ns_server @ 172.23.104.202:8091 [2019-04-26 20:52:53,179] - [cluster_helper] [140043376469760] - INFO - ns_server @ 172.23.104.202:8091 is running [2019-04-26 20:52:53,179] - [xdcrnewbasetests] [140043376469760] - INFO - removing xdcr/nodes settings [2019-04-26 20:52:53,236] - [xdcrnewbasetests] [140043376469760] - INFO - cleanup [ip:172.23.105.44 port:8091 ssh_username:root, ip:172.23.105.45 port:8091 ssh_username:root] [2019-04-26 20:52:53,303] - [cluster_helper] [140043376469760] - INFO - waiting for ns_server @ 172.23.105.44:8091 [2019-04-26 20:52:53,313] - [cluster_helper] [140043376469760] - INFO - ns_server @ 172.23.105.44:8091 is running [2019-04-26 20:52:53,383] - [cluster_helper] [140043376469760] - INFO - waiting for ns_server @ 172.23.105.45:8091 [2019-04-26 20:52:53,394] - [cluster_helper] [140043376469760] - INFO - ns_server @ 172.23.105.45:8091 is running [2019-04-26 20:52:53,394] - [xdcrnewbasetests] [140043376469760] - INFO - Initializing all clusters... [2019-04-26 20:52:53,408] - [remote_util] [140043376469760] - INFO - connecting to 172.23.104.201 with username:root [2019-04-26 20:52:53,576] - [remote_util] [140043376469760] - INFO - Connected to 172.23.104.201 [2019-04-26 20:52:54,075] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:54,639] - [rest_client] [140043376469760] - ERROR - GET http://172.23.104.201:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-04-26 20:52:54,640] - [remote_util] [140043376469760] - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2019-04-26 20:52:54,641] - [xdcrnewbasetests] [140043376469760] - INFO - Running in compatibility mode, not enabled diag/eval for non-local hosts [2019-04-26 20:52:54,653] - [rest_client] [140043376469760] - ERROR - POST http://172.23.104.201:8091/diag/eval/ body: cluster_compat_mode:get_compat_version(). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password [2019-04-26 20:52:54,654] - [rest_client] [140043376469760] - INFO - /diag/eval status on 172.23.104.201:8091: False content: API is accessible from localhost only command: cluster_compat_mode:get_compat_version(). [2019-04-26 20:52:54,966] - [task] [140039254161152] - INFO - server: ip:172.23.104.201 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.201', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.104.201', 'uptime': u'8781', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4023, 'hostname': u'172.23.104.201:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-3084-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3593068544, 'memoryTotal': 4219432960, 'memoryQuota': 1748, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-04-26 20:52:54,983] - [rest_client] [140039254161152] - ERROR - GET http://172.23.104.201:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-04-26 20:52:54,985] - [rest_client] [140039254161152] - INFO - pools/default params : memoryQuota=2146 [2019-04-26 20:52:54,999] - [rest_client] [140039254161152] - INFO - settings/web params on 172.23.104.201:8091:username=Administrator&password=password&port=8091 [2019-04-26 20:52:55,119] - [remote_util] [140039254161152] - INFO - connecting to 172.23.104.201 with username:root [2019-04-26 20:52:55,288] - [remote_util] [140039254161152] - INFO - Connected to 172.23.104.201 [2019-04-26 20:52:55,748] - [remote_util] [140039254161152] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:56,245] - [remote_util] [140039254161152] - INFO - running command.raw on 172.23.104.201: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-04-26 20:52:56,317] - [remote_util] [140039254161152] - INFO - command executed successfully [2019-04-26 20:52:56,330] - [rest_client] [140039254161152] - INFO - /diag/eval status on 172.23.104.201:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-04-26 20:52:56,330] - [rest_client] [140039254161152] - INFO - settings/indexes params : storageMode=forestdb [2019-04-26 20:52:56,343] - [rest_client] [140039254161152] - ERROR - POST http://172.23.104.201:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2019-04-26 20:52:56,408] - [task] [140039254161152] - INFO - server: ip:172.23.104.202 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.104.202', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.104.202', 'uptime': u'8771', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4023, 'hostname': u'172.23.104.202:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-3084-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3507781632, 'memoryTotal': 4219445248, 'memoryQuota': 2146, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-04-26 20:52:56,426] - [rest_client] [140039254161152] - INFO - pools/default params : memoryQuota=2146 [2019-04-26 20:52:56,441] - [rest_client] [140039254161152] - INFO - settings/web params on 172.23.104.202:8091:username=Administrator&password=password&port=8091 [2019-04-26 20:52:56,552] - [remote_util] [140039254161152] - INFO - connecting to 172.23.104.202 with username:root [2019-04-26 20:52:56,720] - [remote_util] [140039254161152] - INFO - Connected to 172.23.104.202 [2019-04-26 20:52:57,167] - [remote_util] [140039254161152] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:52:57,658] - [remote_util] [140039254161152] - INFO - running command.raw on 172.23.104.202: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-04-26 20:52:57,728] - [remote_util] [140039254161152] - INFO - command executed successfully [2019-04-26 20:52:57,746] - [rest_client] [140039254161152] - INFO - /diag/eval status on 172.23.104.202:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-04-26 20:52:57,747] - [rest_client] [140039254161152] - INFO - settings/indexes params : storageMode=forestdb [2019-04-26 20:52:57,760] - [rest_client] [140039254161152] - ERROR - POST http://172.23.104.202:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2019-04-26 20:52:58,797] - [task] [140039254161152] - INFO - adding node 172.23.104.202:8091 to cluster [2019-04-26 20:52:58,797] - [rest_client] [140039254161152] - INFO - adding remote node @172.23.104.202:8091 to this cluster @172.23.104.201:8091 [2019-04-26 20:53:02,249] - [rest_client] [140039254161152] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.201,ns_1@172.23.104.202'} [2019-04-26 20:53:02,266] - [rest_client] [140039254161152] - INFO - rebalance operation started [2019-04-26 20:53:02,283] - [rest_client] [140039254161152] - INFO - rebalance percentage : 0.00 % [2019-04-26 20:53:02,283] - [task] [140039254161152] - INFO - Rebalance - status: running, progress: 0.00% [2019-04-26 20:53:12,327] - [task] [140039254161152] - INFO - Rebalance - status: none, progress: 100.00% [2019-04-26 20:53:12,345] - [task] [140039254161152] - INFO - rebalancing was completed with progress: 100% in 10.0783140659 sec [2019-04-26 20:53:12,365] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.44 with username:root [2019-04-26 20:53:12,502] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.44 [2019-04-26 20:53:12,803] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:53:13,280] - [rest_client] [140043376469760] - ERROR - GET http://172.23.105.44:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-04-26 20:53:13,281] - [remote_util] [140043376469760] - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2019-04-26 20:53:13,282] - [xdcrnewbasetests] [140043376469760] - INFO - Running in compatibility mode, not enabled diag/eval for non-local hosts [2019-04-26 20:53:13,293] - [rest_client] [140043376469760] - ERROR - POST http://172.23.105.44:8091/diag/eval/ body: cluster_compat_mode:get_compat_version(). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password [2019-04-26 20:53:13,297] - [rest_client] [140043376469760] - INFO - /diag/eval status on 172.23.105.44:8091: False content: API is accessible from localhost only command: cluster_compat_mode:get_compat_version(). [2019-04-26 20:53:13,986] - [task] [140040428582656] - INFO - server: ip:172.23.105.44 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.44', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.44', 'uptime': u'8855', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.44:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-3084-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3592626176, 'memoryTotal': 4201840640, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-04-26 20:53:14,000] - [rest_client] [140040428582656] - ERROR - GET http://172.23.105.44:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-04-26 20:53:14,002] - [rest_client] [140040428582656] - INFO - pools/default params : memoryQuota=2136 [2019-04-26 20:53:14,016] - [rest_client] [140040428582656] - INFO - settings/web params on 172.23.105.44:8091:username=Administrator&password=password&port=8091 [2019-04-26 20:53:14,081] - [remote_util] [140040428582656] - INFO - connecting to 172.23.105.44 with username:root [2019-04-26 20:53:14,217] - [remote_util] [140040428582656] - INFO - Connected to 172.23.105.44 [2019-04-26 20:53:14,508] - [remote_util] [140040428582656] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:53:14,979] - [remote_util] [140040428582656] - INFO - running command.raw on 172.23.105.44: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-04-26 20:53:15,022] - [remote_util] [140040428582656] - INFO - command executed successfully [2019-04-26 20:53:15,035] - [rest_client] [140040428582656] - INFO - /diag/eval status on 172.23.105.44:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-04-26 20:53:15,035] - [rest_client] [140040428582656] - INFO - settings/indexes params : storageMode=forestdb [2019-04-26 20:53:15,048] - [rest_client] [140040428582656] - ERROR - POST http://172.23.105.44:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2019-04-26 20:53:15,103] - [task] [140040428582656] - INFO - server: ip:172.23.105.45 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.45', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.45', 'uptime': u'8839', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.45:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-3084-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3472175104, 'memoryTotal': 4201840640, 'memoryQuota': 2136, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-04-26 20:53:15,119] - [rest_client] [140040428582656] - INFO - pools/default params : memoryQuota=2136 [2019-04-26 20:53:15,136] - [rest_client] [140040428582656] - INFO - settings/web params on 172.23.105.45:8091:username=Administrator&password=password&port=8091 [2019-04-26 20:53:15,191] - [remote_util] [140040428582656] - INFO - connecting to 172.23.105.45 with username:root [2019-04-26 20:53:15,327] - [remote_util] [140040428582656] - INFO - Connected to 172.23.105.45 [2019-04-26 20:53:15,684] - [remote_util] [140040428582656] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:53:16,144] - [remote_util] [140040428582656] - INFO - running command.raw on 172.23.105.45: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-04-26 20:53:16,188] - [remote_util] [140040428582656] - INFO - command executed successfully [2019-04-26 20:53:16,200] - [rest_client] [140040428582656] - INFO - /diag/eval status on 172.23.105.45:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-04-26 20:53:16,200] - [rest_client] [140040428582656] - INFO - settings/indexes params : storageMode=forestdb [2019-04-26 20:53:16,212] - [rest_client] [140040428582656] - ERROR - POST http://172.23.105.45:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}} auth: Administrator:password [2019-04-26 20:53:17,247] - [task] [140040428582656] - INFO - adding node 172.23.105.45:8091 to cluster [2019-04-26 20:53:17,248] - [rest_client] [140040428582656] - INFO - adding remote node @172.23.105.45:8091 to this cluster @172.23.105.44:8091 [2019-04-26 20:53:19,535] - [rest_client] [140040428582656] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.45,ns_1@172.23.105.44'} [2019-04-26 20:53:19,554] - [rest_client] [140040428582656] - INFO - rebalance operation started [2019-04-26 20:53:19,573] - [rest_client] [140040428582656] - INFO - rebalance percentage : 0.00 % [2019-04-26 20:53:19,574] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 0.00% [2019-04-26 20:53:29,614] - [task] [140040428582656] - INFO - Rebalance - status: none, progress: 100.00% [2019-04-26 20:53:29,630] - [task] [140040428582656] - INFO - rebalancing was completed with progress: 100% in 10.0746459961 sec [2019-04-26 20:53:29,697] - [rest_client] [140043376469760] - ERROR - DELETE http://172.23.104.201:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2019-04-26 20:53:29,699] - [internal_user] [140043376469760] - INFO - Exception while deleting user. Exception is -"User was not found." [2019-04-26 20:53:29,953] - [rest_client] [140043376469760] - ERROR - DELETE http://172.23.105.44:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2019-04-26 20:53:29,954] - [internal_user] [140043376469760] - INFO - Exception while deleting user. Exception is -"User was not found." [2019-04-26 20:53:30,436] - [rest_client] [140039254161152] - INFO - http://172.23.104.201:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1646&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2019-04-26 20:53:30,454] - [rest_client] [140039254161152] - INFO - 0.02 seconds to create bucket default [2019-04-26 20:53:30,455] - [bucket_helper] [140039254161152] - INFO - waiting for memcached bucket : default in 172.23.104.201 to accept set ops [2019-04-26 20:53:32,146] - [data_helper] [140039254161152] - INFO - creating direct client 172.23.104.201:11210 default [2019-04-26 20:53:32,344] - [data_helper] [140039254161152] - INFO - creating direct client 172.23.104.202:11210 default [2019-04-26 20:53:32,701] - [data_helper] [140039254161152] - INFO - creating direct client 172.23.104.201:11210 default [2019-04-26 20:53:32,932] - [data_helper] [140039254161152] - INFO - creating direct client 172.23.104.202:11210 default [2019-04-26 20:53:33,973] - [task] [140039254161152] - INFO - bucket 'default' was created with per node RAM quota: 1646 [2019-04-26 20:53:34,717] - [rest_client] [140040428582656] - INFO - http://172.23.105.44:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1636&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2019-04-26 20:53:34,733] - [rest_client] [140040428582656] - INFO - 0.02 seconds to create bucket default [2019-04-26 20:53:34,734] - [bucket_helper] [140040428582656] - INFO - waiting for memcached bucket : default in 172.23.105.44 to accept set ops [2019-04-26 20:53:36,042] - [data_helper] [140040428582656] - INFO - creating direct client 172.23.105.44:11210 default [2019-04-26 20:53:36,284] - [data_helper] [140040428582656] - INFO - creating direct client 172.23.105.45:11210 default [2019-04-26 20:53:36,575] - [data_helper] [140040428582656] - INFO - creating direct client 172.23.105.44:11210 default [2019-04-26 20:53:36,796] - [data_helper] [140040428582656] - INFO - creating direct client 172.23.105.45:11210 default [2019-04-26 20:53:37,518] - [task] [140040428582656] - INFO - bucket 'default' was created with per node RAM quota: 1636 [2019-04-26 20:53:37,548] - [xdcrnewbasetests] [140043376469760] - INFO - ==== XDCRNewbasetests setup is finished for test #9 swap_rebalance ==== [2019-04-26 20:53:37,589] - [rest_client] [140043376469760] - INFO - adding remote cluster hostname:172.23.105.44:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.104.201:8091 [2019-04-26 20:53:37,608] - [rest_client] [140043376469760] - INFO - Node versions in cluster [u'6.5.0-3084-enterprise', u'6.5.0-3084-enterprise'] [2019-04-26 20:53:37,634] - [rest_client] [140043376469760] - INFO - Node versions in cluster [u'6.5.0-3084-enterprise', u'6.5.0-3084-enterprise'] [2019-04-26 20:53:37,811] - [rest_client] [140043376469760] - INFO - adding remote cluster hostname:172.23.104.201:8091 with username:password Administrator:password name:remote_cluster_C2-C1 to source node: 172.23.105.44:8091 [2019-04-26 20:53:37,824] - [rest_client] [140043376469760] - INFO - Node versions in cluster [u'6.5.0-3084-enterprise', u'6.5.0-3084-enterprise'] [2019-04-26 20:53:37,863] - [rest_client] [140043376469760] - INFO - Node versions in cluster [u'6.5.0-3084-enterprise', u'6.5.0-3084-enterprise'] [2019-04-26 20:53:38,027] - [rest_client] [140043376469760] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {} [2019-04-26 20:53:38,212] - [rest_client] [140043376469760] - INFO - Replication created with id: 3a13f23fa6a924858b6b9a3cba825e6a/default/default [2019-04-26 20:53:48,239] - [rest_client] [140043376469760] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C2-C1 with settings {} [2019-04-26 20:53:48,506] - [rest_client] [140043376469760] - INFO - Replication created with id: f56fe116ac9a5b5af001a639970420ff/default/default [2019-04-26 20:53:58,577] - [rest_client] [140043376469760] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.104.201 [2019-04-26 20:53:58,626] - [rest_client] [140043376469760] - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.105.44 [2019-04-26 20:53:58,830] - [data_helper] [140043376469760] - INFO - creating direct client 172.23.104.201:11210 default [2019-04-26 20:53:58,988] - [data_helper] [140043376469760] - INFO - creating direct client 172.23.104.202:11210 default [2019-04-26 20:54:00,221] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.104.201:11210 default [2019-04-26 20:54:00,252] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.104.201:11210 default [2019-04-26 20:54:00,273] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.104.201:11210 default [2019-04-26 20:54:00,320] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.104.201:11210 default [2019-04-26 20:54:00,503] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.104.202:11210 default [2019-04-26 20:54:00,544] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.104.202:11210 default [2019-04-26 20:54:00,559] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.104.202:11210 default [2019-04-26 20:54:00,630] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.104.202:11210 default [2019-04-26 20:54:20,711] - [data_helper] [140043376469760] - INFO - creating direct client 172.23.105.44:11210 default [2019-04-26 20:54:20,891] - [data_helper] [140043376469760] - INFO - creating direct client 172.23.105.45:11210 default [2019-04-26 20:54:22,026] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.105.44:11210 default [2019-04-26 20:54:22,075] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.105.44:11210 default [2019-04-26 20:54:22,105] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.105.44:11210 default [2019-04-26 20:54:22,133] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.105.44:11210 default [2019-04-26 20:54:22,198] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.105.45:11210 default [2019-04-26 20:54:22,266] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.105.45:11210 default [2019-04-26 20:54:22,270] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.105.45:11210 default [2019-04-26 20:54:22,296] - [data_helper] [140038658574080] - INFO - creating direct client 172.23.105.45:11210 default [2019-04-26 20:54:40,276] - [xdcrnewbasetests] [140043376469760] - INFO - sleep for 10 secs. ... [2019-04-26 20:54:50,288] - [xdcrnewbasetests] [140043376469760] - INFO - Starting swap-rebalance [remove_node:172.23.105.45] -> [add_node:172.23.105.47] at C2 cluster 172.23.105.44 [2019-04-26 20:54:50,891] - [rest_client] [140040428582656] - INFO - Node versions in cluster [u'6.5.0-3084-enterprise', u'6.5.0-3084-enterprise'] [2019-04-26 20:54:50,960] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:54:50,961] - [task] [140040428582656] - INFO - adding node 172.23.105.47:8091 to cluster [2019-04-26 20:54:50,961] - [rest_client] [140040428582656] - INFO - adding remote node @172.23.105.47:8091 to this cluster @172.23.105.44:8091 [2019-04-26 20:54:52,833] - [rest_client] [140040428582656] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.105.45', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.45,ns_1@172.23.105.47,ns_1@172.23.105.44'} [2019-04-26 20:54:52,850] - [rest_client] [140040428582656] - INFO - rebalance operation started [2019-04-26 20:54:52,851] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:54:52,915] - [rest_client] [140040428582656] - INFO - rebalance percentage : 0.00 % [2019-04-26 20:54:52,916] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 0.00% [2019-04-26 20:55:02,943] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:55:03,018] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:55:03,018] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:55:13,051] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:55:13,111] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:55:13,111] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:55:23,140] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:55:23,312] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:55:23,313] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:55:33,348] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:55:33,420] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:55:33,421] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:55:43,453] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:55:43,546] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:55:43,547] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:55:53,586] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:55:53,664] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:55:53,664] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:56:03,694] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:56:03,769] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:56:03,770] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:56:13,805] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:56:13,863] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:56:13,864] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:56:23,893] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:56:23,979] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:56:23,979] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:56:34,012] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:56:34,182] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:56:34,183] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:56:44,214] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:56:44,302] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:56:44,302] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:56:54,337] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:56:54,412] - [rest_client] [140040428582656] - INFO - rebalance percentage : 20.31 % [2019-04-26 20:56:54,413] - [task] [140040428582656] - INFO - Rebalance - status: running, progress: 20.31% [2019-04-26 20:57:04,443] - [task] [140040428582656] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2019-04-26 20:57:04,503] - [rest_client] [140040428582656] - ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed [2019-04-26 20:57:04,638] - [rest_client] [140040428582656] - INFO - Latest logs from UI on 172.23.105.44: [2019-04-26 20:57:04,639] - [rest_client] [140040428582656] - ERROR - {u'node': u'ns_1@172.23.105.45', u'code': 0, u'text': u'Compactor for database `default` (pid [{type,database},\n {important,true},\n {name,<<"default">>},\n {fa,\n {#Fun,\n [<<"default">>,\n {config,\n {30,undefined},\n {30,undefined},\n undefined,false,false,\n {daemon_config,30,131072,20971520}},\n false,\n {[{type,bucket}]}]}}]) terminated unexpectedly: {{{badmatch,\n {error,\n timeout}},\n [{mc_client_binary,\n stats_recv,\n 4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 169}]},\n {mc_client_binary,\n stats,\n 4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 437}]},\n {ns_memcached,\n do_handle_call,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 459}]},\n {ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 228}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 247}]}]},\n {gen_server,\n call,\n [{\'ns_memcached-default\',\n \'ns_1@172.23.105.45\'},\n {raw_stats,\n <<"diskinfo">>,\n #Fun,\n {<<"0">>,\n <<"0">>}},\n 180000]}}', u'shortText': u'message', u'serverTime': u'2019-04-26T20:57:03.605Z', u'module': u'compaction_daemon', u'tstamp': 1556337423605, u'type': u'warning'} [2019-04-26 20:57:04,639] - [rest_client] [140040428582656] - ERROR - {u'node': u'ns_1@172.23.105.45', u'code': 0, u'text': u'Control connection to memcached on \'ns_1@172.23.105.45\' disconnected: {{badmatch,\n {error,\n timeout}},\n [{mc_client_binary,\n stats_recv,\n 4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 169}]},\n {mc_client_binary,\n stats,\n 4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 437}]},\n {ns_memcached,\n do_handle_call,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 459}]},\n {ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 228}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 247}]}]}', u'shortText': u'message', u'serverTime': u'2019-04-26T20:57:03.603Z', u'module': u'ns_memcached', u'tstamp': 1556337423603, u'type': u'info'} [2019-04-26 20:57:04,639] - [rest_client] [140040428582656] - ERROR - {u'node': u'ns_1@172.23.105.44', u'code': 0, u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.18686.21>,\n {{wait_seqno_persisted_failed,"default",388,\n 207,\n [{\'ns_1@172.23.105.47\',\n {\'EXIT\',\n {{{{case_clause,\n {error,\n {{{badmatch,\n {error,\n {{badmatch,{error,timeout}},\n [{mc_client_binary,\n cmd_vocal_recv,5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,155}]},\n {mc_client_binary,cmd_vocal,\n 3,\n [{file,\n "src/mc_client_binary.erl"},\n {line,140}]},\n {dcp_commands,\n open_connection,5,\n [{file,\n "src/dcp_commands.erl"},\n {line,79}]},\n {dcp_proxy,connect,5,\n [{file,"src/dcp_proxy.erl"},\n {line,247}]},\n {dcp_proxy,maybe_connect,2,\n [{file,"src/dcp_proxy.erl"},\n {line,221}]},\n {dcp_producer_conn,init,2,\n [{file,\n "src/dcp_producer_conn.erl"},\n {line,31}]},\n {dcp_proxy,init,1,\n [{file,"src/dcp_proxy.erl"},\n {line,59}]},\n {gen_server,init_it,2,\n [{file,"gen_server.erl"},\n {line,365}]}]}}},\n [{dcp_replicator,init,1,\n [{file,"src/dcp_replicator.erl"},\n {line,50}]},\n {gen_server,init_it,2,\n [{file,"gen_server.erl"},\n {line,365}]},\n {gen_server,init_it,6,\n [{file,"gen_server.erl"},\n {line,333}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},\n {line,247}]}]},\n {child,undefined,\n {\'ns_1@172.23.105.45\',\n [del_times,json,\n set_consumer_name,snappy,xattr]},\n {dcp_replicator,start_link,\n [\'ns_1@172.23.105.45\',"default",\n [del_times,json,\n set_consumer_name,snappy,\n xattr]]},\n temporary,60000,worker,\n [dcp_replicator]}}}},\n [{dcp_sup,start_replicator,2,\n [{file,"src/dcp_sup.erl"},\n {line,57}]},\n {dcp_sup,\n \'-manage_replicators/2-lc$^3/1-3-\',\n 2,\n [{file,"src/dcp_sup.erl"},\n {line,101}]},\n {dcp_replication_manager,\n handle_call,3,\n [{file,\n "src/dcp_replication_manager.erl"},\n {line,89}]},\n {gen_server,try_handle_call,4,\n [{file,"gen_server.erl"},\n {line,636}]},\n {gen_server,handle_msg,6,\n [{file,"gen_server.erl"},\n {line,665}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},\n {line,247}]}]},\n {gen_server,call,\n [\'dcp_replication_manager-default\',\n {get_replicator_pid,388},\n infinity]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.105.47\'},\n {if_rebalance,<0.32155.20>,\n {wait_seqno_persisted,388,207}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,510}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,232}]}]}}}}. Operation Id = c59f8a7dc2768067c6eb36c11a7dce56', u'shortText': u'message', u'serverTime': u'2019-04-26T20:57:01.038Z', u'module': u'ns_orchestrator', u'tstamp': 1556337421038, u'type': u'critical'} [2019-04-26 20:57:04,640] - [rest_client] [140040428582656] - ERROR - {u'node': u'ns_1@172.23.105.44', u'code': 0, u'text': u'<0.17973.21> exited with {unexpected_exit,\n {\'EXIT\',<0.18686.21>,\n {{wait_seqno_persisted_failed,"default",388,207,\n [{\'ns_1@172.23.105.47\',\n {\'EXIT\',\n {{{{case_clause,\n {error,\n {{{badmatch,\n {error,\n {{badmatch,{error,timeout}},\n [{mc_client_binary,cmd_vocal_recv,\n 5,\n [{file,"src/mc_client_binary.erl"},\n {line,155}]},\n {mc_client_binary,cmd_vocal,3,\n [{file,"src/mc_client_binary.erl"},\n {line,140}]},\n {dcp_commands,open_connection,5,\n [{file,"src/dcp_commands.erl"},\n {line,79}]},\n {dcp_proxy,connect,5,\n [{file,"src/dcp_proxy.erl"},\n {line,247}]},\n {dcp_proxy,maybe_connect,2,\n [{file,"src/dcp_proxy.erl"},\n {line,221}]},\n {dcp_producer_conn,init,2,\n [{file,\n "src/dcp_producer_conn.erl"},\n {line,31}]},\n {dcp_proxy,init,1,\n [{file,"src/dcp_proxy.erl"},\n {line,59}]},\n {gen_server,init_it,2,\n [{file,"gen_server.erl"},\n {line,365}]}]}}},\n [{dcp_replicator,init,1,\n [{file,"src/dcp_replicator.erl"},\n {line,50}]},\n {gen_server,init_it,2,\n [{file,"gen_server.erl"},{line,365}]},\n {gen_server,init_it,6,\n [{file,"gen_server.erl"},{line,333}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,247}]}]},\n {child,undefined,\n {\'ns_1@172.23.105.45\',\n [del_times,json,set_consumer_name,\n snappy,xattr]},\n {dcp_replicator,start_link,\n [\'ns_1@172.23.105.45\',"default",\n [del_times,json,set_consumer_name,\n snappy,xattr]]},\n temporary,60000,worker,\n [dcp_replicator]}}}},\n [{dcp_sup,start_replicator,2,\n [{file,"src/dcp_sup.erl"},{line,57}]},\n {dcp_sup,\n \'-manage_replicators/2-lc$^3/1-3-\',2,\n [{file,"src/dcp_sup.erl"},{line,101}]},\n {dcp_replication_manager,handle_call,3,\n [{file,"src/dcp_replication_manager.erl"},\n {line,89}]},\n {gen_server,try_handle_call,4,\n [{file,"gen_server.erl"},{line,636}]},\n {gen_server,handle_msg,6,\n [{file,"gen_server.erl"},{line,665}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,247}]}]},\n {gen_server,call,\n [\'dcp_replication_manager-default\',\n {get_replicator_pid,388},\n infinity]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.105.47\'},\n {if_rebalance,<0.32155.20>,\n {wait_seqno_persisted,388,207}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,510}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,232}]}]}}}', u'shortText': u'message', u'serverTime': u'2019-04-26T20:57:01.030Z', u'module': u'ns_vbucket_mover', u'tstamp': 1556337421030, u'type': u'critical'} [2019-04-26 20:57:04,640] - [rest_client] [140040428582656] - ERROR - {u'node': u'ns_1@172.23.105.47', u'code': 0, u'text': u"Service 'goxdcr' exited with status 137. Restarting. Messages:\n2019-04-26T20:54:32.774-07:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false backlogExist: false maxTP: 0 highTPNeeded: 0 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration: None dcpAction: Reset processCpu: 0 idleCpu: 99\n2019-04-26T20:54:32.774-07:00 INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=8919 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0\n2019-04-26T20:54:32.774-07:00 INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[]\nongoingReplMap=map[]\n2019-04-26T20:54:37.774-07:00 INFO GOXDCR.PipelineMgr: Replication Status = map[]\n2019-04-26T20:54:42.774-07:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false backlogExist: false maxTP: 0 highTPNeeded: 0 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration: None dcpAction: Reset processCpu: 0 idleCpu: 99\n2019-04-26T20:54:42.774-07:00 INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=8930 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0\n2019-04-26T20:54:42.774-07:00 INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[]\nongoingReplMap=map[]\n", u'shortText': u'message', u'serverTime': u'2019-04-26T20:54:57.997Z', u'module': u'ns_log', u'tstamp': 1556337297997, u'type': u'info'} [2019-04-26 20:57:04,640] - [rest_client] [140040428582656] - ERROR - {u'node': u'ns_1@172.23.105.47', u'code': 0, u'text': u'Service \'projector\' exited with status 137. Restarting. Messages:\n2019-04-26T20:51:49.422-07:00 [Info] memstats {"Alloc":2630680, "TotalAlloc":2630680, "Sys":72153336, "Lookups":0, "Mallocs":8504,"Frees":1311, "HeapAlloc":2630680, "HeapSys":66420736, "HeapIdle":62251008, "HeapInuse":4169728,"HeapReleased":0, "HeapObjects":7193,"GCSys":2234368, "LastGC":0,"PauseTotalNs":0, "PauseNs":[], "NumGC":0}\n2019-04-26T20:52:49.422-07:00 [Info] memstats {"Alloc":2631504, "TotalAlloc":2631504, "Sys":72153336, "Lookups":0, "Mallocs":8521,"Frees":1317, "HeapAlloc":2631504, "HeapSys":66420736, "HeapIdle":62251008, "HeapInuse":4169728,"HeapReleased":62251008, "HeapObjects":7204,"GCSys":2234368, "LastGC":0,"PauseTotalNs":0, "PauseNs":[], "NumGC":0}\n2019-04-26T20:53:49.422-07:00 [Info] memstats {"Alloc":2632344, "TotalAlloc":2632344, "Sys":72153336, "Lookups":0, "Mallocs":8539,"Frees":1323, "HeapAlloc":2632344, "HeapSys":66420736, "HeapIdle":62251008, "HeapInuse":4169728,"HeapReleased":62251008, "HeapObjects":7216,"GCSys":2234368, "LastGC":0,"PauseTotalNs":0, "PauseNs":[], "NumGC":0}\n2019-04-26T20:54:49.422-07:00 [Info] memstats {"Alloc":2633184, "TotalAlloc":2633184, "Sys":72153336, "Lookups":0, "Mallocs":8557,"Frees":1329, "HeapAlloc":2633184, "HeapSys":66420736, "HeapIdle":62251008, "HeapInuse":4169728,"HeapReleased":62251008, "HeapObjects":7228,"GCSys":2234368, "LastGC":0,"PauseTotalNs":0, "PauseNs":[], "NumGC":0}\n', u'shortText': u'message', u'serverTime': u'2019-04-26T20:54:57.994Z', u'module': u'ns_log', u'tstamp': 1556337297994, u'type': u'info'} [2019-04-26 20:57:04,641] - [rest_client] [140040428582656] - ERROR - {u'node': u'ns_1@172.23.105.44', u'code': 0, u'text': u'Bucket "default" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2019-04-26T20:54:54.837Z', u'module': u'ns_vbucket_mover', u'tstamp': 1556337294837, u'type': u'info'} [2019-04-26 20:57:04,641] - [rest_client] [140040428582656] - ERROR - {u'node': u'ns_1@172.23.105.47', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.105.47\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2019-04-26T20:54:53.495Z', u'module': u'ns_memcached', u'tstamp': 1556337293495, u'type': u'info'} [2019-04-26 20:57:04,641] - [rest_client] [140040428582656] - ERROR - {u'node': u'ns_1@172.23.105.44', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2019-04-26T20:54:53.359Z', u'module': u'ns_rebalancer', u'tstamp': 1556337293359, u'type': u'info'} [2019-04-26 20:57:04,642] - [rest_client] [140040428582656] - ERROR - {u'node': u'ns_1@172.23.105.44', u'code': 0, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.47','ns_1@172.23.105.44'], EjectNodes = ['ns_1@172.23.105.45'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = c59f8a7dc2768067c6eb36c11a7dce56", u'shortText': u'message', u'serverTime': u'2019-04-26T20:54:52.840Z', u'module': u'ns_orchestrator', u'tstamp': 1556337292840, u'type': u'info'} [2019-04-26 20:57:04,654] - [xdcrnewbasetests] [140043376469760] - INFO - Collecting logs @ 172.23.104.201 [2019-04-26 20:57:04,658] - [remote_util] [140043376469760] - INFO - connecting to 172.23.104.201 with username:root [2019-04-26 20:57:04,832] - [remote_util] [140043376469760] - INFO - Connected to 172.23.104.201 [2019-04-26 20:57:05,283] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 20:57:05,843] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.201: /opt/couchbase/bin/cbcollect_info 172.23.104.201-20190426-2057-diag.zip [2019-04-26 21:00:01,512] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:00:03,424] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.201: rm -f /root/172.23.104.201-20190426-2057-diag.zip [2019-04-26 21:00:03,526] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:00:03,647] - [remote_util] [140043376469760] - INFO - connecting to 172.23.104.201 with username:root [2019-04-26 21:00:03,816] - [remote_util] [140043376469760] - INFO - Connected to 172.23.104.201 [2019-04-26 21:00:04,350] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 21:00:04,875] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.201: tar -zcvf 172.23.104.201-4262019-210-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-04-26 21:00:11,074] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:00:12,844] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.201: rm -f /root/172.23.104.201-4262019-210-couch.tar.gz [2019-04-26 21:00:12,948] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:00:13,063] - [xdcrnewbasetests] [140043376469760] - INFO - Collecting logs @ 172.23.104.202 [2019-04-26 21:00:13,071] - [remote_util] [140043376469760] - INFO - connecting to 172.23.104.202 with username:root [2019-04-26 21:00:13,240] - [remote_util] [140043376469760] - INFO - Connected to 172.23.104.202 [2019-04-26 21:00:13,710] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 21:00:14,225] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.202: /opt/couchbase/bin/cbcollect_info 172.23.104.202-20190426-2100-diag.zip [2019-04-26 21:03:03,678] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:03:05,498] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.202: rm -f /root/172.23.104.202-20190426-2100-diag.zip [2019-04-26 21:03:05,599] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:03:05,722] - [remote_util] [140043376469760] - INFO - connecting to 172.23.104.202 with username:root [2019-04-26 21:03:05,890] - [remote_util] [140043376469760] - INFO - Connected to 172.23.104.202 [2019-04-26 21:03:06,328] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 21:03:06,843] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.202: tar -zcvf 172.23.104.202-4262019-213-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-04-26 21:03:12,084] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:03:14,060] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.104.202: rm -f /root/172.23.104.202-4262019-213-couch.tar.gz [2019-04-26 21:03:14,164] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:03:14,279] - [xdcrnewbasetests] [140043376469760] - INFO - Collecting logs @ 172.23.105.44 [2019-04-26 21:03:14,288] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.44 with username:root [2019-04-26 21:03:14,425] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.44 [2019-04-26 21:03:14,741] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 21:03:15,255] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.44: /opt/couchbase/bin/cbcollect_info 172.23.105.44-20190426-2103-diag.zip [2019-04-26 21:05:15,580] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:05:17,040] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.44: rm -f /root/172.23.105.44-20190426-2103-diag.zip [2019-04-26 21:05:17,141] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:05:17,260] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.44 with username:root [2019-04-26 21:05:17,395] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.44 [2019-04-26 21:05:17,721] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 21:05:18,287] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.44: tar -zcvf 172.23.105.44-4262019-215-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-04-26 21:05:20,314] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:05:21,376] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.44: rm -f /root/172.23.105.44-4262019-215-couch.tar.gz [2019-04-26 21:05:21,477] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:05:21,592] - [xdcrnewbasetests] [140043376469760] - INFO - Collecting logs @ 172.23.105.45 [2019-04-26 21:05:21,596] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.45 with username:root [2019-04-26 21:05:21,730] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.45 [2019-04-26 21:05:22,062] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 21:05:22,500] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.45: /opt/couchbase/bin/cbcollect_info 172.23.105.45-20190426-2105-diag.zip [2019-04-26 21:26:38,244] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:26:39,391] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.45: rm -f /root/172.23.105.45-20190426-2105-diag.zip [2019-04-26 21:26:39,479] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:26:39,602] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.45 with username:root [2019-04-26 21:26:39,737] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.45 [2019-04-26 21:26:40,052] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 21:26:40,517] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.45: tar -zcvf 172.23.105.45-4262019-2126-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-04-26 21:26:43,401] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:26:44,794] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.45: rm -f /root/172.23.105.45-4262019-2126-couch.tar.gz [2019-04-26 21:26:44,894] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:26:45,008] - [xdcrnewbasetests] [140043376469760] - INFO - Collecting logs @ 172.23.105.46 [2019-04-26 21:26:45,017] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.46 with username:root [2019-04-26 21:26:45,154] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.46 [2019-04-26 21:26:45,498] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 21:26:45,936] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.46: /opt/couchbase/bin/cbcollect_info 172.23.105.46-20190426-2126-diag.zip [2019-04-26 21:28:07,426] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:28:07,758] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.46: rm -f /root/172.23.105.46-20190426-2126-diag.zip [2019-04-26 21:28:07,842] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:28:07,963] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.46 with username:root [2019-04-26 21:28:08,099] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.46 [2019-04-26 21:28:08,433] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 21:28:08,898] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.46: tar -zcvf 172.23.105.46-4262019-2128-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-04-26 21:28:09,164] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:28:09,329] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.46: rm -f /root/172.23.105.46-4262019-2128-couch.tar.gz [2019-04-26 21:28:09,414] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:28:09,530] - [xdcrnewbasetests] [140043376469760] - INFO - Collecting logs @ 172.23.105.47 [2019-04-26 21:28:09,534] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.47 with username:root [2019-04-26 21:28:09,670] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.47 [2019-04-26 21:28:09,994] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 21:28:10,502] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.47: /opt/couchbase/bin/cbcollect_info 172.23.105.47-20190426-2128-diag.zip [2019-04-26 21:29:45,890] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:29:46,270] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.47: rm -f /root/172.23.105.47-20190426-2128-diag.zip [2019-04-26 21:29:46,292] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:29:46,410] - [remote_util] [140043376469760] - INFO - connecting to 172.23.105.47 with username:root [2019-04-26 21:29:46,545] - [remote_util] [140043376469760] - INFO - Connected to 172.23.105.47 [2019-04-26 21:29:46,878] - [remote_util] [140043376469760] - INFO - It could be other operating system. Go to check at other location [2019-04-26 21:29:47,337] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.47: tar -zcvf 172.23.105.47-4262019-2129-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2019-04-26 21:29:47,826] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:29:48,113] - [remote_util] [140043376469760] - INFO - running command.raw on 172.23.105.47: rm -f /root/172.23.105.47-4262019-2129-couch.tar.gz [2019-04-26 21:29:48,197] - [remote_util] [140043376469760] - INFO - command executed successfully [2019-04-26 21:29:48,394] - [rest_client] [140043376469760] - INFO - Node 172.23.105.45 not part of cluster inactiveFailed [2019-04-26 21:29:48,408] - [xdcrnewbasetests] [140043376469760] - INFO - ==== XDCRNewbasetests cleanup is started for test #9 swap_rebalance ==== [2019-04-26 21:29:48,408] - [xdcrnewbasetests] [140043376469760] - INFO - removing xdcr/nodes settings [2019-04-26 21:29:48,446] - [rest_client] [140043376469760] - INFO - Deleting replication /controller/cancelXDCR/3a13f23fa6a924858b6b9a3cba825e6a%2Fdefault%2Fdefault [2019-04-26 21:29:48,485] - [rest_client] [140043376469760] - INFO - Replication deleted successfully [2019-04-26 21:29:48,505] - [rest_client] [140043376469760] - INFO - removing remote cluster name:remote_cluster_C1-C2 [2019-04-26 21:29:48,574] - [xdcrnewbasetests] [140043376469760] - INFO - cleanup [ip:172.23.104.201 port:8091 ssh_username:root, ip:172.23.104.202 port:8091 ssh_username:root] [2019-04-26 21:29:48,624] - [bucket_helper] [140043376469760] - INFO - deleting existing buckets [u'default'] on 172.23.104.201 [2019-04-26 21:29:48,624] - [bucket_helper] [140043376469760] - INFO - remove bucket default ... [2019-04-26 21:29:51,117] - [bucket_helper] [140043376469760] - INFO - deleted bucket : default from 172.23.104.201 [2019-04-26 21:29:51,118] - [bucket_helper] [140043376469760] - INFO - waiting for bucket deletion to complete.... [2019-04-26 21:29:51,136] - [rest_client] [140043376469760] - INFO - node 172.23.104.201 existing buckets : [] [2019-04-26 21:29:51,136] - [bucket_helper] [140043376469760] - INFO - sleep 2 seconds to make sure all buckets ([u'default']) were deleted completely. [2019-04-26 21:29:53,198] - [cluster_helper] [140043376469760] - INFO - rebalancing all nodes in order to remove nodes [2019-04-26 21:29:53,210] - [rest_client] [140043376469760] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.104.202', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.201,ns_1@172.23.104.202'} [2019-04-26 21:29:53,224] - [rest_client] [140043376469760] - INFO - rebalance operation started [2019-04-26 21:29:53,237] - [rest_client] [140043376469760] - INFO - rebalance percentage : 0.00 % [2019-04-26 21:30:13,281] - [rest_client] [140043376469760] - INFO - rebalance progress took 20.06 seconds [2019-04-26 21:30:13,281] - [rest_client] [140043376469760] - INFO - sleep for 10 seconds after rebalance... [2019-04-26 21:30:23,321] - [cluster_helper] [140043376469760] - INFO - removed all the nodes from cluster associated with ip:172.23.104.201 port:8091 ssh_username:root ? [(u'ns_1@172.23.104.202', 8091)] [2019-04-26 21:30:23,341] - [cluster_helper] [140043376469760] - INFO - waiting for ns_server @ 172.23.104.201:8091 [2019-04-26 21:30:23,359] - [cluster_helper] [140043376469760] - INFO - ns_server @ 172.23.104.201:8091 is running [2019-04-26 21:30:23,447] - [cluster_helper] [140043376469760] - INFO - waiting for ns_server @ 172.23.104.202:8091 [2019-04-26 21:30:23,463] - [cluster_helper] [140043376469760] - INFO - ns_server @ 172.23.104.202:8091 is running [2019-04-26 21:30:23,463] - [xdcrnewbasetests] [140043376469760] - INFO - removing xdcr/nodes settings [2019-04-26 21:30:23,502] - [rest_client] [140043376469760] - INFO - removing remote cluster name:remote_cluster_C2-C1 [2019-04-26 21:30:23,563] - [xdcrnewbasetests] [140043376469760] - INFO - cleanup [ip:172.23.105.44 port:8091 ssh_username:root, ip:172.23.105.47 port:8091 ssh_username:root] [2019-04-26 21:30:23,609] - [bucket_helper] [140043376469760] - INFO - deleting existing buckets [u'default'] on 172.23.105.44 [2019-04-26 21:30:23,610] - [bucket_helper] [140043376469760] - INFO - remove bucket default ... [2019-04-26 21:30:25,036] - [bucket_helper] [140043376469760] - INFO - deleted bucket : default from 172.23.105.44 [2019-04-26 21:30:25,037] - [bucket_helper] [140043376469760] - INFO - waiting for bucket deletion to complete.... [2019-04-26 21:30:25,050] - [rest_client] [140043376469760] - INFO - node 172.23.105.44 existing buckets : [] [2019-04-26 21:30:25,051] - [bucket_helper] [140043376469760] - INFO - sleep 2 seconds to make sure all buckets ([u'default']) were deleted completely. [2019-04-26 21:30:27,109] - [cluster_helper] [140043376469760] - INFO - rebalancing all nodes in order to remove nodes [2019-04-26 21:30:27,120] - [rest_client] [140043376469760] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.105.45,ns_1@172.23.105.47', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.45,ns_1@172.23.105.47,ns_1@172.23.105.44'} [2019-04-26 21:30:27,133] - [rest_client] [140043376469760] - INFO - rebalance operation started [2019-04-26 21:30:27,149] - [rest_client] [140043376469760] - INFO - rebalance percentage : 0.00 % [2019-04-26 21:30:47,194] - [rest_client] [140043376469760] - INFO - rebalance progress took 20.06 seconds [2019-04-26 21:30:47,195] - [rest_client] [140043376469760] - INFO - sleep for 10 seconds after rebalance... [2019-04-26 21:30:57,257] - [cluster_helper] [140043376469760] - INFO - removed all the nodes from cluster associated with ip:172.23.105.44 port:8091 ssh_username:root ? [(u'ns_1@172.23.105.45', 8091), (u'ns_1@172.23.105.47', 8091)] [2019-04-26 21:30:57,271] - [cluster_helper] [140043376469760] - INFO - waiting for ns_server @ 172.23.105.44:8091 [2019-04-26 21:30:57,285] - [cluster_helper] [140043376469760] - INFO - ns_server @ 172.23.105.44:8091 is running [2019-04-26 21:30:57,369] - [cluster_helper] [140043376469760] - INFO - waiting for ns_server @ 172.23.105.47:8091 [2019-04-26 21:30:57,380] - [cluster_helper] [140043376469760] - INFO - ns_server @ 172.23.105.47:8091 is running [2019-04-26 21:30:57,381] - [xdcrnewbasetests] [140043376469760] - INFO - ==== XDCRNewbasetests cleanup is finished for test #9 swap_rebalance ===