./testrunner -i /tmp/testexec.8166.ini -p stop-on-failure=False,fail_on_errors=1,get-cbcollect-info=False,get-cbcollect-info=True -t xdcr.capiXDCR.Capi.test_crud_ops_from_cb_to_es,replication_type=capi,update=C1,delete=C1,GROUP=P2 Test Input params: {'replication_type': 'capi', 'update': 'C1', 'delete': 'C1', 'GROUP': 'P2', 'ini': '/tmp/testexec.8166.ini', 'cluster_name': 'testexec.8166', 'spec': 'py-xdcr-capi', 'conf_file': 'py-xdcr-capi.conf', 'stop-on-failure': 'False', 'fail_on_errors': '1', 'get-cbcollect-info': 'True', 'num_nodes': 5, 'case_number': 1, 'logs_folder': '/data/workspace/centos-p0-xdcr-vset00-00-capi/logs/testrunner-20-Jul-16_15-34-39/test_1'} Run before suite setup for xdcr.capiXDCR.Capi.test_crud_ops_from_cb_to_es lib/couchbase_helper/data_analysis_helper.py:796: DeprecationWarning: invalid escape sequence \{ master_key = ".\{12\}$" lib/couchbase_helper/tuq_generators.py:113: DeprecationWarning: invalid escape sequence \w regex = re.compile("[\w']+\.[\w']+") lib/couchbase_helper/tuq_generators.py:363: DeprecationWarning: invalid escape sequence \[ diff = set(order_clause.split(',')) - set(re.compile('doc\["[\w\']+"\]').findall(select_clause)) /data/workspace/centos-p0-xdcr-vset00-00-capi/scripts/install.py:1126: DeprecationWarning: invalid escape sequence \d if re.match('\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}', server.ip): pytests/security/SecretsMasterBase.py:63: DeprecationWarning: invalid escape sequence \. status, ipAddress = subprocess.getstatusoutput("ifconfig eth0 | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | awk '{print $2}'") suite_setUp (xdcr.capiXDCR.Capi) ... -->before_suite_name:xdcr.capiXDCR.Capi.suite_setUp,suite: ]> 2020-07-16 15:34:40,377 - root - ERROR - POST http://172.23.123.103:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2020-07-16 15:34:40,389 - root - INFO - /diag/eval status on 172.23.123.103:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2020-07-16 15:34:40,391 - root - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 2020-07-16 15:34:40,498 - root - INFO - SSH Connected to 172.23.123.103 as root 2020-07-16 15:34:40,786 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:41,101 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:41,102 - root - INFO - running command.raw on 172.23.123.103: zgrep "panic:" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:41,191 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:41,191 - root - INFO - 0 2020-07-16 15:34:41,193 - root - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 2020-07-16 15:34:41,295 - root - INFO - SSH Connected to 172.23.123.103 as root 2020-07-16 15:34:41,580 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:41,887 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:41,887 - root - INFO - running command.raw on 172.23.123.103: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:41,975 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:41,976 - root - INFO - 0 2020-07-16 15:34:41,978 - root - INFO - SSH Connecting to 172.23.123.101 with username:root, attempt#1 of 5 2020-07-16 15:34:42,081 - root - INFO - SSH Connected to 172.23.123.101 as root 2020-07-16 15:34:42,357 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:42,669 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:42,671 - root - INFO - running command.raw on 172.23.123.101: zgrep "panic:" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:42,757 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:42,758 - root - INFO - 0 2020-07-16 15:34:42,760 - root - INFO - SSH Connecting to 172.23.123.101 with username:root, attempt#1 of 5 2020-07-16 15:34:42,859 - root - INFO - SSH Connected to 172.23.123.101 as root 2020-07-16 15:34:43,130 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:43,442 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:43,443 - root - INFO - running command.raw on 172.23.123.101: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:43,529 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:43,530 - root - INFO - 0 2020-07-16 15:34:43,532 - root - INFO - SSH Connecting to 172.23.107.135 with username:root, attempt#1 of 5 2020-07-16 15:34:43,710 - root - INFO - SSH Connected to 172.23.107.135 as root 2020-07-16 15:34:44,252 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:44,687 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:44,688 - root - INFO - running command.raw on 172.23.107.135: zgrep "panic:" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:44,838 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:44,839 - root - INFO - 0 2020-07-16 15:34:44,841 - root - INFO - SSH Connecting to 172.23.107.135 with username:root, attempt#1 of 5 2020-07-16 15:34:45,023 - root - INFO - SSH Connected to 172.23.107.135 as root 2020-07-16 15:34:45,542 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:45,977 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:45,978 - root - INFO - running command.raw on 172.23.107.135: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:46,120 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:46,120 - root - INFO - 0 2020-07-16 15:34:46,122 - root - INFO - SSH Connecting to 172.23.123.102 with username:root, attempt#1 of 5 2020-07-16 15:34:46,224 - root - INFO - SSH Connected to 172.23.123.102 as root 2020-07-16 15:34:46,492 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:46,805 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:46,807 - root - INFO - running command.raw on 172.23.123.102: zgrep "panic:" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:46,893 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:46,893 - root - INFO - 0 2020-07-16 15:34:46,895 - root - INFO - SSH Connecting to 172.23.123.102 with username:root, attempt#1 of 5 2020-07-16 15:34:46,996 - root - INFO - SSH Connected to 172.23.123.102 as root 2020-07-16 15:34:47,261 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:47,615 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:47,616 - root - INFO - running command.raw on 172.23.123.102: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:47,702 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:47,703 - root - INFO - 0 2020-07-16 15:34:47,706 - root - INFO - SSH Connecting to 172.23.123.100 with username:root, attempt#1 of 5 2020-07-16 15:34:47,808 - root - INFO - SSH Connected to 172.23.123.100 as root 2020-07-16 15:34:48,100 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:48,409 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:48,410 - root - INFO - running command.raw on 172.23.123.100: zgrep "panic:" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:48,496 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:48,497 - root - INFO - 0 2020-07-16 15:34:48,499 - root - INFO - SSH Connecting to 172.23.123.100 with username:root, attempt#1 of 5 2020-07-16 15:34:48,599 - root - INFO - SSH Connected to 172.23.123.100 as root 2020-07-16 15:34:48,882 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:49,193 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:49,194 - root - INFO - running command.raw on 172.23.123.100: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:49,280 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:49,281 - root - INFO - 0 2020-07-16 15:34:49,281 - root - INFO - {'172.23.123.103': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.123.101': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.107.135': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.123.102': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.123.100': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}} 2020-07-16 15:34:49,288 - root - ERROR - POST http://172.23.123.103:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2020-07-16 15:34:49,289 - root - INFO - /diag/eval status on 172.23.123.103:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2020-07-16 15:34:49,290 - root - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 2020-07-16 15:34:49,396 - root - INFO - SSH Connected to 172.23.123.103 as root 2020-07-16 15:34:49,676 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:49,987 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:49,988 - root - INFO - running command.raw on 172.23.123.103: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:50,076 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:50,077 - root - INFO - 0 2020-07-16 15:34:50,079 - root - INFO - SSH Connecting to 172.23.123.101 with username:root, attempt#1 of 5 2020-07-16 15:34:50,180 - root - INFO - SSH Connected to 172.23.123.101 as root 2020-07-16 15:34:50,450 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:50,759 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:50,760 - root - INFO - running command.raw on 172.23.123.101: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:50,845 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:50,845 - root - INFO - 0 2020-07-16 15:34:50,847 - root - INFO - SSH Connecting to 172.23.107.135 with username:root, attempt#1 of 5 2020-07-16 15:34:51,022 - root - INFO - SSH Connected to 172.23.107.135 as root 2020-07-16 15:34:51,558 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:51,996 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:51,997 - root - INFO - running command.raw on 172.23.107.135: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:52,137 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:52,138 - root - INFO - 0 2020-07-16 15:34:52,140 - root - INFO - SSH Connecting to 172.23.123.102 with username:root, attempt#1 of 5 2020-07-16 15:34:52,241 - root - INFO - SSH Connected to 172.23.123.102 as root 2020-07-16 15:34:52,508 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:52,814 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:52,815 - root - INFO - running command.raw on 172.23.123.102: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:52,899 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:52,900 - root - INFO - 0 2020-07-16 15:34:52,902 - root - INFO - SSH Connecting to 172.23.123.100 with username:root, attempt#1 of 5 2020-07-16 15:34:53,003 - root - INFO - SSH Connected to 172.23.123.100 as root 2020-07-16 15:34:53,292 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:53,603 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:53,603 - root - INFO - running command.raw on 172.23.123.100: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* 2020-07-16 15:34:53,688 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2020-07-16 15:34:53,690 - root - INFO - 0 2020-07-16 15:34:53,690 - root - INFO - {'172.23.123.103': 0, '172.23.123.101': 0, '172.23.107.135': 0, '172.23.123.102': 0, '172.23.123.100': 0} 2020-07-16 15:34:53,690 - root - INFO - ==== XDCRNewbasetests setup is started for test #1 suite_setUp ==== 2020-07-16 15:34:53,692 - root - INFO - removing xdcr/nodes settings 2020-07-16 15:34:53,715 - root - INFO - cleanup [ip:172.23.123.103 port:8091 ssh_username:root, ip:172.23.123.101 port:8091 ssh_username:root] 2020-07-16 15:34:53,720 - root - INFO - Could not find any buckets for node 172.23.123.103, nothing to delete 2020-07-16 15:34:53,724 - root - INFO - -->is_ns_server_running? 2020-07-16 15:34:53,739 - root - INFO - waiting for ns_server @ 172.23.123.103:8091 2020-07-16 15:34:53,739 - root - INFO - -->is_ns_server_running? 2020-07-16 15:34:53,742 - root - INFO - ns_server @ 172.23.123.103:8091 is running 2020-07-16 15:34:53,749 - root - INFO - Could not find any buckets for node 172.23.123.101, nothing to delete 2020-07-16 15:34:53,752 - root - INFO - -->is_ns_server_running? 2020-07-16 15:34:53,768 - root - INFO - waiting for ns_server @ 172.23.123.101:8091 2020-07-16 15:34:53,768 - root - INFO - -->is_ns_server_running? 2020-07-16 15:34:53,771 - root - INFO - ns_server @ 172.23.123.101:8091 is running 2020-07-16 15:34:53,772 - root - INFO - removing xdcr/nodes settings 2020-07-16 15:34:53,821 - root - INFO - cleanup [ip:172.23.107.135 port:8091 ssh_username:root, ip:172.23.123.102 port:8091 ssh_username:root] 2020-07-16 15:34:53,831 - root - INFO - Could not find any buckets for node 172.23.107.135, nothing to delete 2020-07-16 15:34:53,837 - root - INFO - -->is_ns_server_running? 2020-07-16 15:34:53,872 - root - INFO - waiting for ns_server @ 172.23.107.135:8091 2020-07-16 15:34:53,873 - root - INFO - -->is_ns_server_running? 2020-07-16 15:34:53,879 - root - INFO - ns_server @ 172.23.107.135:8091 is running 2020-07-16 15:34:53,886 - root - INFO - Could not find any buckets for node 172.23.123.102, nothing to delete 2020-07-16 15:34:53,890 - root - INFO - -->is_ns_server_running? 2020-07-16 15:34:53,908 - root - INFO - waiting for ns_server @ 172.23.123.102:8091 2020-07-16 15:34:53,909 - root - INFO - -->is_ns_server_running? 2020-07-16 15:34:53,913 - root - INFO - ns_server @ 172.23.123.102:8091 is running 2020-07-16 15:34:53,913 - root - INFO - Initializing all clusters... 2020-07-16 15:34:53,919 - root - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 2020-07-16 15:34:54,026 - root - INFO - SSH Connected to 172.23.123.103 as root 2020-07-16 15:34:54,304 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:54,607 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:54,618 - root - INFO - running command.raw on 172.23.123.103: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:34:54,676 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 223 4573 --:--:-- --:--:-- --:--:-- 5125'] ... 2020-07-16 15:34:54,677 - root - INFO - ['ok'] 2020-07-16 15:34:54,677 - root - INFO - Enabled diag/eval for non-local hosts from 172.23.123.103 2020-07-16 15:34:54,680 - root - INFO - /diag/eval status on 172.23.123.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:34:54,696 - root - INFO - server: ip:172.23.123.103 port:8091 ssh_username:root, nodes/self 2020-07-16 15:34:54,700 - root - INFO - {'uptime': '183', 'memoryTotal': 4201672704, 'memoryFree': 3624046592, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.103:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-2595-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.103', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2020-07-16 15:34:54,705 - root - INFO - pools/default params : memoryQuota=2147 2020-07-16 15:34:54,708 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-16 15:34:54,709 - root - INFO - settings/web params on 172.23.123.103:8091:port=8091&username=Administrator&password=password 2020-07-16 15:34:54,743 - root - INFO - --> status:True 2020-07-16 15:34:54,745 - root - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 2020-07-16 15:34:54,849 - root - INFO - SSH Connected to 172.23.123.103 as root 2020-07-16 15:34:55,131 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:55,444 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:55,453 - root - INFO - running command.raw on 172.23.123.103: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:34:55,510 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 288 5912 --:--:-- --:--:-- --:--:-- 6833'] ... 2020-07-16 15:34:55,511 - root - INFO - ['ok'] 2020-07-16 15:34:55,515 - root - INFO - /diag/eval status on 172.23.123.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:34:55,515 - root - INFO - settings/indexes params : storageMode=forestdb 2020-07-16 15:34:55,520 - root - ERROR - POST http://172.23.123.103:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:password 2020-07-16 15:34:55,533 - root - INFO - server: ip:172.23.123.101 port:8091 ssh_username:root, nodes/self 2020-07-16 15:34:55,536 - root - INFO - {'uptime': '183', 'memoryTotal': 4201672704, 'memoryFree': 3591946240, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-2595-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2020-07-16 15:34:55,540 - root - INFO - pools/default params : memoryQuota=2147 2020-07-16 15:34:55,544 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-16 15:34:55,544 - root - INFO - settings/web params on 172.23.123.101:8091:port=8091&username=Administrator&password=password 2020-07-16 15:34:55,578 - root - INFO - --> status:True 2020-07-16 15:34:55,580 - root - INFO - SSH Connecting to 172.23.123.101 with username:root, attempt#1 of 5 2020-07-16 15:34:55,681 - root - INFO - SSH Connected to 172.23.123.101 as root 2020-07-16 15:34:55,951 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:34:56,258 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:34:56,268 - root - INFO - running command.raw on 172.23.123.101: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:34:56,326 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 282 5784 --:--:-- --:--:-- --:--:-- 6833'] ... 2020-07-16 15:34:56,327 - root - INFO - ['ok'] 2020-07-16 15:34:56,330 - root - INFO - /diag/eval status on 172.23.123.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:34:56,330 - root - INFO - settings/indexes params : storageMode=forestdb 2020-07-16 15:34:56,335 - root - ERROR - POST http://172.23.123.101:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:password 2020-07-16 15:34:57,343 - root - INFO - adding node 172.23.123.101:8091 to cluster 2020-07-16 15:34:57,343 - root - INFO - adding remote node @172.23.123.101:8091 to this cluster @172.23.123.103:8091 2020-07-16 15:34:59,498 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.101,ns_1@172.23.123.103', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2020-07-16 15:34:59,504 - root - INFO - rebalance operation started 2020-07-16 15:34:59,508 - root - INFO - rebalance percentage : 0.00 % 2020-07-16 15:34:59,508 - root - INFO - Rebalance - status: running, progress: 0.00% 2020-07-16 15:35:09,528 - root - INFO - Rebalance - status: none, progress: 100.00% 2020-07-16 15:35:09,534 - root - INFO - rebalancing was completed with progress: 100% in 10.029352188110352 sec 2020-07-16 15:35:09,543 - root - INFO - SSH Connecting to 172.23.107.135 with username:root, attempt#1 of 5 2020-07-16 15:35:09,722 - root - INFO - SSH Connected to 172.23.107.135 as root 2020-07-16 15:35:10,265 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:35:10,702 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:35:10,718 - root - INFO - running command.raw on 172.23.107.135: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:35:10,826 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 318 6533 --:--:-- --:--:-- --:--:-- 8200'] ... 2020-07-16 15:35:10,827 - root - INFO - ['ok'] 2020-07-16 15:35:10,827 - root - INFO - Enabled diag/eval for non-local hosts from 172.23.107.135 2020-07-16 15:35:10,831 - root - INFO - /diag/eval status on 172.23.107.135:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:35:11,719 - root - INFO - server: ip:172.23.107.135 port:8091 ssh_username:root, nodes/self 2020-07-16 15:35:11,724 - root - INFO - {'uptime': '130', 'memoryTotal': 4219432960, 'memoryFree': 1219932160, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.107.135:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-2595-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2156, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.135', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2020-07-16 15:35:11,733 - root - INFO - pools/default params : memoryQuota=2156 2020-07-16 15:35:11,740 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-16 15:35:11,740 - root - INFO - settings/web params on 172.23.107.135:8091:port=8091&username=Administrator&password=password 2020-07-16 15:35:11,850 - root - INFO - --> status:True 2020-07-16 15:35:11,853 - root - INFO - SSH Connecting to 172.23.107.135 with username:root, attempt#1 of 5 2020-07-16 15:35:12,028 - root - INFO - SSH Connected to 172.23.107.135 as root 2020-07-16 15:35:12,547 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:35:12,986 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:35:13,000 - root - INFO - running command.raw on 172.23.107.135: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:35:13,073 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 338 6949 --:--:-- --:--:-- --:--:-- 8200'] ... 2020-07-16 15:35:13,074 - root - INFO - ['ok'] 2020-07-16 15:35:13,080 - root - INFO - /diag/eval status on 172.23.107.135:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:35:13,080 - root - INFO - settings/indexes params : storageMode=forestdb 2020-07-16 15:35:13,094 - root - ERROR - POST http://172.23.107.135:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:password 2020-07-16 15:35:13,112 - root - INFO - server: ip:172.23.123.102 port:8091 ssh_username:root, nodes/self 2020-07-16 15:35:13,115 - root - INFO - {'uptime': '203', 'memoryTotal': 4201672704, 'memoryFree': 3615469568, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.102:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-2595-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.102', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2020-07-16 15:35:13,119 - root - INFO - pools/default params : memoryQuota=2147 2020-07-16 15:35:13,123 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-16 15:35:13,123 - root - INFO - settings/web params on 172.23.123.102:8091:port=8091&username=Administrator&password=password 2020-07-16 15:35:13,157 - root - INFO - --> status:True 2020-07-16 15:35:13,159 - root - INFO - SSH Connecting to 172.23.123.102 with username:root, attempt#1 of 5 2020-07-16 15:35:13,258 - root - INFO - SSH Connected to 172.23.123.102 as root 2020-07-16 15:35:13,520 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:35:13,826 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:35:13,836 - root - INFO - running command.raw on 172.23.123.102: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:35:13,893 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 296 6071 --:--:-- --:--:-- --:--:-- 6833'] ... 2020-07-16 15:35:13,894 - root - INFO - ['ok'] 2020-07-16 15:35:13,897 - root - INFO - /diag/eval status on 172.23.123.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:35:13,898 - root - INFO - settings/indexes params : storageMode=forestdb 2020-07-16 15:35:13,906 - root - ERROR - POST http://172.23.123.102:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:password 2020-07-16 15:35:14,917 - root - INFO - adding node 172.23.123.102:8091 to cluster 2020-07-16 15:35:14,917 - root - INFO - adding remote node @172.23.123.102:8091 to this cluster @172.23.107.135:8091 2020-07-16 15:35:19,120 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.135,ns_1@172.23.123.102', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2020-07-16 15:35:19,131 - root - INFO - rebalance operation started 2020-07-16 15:35:19,135 - root - INFO - rebalance percentage : 0.00 % 2020-07-16 15:35:19,136 - root - INFO - Rebalance - status: running, progress: 0.00% 2020-07-16 15:35:29,169 - root - INFO - Rebalance - status: none, progress: 100.00% 2020-07-16 15:35:29,178 - root - INFO - rebalancing was completed with progress: 100% in 10.046957015991211 sec 2020-07-16 15:35:29,193 - root - ERROR - DELETE http://172.23.123.103:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2020-07-16 15:35:29,194 - root - INFO - Exception while deleting user. Exception is -b'"User was not found."' 2020-07-16 15:35:29,269 - root - ERROR - DELETE http://172.23.107.135:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2020-07-16 15:35:29,270 - root - INFO - Exception while deleting user. Exception is -b'"User was not found."' 2020-07-16 15:35:29,574 - root - INFO - http://172.23.123.103:8091/pools/default/buckets with param: name=default&authType=sasl&saslPassword=None&ramQuotaMB=1647&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2020-07-16 15:35:29,580 - root - INFO - 0.01 seconds to create bucket default 2020-07-16 15:35:29,580 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:35:30,228 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:30,282 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:30,404 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:30,460 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:30,531 - root - WARNING - vbucket map not ready after try 0 2020-07-16 15:35:30,532 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:35:30,633 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:30,701 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:30,807 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:30,861 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:30,936 - root - WARNING - vbucket map not ready after try 1 2020-07-16 15:35:30,937 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:35:31,029 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:31,121 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:31,210 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:31,278 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:31,373 - root - WARNING - vbucket map not ready after try 2 2020-07-16 15:35:31,373 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:35:31,462 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:31,562 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:31,647 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:31,738 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:31,785 - root - WARNING - vbucket map not ready after try 3 2020-07-16 15:35:31,785 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:35:31,893 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:31,958 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:32,053 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:32,141 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:32,188 - root - WARNING - vbucket map not ready after try 4 2020-07-16 15:35:32,189 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:35:32,300 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:32,356 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:32,466 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:35:32,523 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:35:32,573 - root - WARNING - vbucket map not ready after try 5 2020-07-16 15:35:33,222 - root - INFO - http://172.23.107.135:8091/pools/default/buckets with param: name=default&authType=sasl&saslPassword=None&ramQuotaMB=1647&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2020-07-16 15:35:33,232 - root - INFO - 0.01 seconds to create bucket default 2020-07-16 15:35:33,232 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:35:33,952 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:34,036 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:34,179 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:34,260 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:34,338 - root - WARNING - vbucket map not ready after try 0 2020-07-16 15:35:34,338 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:35:34,470 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:34,558 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:34,704 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:34,800 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:34,885 - root - WARNING - vbucket map not ready after try 1 2020-07-16 15:35:34,885 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:35:35,046 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:35,176 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:35,284 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:35,371 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:35,449 - root - WARNING - vbucket map not ready after try 2 2020-07-16 15:35:35,450 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:35:35,582 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:35,704 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:35,801 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:35,919 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:35,962 - root - WARNING - vbucket map not ready after try 3 2020-07-16 15:35:35,962 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:35:36,162 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:36,265 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:36,390 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:36,520 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:36,567 - root - WARNING - vbucket map not ready after try 4 2020-07-16 15:35:36,567 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:35:36,734 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:36,827 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:36,973 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:35:37,068 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:35:37,126 - root - WARNING - vbucket map not ready after try 5 2020-07-16 15:35:37,130 - root - INFO - ==== XDCRNewbasetests setup is finished for test #1 suite_setUp ==== 2020-07-16 15:35:37,169 - root - INFO - ==== XDCRNewbasetests cleanup is started for test #1 suite_setUp ==== 2020-07-16 15:35:37,169 - root - INFO - removing xdcr/nodes settings 2020-07-16 15:35:37,193 - root - INFO - cleanup [ip:172.23.123.103 port:8091 ssh_username:root, ip:172.23.123.101 port:8091 ssh_username:root] 2020-07-16 15:35:37,212 - root - INFO - deleting existing buckets ['default'] on 172.23.123.103 2020-07-16 15:35:38,098 - root - INFO - waiting for bucket deletion to complete.... 2020-07-16 15:35:38,102 - root - INFO - node 172.23.123.103 existing buckets : [] 2020-07-16 15:35:38,102 - root - INFO - deleted bucket : default from 172.23.123.103 2020-07-16 15:35:38,106 - root - INFO - -->is_ns_server_running? 2020-07-16 15:35:38,118 - root - INFO - rebalancing all nodes in order to remove nodes 2020-07-16 15:35:38,120 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.101,ns_1@172.23.123.103', 'ejectedNodes': 'ns_1@172.23.123.101', 'user': 'Administrator', 'password': 'password'} 2020-07-16 15:35:38,124 - root - INFO - rebalance operation started 2020-07-16 15:35:38,128 - root - INFO - rebalance percentage : 0.00 % 2020-07-16 15:35:58,149 - root - INFO - rebalance progress took 20.03 seconds 2020-07-16 15:35:58,150 - root - INFO - sleep for 10 seconds after rebalance... 2020-07-16 15:36:08,168 - root - INFO - removed all the nodes from cluster associated with ip:172.23.123.103 port:8091 ssh_username:root ? [('ns_1@172.23.123.101', 8091)] 2020-07-16 15:36:08,171 - root - INFO - waiting for ns_server @ 172.23.123.103:8091 2020-07-16 15:36:08,172 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:08,175 - root - INFO - ns_server @ 172.23.123.103:8091 is running 2020-07-16 15:36:08,181 - root - INFO - Could not find any buckets for node 172.23.123.101, nothing to delete 2020-07-16 15:36:08,185 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:08,199 - root - INFO - waiting for ns_server @ 172.23.123.101:8091 2020-07-16 15:36:08,199 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:08,202 - root - INFO - ns_server @ 172.23.123.101:8091 is running Cluster instance shutdown with force 2020-07-16 15:36:08,203 - root - INFO - removing xdcr/nodes settings 2020-07-16 15:36:08,238 - root - INFO - cleanup [ip:172.23.107.135 port:8091 ssh_username:root, ip:172.23.123.102 port:8091 ssh_username:root] 2020-07-16 15:36:08,301 - root - INFO - deleting existing buckets ['default'] on 172.23.107.135 2020-07-16 15:36:09,727 - root - INFO - waiting for bucket deletion to complete.... 2020-07-16 15:36:09,731 - root - INFO - node 172.23.107.135 existing buckets : [] 2020-07-16 15:36:09,732 - root - INFO - deleted bucket : default from 172.23.107.135 2020-07-16 15:36:09,739 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:09,759 - root - INFO - rebalancing all nodes in order to remove nodes 2020-07-16 15:36:09,768 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.135,ns_1@172.23.123.102', 'ejectedNodes': 'ns_1@172.23.123.102', 'user': 'Administrator', 'password': 'password'} 2020-07-16 15:36:09,776 - root - INFO - rebalance operation started 2020-07-16 15:36:09,784 - root - INFO - rebalance percentage : 0.00 % 2020-07-16 15:36:29,814 - root - INFO - rebalance progress took 20.04 seconds 2020-07-16 15:36:29,815 - root - INFO - sleep for 10 seconds after rebalance... 2020-07-16 15:36:39,833 - root - INFO - removed all the nodes from cluster associated with ip:172.23.107.135 port:8091 ssh_username:root ? [('ns_1@172.23.123.102', 8091)] 2020-07-16 15:36:39,840 - root - INFO - waiting for ns_server @ 172.23.107.135:8091 2020-07-16 15:36:39,840 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:39,846 - root - INFO - ns_server @ 172.23.107.135:8091 is running 2020-07-16 15:36:39,852 - root - INFO - Could not find any buckets for node 172.23.123.102, nothing to delete 2020-07-16 15:36:39,855 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:39,867 - root - INFO - waiting for ns_server @ 172.23.123.102:8091 2020-07-16 15:36:39,867 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:39,871 - root - INFO - ns_server @ 172.23.123.102:8091 is running Cluster instance shutdown with force 2020-07-16 15:36:39,871 - root - INFO - ==== XDCRNewbasetests cleanup is finished for test #1 suite_setUp === ok ---------------------------------------------------------------------- Ran 1 test in 119.505s OK test_crud_ops_from_cb_to_es (xdcr.capiXDCR.Capi) ... Cluster instance shutdown with force -->result: 2020-07-16 15:36:39,882 - root - INFO - /diag/eval status on 172.23.123.103:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2020-07-16 15:36:39,884 - root - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 2020-07-16 15:36:39,984 - root - INFO - SSH Connected to 172.23.123.103 as root 2020-07-16 15:36:40,266 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:40,580 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:40,581 - root - INFO - running command.raw on 172.23.123.103: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:40,642 - root - INFO - command executed successfully with root 2020-07-16 15:36:40,643 - root - INFO - 0 2020-07-16 15:36:40,645 - root - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 2020-07-16 15:36:40,745 - root - INFO - SSH Connected to 172.23.123.103 as root 2020-07-16 15:36:41,022 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:41,332 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:41,333 - root - INFO - running command.raw on 172.23.123.103: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:41,394 - root - INFO - command executed successfully with root 2020-07-16 15:36:41,394 - root - INFO - 0 2020-07-16 15:36:41,396 - root - INFO - SSH Connecting to 172.23.123.101 with username:root, attempt#1 of 5 2020-07-16 15:36:41,497 - root - INFO - SSH Connected to 172.23.123.101 as root 2020-07-16 15:36:41,768 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:42,072 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:42,072 - root - INFO - running command.raw on 172.23.123.101: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:42,131 - root - INFO - command executed successfully with root 2020-07-16 15:36:42,132 - root - INFO - 0 2020-07-16 15:36:42,133 - root - INFO - SSH Connecting to 172.23.123.101 with username:root, attempt#1 of 5 2020-07-16 15:36:42,231 - root - INFO - SSH Connected to 172.23.123.101 as root 2020-07-16 15:36:42,501 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:42,810 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:42,810 - root - INFO - running command.raw on 172.23.123.101: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:42,870 - root - INFO - command executed successfully with root 2020-07-16 15:36:42,871 - root - INFO - 0 2020-07-16 15:36:42,873 - root - INFO - SSH Connecting to 172.23.107.135 with username:root, attempt#1 of 5 2020-07-16 15:36:43,070 - root - INFO - SSH Connected to 172.23.107.135 as root 2020-07-16 15:36:43,614 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:44,055 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:44,056 - root - INFO - running command.raw on 172.23.107.135: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:44,140 - root - INFO - command executed successfully with root 2020-07-16 15:36:44,141 - root - INFO - 0 2020-07-16 15:36:44,143 - root - INFO - SSH Connecting to 172.23.107.135 with username:root, attempt#1 of 5 2020-07-16 15:36:44,323 - root - INFO - SSH Connected to 172.23.107.135 as root 2020-07-16 15:36:44,864 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:45,293 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:45,295 - root - INFO - running command.raw on 172.23.107.135: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:45,379 - root - INFO - command executed successfully with root 2020-07-16 15:36:45,380 - root - INFO - 0 2020-07-16 15:36:45,382 - root - INFO - SSH Connecting to 172.23.123.102 with username:root, attempt#1 of 5 2020-07-16 15:36:45,487 - root - INFO - SSH Connected to 172.23.123.102 as root 2020-07-16 15:36:45,752 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:46,061 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:46,062 - root - INFO - running command.raw on 172.23.123.102: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:46,124 - root - INFO - command executed successfully with root 2020-07-16 15:36:46,124 - root - INFO - 0 2020-07-16 15:36:46,126 - root - INFO - SSH Connecting to 172.23.123.102 with username:root, attempt#1 of 5 2020-07-16 15:36:46,227 - root - INFO - SSH Connected to 172.23.123.102 as root 2020-07-16 15:36:46,488 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:46,797 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:46,798 - root - INFO - running command.raw on 172.23.123.102: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:46,858 - root - INFO - command executed successfully with root 2020-07-16 15:36:46,859 - root - INFO - 0 2020-07-16 15:36:46,861 - root - INFO - SSH Connecting to 172.23.123.100 with username:root, attempt#1 of 5 2020-07-16 15:36:46,962 - root - INFO - SSH Connected to 172.23.123.100 as root 2020-07-16 15:36:47,251 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:47,558 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:47,559 - root - INFO - running command.raw on 172.23.123.100: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:47,620 - root - INFO - command executed successfully with root 2020-07-16 15:36:47,621 - root - INFO - 0 2020-07-16 15:36:47,623 - root - INFO - SSH Connecting to 172.23.123.100 with username:root, attempt#1 of 5 2020-07-16 15:36:47,723 - root - INFO - SSH Connected to 172.23.123.100 as root 2020-07-16 15:36:48,005 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:48,321 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:48,322 - root - INFO - running command.raw on 172.23.123.100: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:48,382 - root - INFO - command executed successfully with root 2020-07-16 15:36:48,383 - root - INFO - 0 2020-07-16 15:36:48,383 - root - INFO - {'172.23.123.103': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.123.101': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.107.135': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.123.102': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.123.100': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}} 2020-07-16 15:36:48,391 - root - INFO - /diag/eval status on 172.23.123.103:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2020-07-16 15:36:48,394 - root - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 2020-07-16 15:36:48,496 - root - INFO - SSH Connected to 172.23.123.103 as root 2020-07-16 15:36:48,781 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:49,093 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:49,093 - root - INFO - running command.raw on 172.23.123.103: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:49,154 - root - INFO - command executed successfully with root 2020-07-16 15:36:49,154 - root - INFO - 0 2020-07-16 15:36:49,156 - root - INFO - SSH Connecting to 172.23.123.101 with username:root, attempt#1 of 5 2020-07-16 15:36:49,259 - root - INFO - SSH Connected to 172.23.123.101 as root 2020-07-16 15:36:49,531 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:49,838 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:49,839 - root - INFO - running command.raw on 172.23.123.101: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:49,899 - root - INFO - command executed successfully with root 2020-07-16 15:36:49,900 - root - INFO - 0 2020-07-16 15:36:49,902 - root - INFO - SSH Connecting to 172.23.107.135 with username:root, attempt#1 of 5 2020-07-16 15:36:50,083 - root - INFO - SSH Connected to 172.23.107.135 as root 2020-07-16 15:36:50,609 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:51,053 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:51,055 - root - INFO - running command.raw on 172.23.107.135: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:51,148 - root - INFO - command executed successfully with root 2020-07-16 15:36:51,149 - root - INFO - 0 2020-07-16 15:36:51,151 - root - INFO - SSH Connecting to 172.23.123.102 with username:root, attempt#1 of 5 2020-07-16 15:36:51,249 - root - INFO - SSH Connected to 172.23.123.102 as root 2020-07-16 15:36:51,511 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:51,822 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:51,822 - root - INFO - running command.raw on 172.23.123.102: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:51,882 - root - INFO - command executed successfully with root 2020-07-16 15:36:51,882 - root - INFO - 0 2020-07-16 15:36:51,884 - root - INFO - SSH Connecting to 172.23.123.100 with username:root, attempt#1 of 5 2020-07-16 15:36:51,984 - root - INFO - SSH Connected to 172.23.123.100 as root 2020-07-16 15:36:52,265 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:52,570 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:52,572 - root - INFO - running command.raw on 172.23.123.100: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2020-07-16 15:36:52,633 - root - INFO - command executed successfully with root 2020-07-16 15:36:52,634 - root - INFO - 0 2020-07-16 15:36:52,634 - root - INFO - {'172.23.123.103': 0, '172.23.123.101': 0, '172.23.107.135': 0, '172.23.123.102': 0, '172.23.123.100': 0} 2020-07-16 15:36:52,634 - root - INFO - ==== XDCRNewbasetests setup is started for test #1 test_crud_ops_from_cb_to_es ==== 2020-07-16 15:36:52,636 - root - INFO - removing xdcr/nodes settings 2020-07-16 15:36:52,655 - root - INFO - cleanup [ip:172.23.123.103 port:8091 ssh_username:root, ip:172.23.123.101 port:8091 ssh_username:root] 2020-07-16 15:36:52,661 - root - INFO - Could not find any buckets for node 172.23.123.103, nothing to delete 2020-07-16 15:36:52,664 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:52,677 - root - INFO - waiting for ns_server @ 172.23.123.103:8091 2020-07-16 15:36:52,678 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:52,681 - root - INFO - ns_server @ 172.23.123.103:8091 is running 2020-07-16 15:36:52,687 - root - INFO - Could not find any buckets for node 172.23.123.101, nothing to delete 2020-07-16 15:36:52,690 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:52,701 - root - INFO - waiting for ns_server @ 172.23.123.101:8091 2020-07-16 15:36:52,702 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:52,705 - root - INFO - ns_server @ 172.23.123.101:8091 is running 2020-07-16 15:36:52,705 - root - INFO - removing xdcr/nodes settings 2020-07-16 15:36:52,736 - root - INFO - cleanup [ip:172.23.107.135 port:8091 ssh_username:root, ip:172.23.123.102 port:8091 ssh_username:root] 2020-07-16 15:36:52,745 - root - INFO - Could not find any buckets for node 172.23.107.135, nothing to delete 2020-07-16 15:36:52,751 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:52,774 - root - INFO - waiting for ns_server @ 172.23.107.135:8091 2020-07-16 15:36:52,774 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:52,779 - root - INFO - ns_server @ 172.23.107.135:8091 is running 2020-07-16 15:36:52,786 - root - INFO - Could not find any buckets for node 172.23.123.102, nothing to delete 2020-07-16 15:36:52,789 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:52,800 - root - INFO - waiting for ns_server @ 172.23.123.102:8091 2020-07-16 15:36:52,800 - root - INFO - -->is_ns_server_running? 2020-07-16 15:36:52,803 - root - INFO - ns_server @ 172.23.123.102:8091 is running 2020-07-16 15:36:52,804 - root - INFO - Initializing all clusters... 2020-07-16 15:36:52,809 - root - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 2020-07-16 15:36:52,910 - root - INFO - SSH Connected to 172.23.123.103 as root 2020-07-16 15:36:53,195 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:53,503 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:53,514 - root - INFO - running command.raw on 172.23.123.103: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:36:53,569 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 292 6002 --:--:-- --:--:-- --:--:-- 6833'] ... 2020-07-16 15:36:53,569 - root - INFO - ['ok'] 2020-07-16 15:36:53,570 - root - INFO - Enabled diag/eval for non-local hosts from 172.23.123.103 2020-07-16 15:36:53,573 - root - INFO - /diag/eval status on 172.23.123.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:36:53,641 - root - INFO - server: ip:172.23.123.103 port:8091 ssh_username:root, nodes/self 2020-07-16 15:36:53,645 - root - INFO - {'uptime': '304', 'memoryTotal': 4201672704, 'memoryFree': 3576897536, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.103:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-2595-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.103', 'ip': '172.23.123.103', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2020-07-16 15:36:53,650 - root - INFO - pools/default params : memoryQuota=2147 2020-07-16 15:36:53,654 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-16 15:36:53,654 - root - INFO - settings/web params on 172.23.123.103:8091:port=8091&username=Administrator&password=password 2020-07-16 15:36:53,691 - root - INFO - --> status:True 2020-07-16 15:36:53,693 - root - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 2020-07-16 15:36:53,791 - root - INFO - SSH Connected to 172.23.123.103 as root 2020-07-16 15:36:54,065 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:54,376 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:54,387 - root - INFO - running command.raw on 172.23.123.103: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:36:54,443 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 296 6086 --:--:-- --:--:-- --:--:-- 6833'] ... 2020-07-16 15:36:54,443 - root - INFO - ['ok'] 2020-07-16 15:36:54,446 - root - INFO - /diag/eval status on 172.23.123.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:36:54,447 - root - INFO - settings/indexes params : storageMode=forestdb 2020-07-16 15:36:54,449 - root - ERROR - POST http://172.23.123.103:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:password 2020-07-16 15:36:54,464 - root - INFO - server: ip:172.23.123.101 port:8091 ssh_username:root, nodes/self 2020-07-16 15:36:54,468 - root - INFO - {'uptime': '302', 'memoryTotal': 4201672704, 'memoryFree': 3585970176, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-2595-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.101', 'ip': '172.23.123.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2020-07-16 15:36:54,470 - root - ERROR - GET http://172.23.123.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2020-07-16 15:36:54,470 - root - INFO - pools/default params : memoryQuota=2147 2020-07-16 15:36:54,474 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-16 15:36:54,475 - root - INFO - settings/web params on 172.23.123.101:8091:port=8091&username=Administrator&password=password 2020-07-16 15:36:54,511 - root - INFO - --> status:True 2020-07-16 15:36:54,512 - root - INFO - SSH Connecting to 172.23.123.101 with username:root, attempt#1 of 5 2020-07-16 15:36:54,615 - root - INFO - SSH Connected to 172.23.123.101 as root 2020-07-16 15:36:54,885 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:36:55,194 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:36:55,204 - root - INFO - running command.raw on 172.23.123.101: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:36:55,259 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 298 6109 --:--:-- --:--:-- --:--:-- 6833'] ... 2020-07-16 15:36:55,260 - root - INFO - ['ok'] 2020-07-16 15:36:55,263 - root - INFO - /diag/eval status on 172.23.123.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:36:55,263 - root - INFO - settings/indexes params : storageMode=forestdb 2020-07-16 15:36:55,265 - root - ERROR - POST http://172.23.123.101:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:password 2020-07-16 15:36:56,272 - root - INFO - adding node 172.23.123.101:8091 to cluster 2020-07-16 15:36:56,272 - root - INFO - adding remote node @172.23.123.101:8091 to this cluster @172.23.123.103:8091 2020-07-16 15:36:58,207 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.101,ns_1@172.23.123.103', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2020-07-16 15:36:58,212 - root - INFO - rebalance operation started 2020-07-16 15:36:58,215 - root - INFO - rebalance percentage : 0.00 % 2020-07-16 15:36:58,215 - root - INFO - Rebalance - status: running, progress: 0.00% 2020-07-16 15:37:08,238 - root - INFO - Rebalance - status: none, progress: 100.00% 2020-07-16 15:37:08,243 - root - INFO - rebalancing was completed with progress: 100% in 10.03073787689209 sec 2020-07-16 15:37:08,252 - root - INFO - SSH Connecting to 172.23.107.135 with username:root, attempt#1 of 5 2020-07-16 15:37:08,434 - root - INFO - SSH Connected to 172.23.107.135 as root 2020-07-16 15:37:08,986 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:37:09,432 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:37:09,448 - root - INFO - running command.raw on 172.23.107.135: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:37:09,531 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 315 6471 --:--:-- --:--:-- --:--:-- 8200'] ... 2020-07-16 15:37:09,532 - root - INFO - ['ok'] 2020-07-16 15:37:09,532 - root - INFO - Enabled diag/eval for non-local hosts from 172.23.107.135 2020-07-16 15:37:09,536 - root - INFO - /diag/eval status on 172.23.107.135:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:37:09,663 - root - INFO - server: ip:172.23.107.135 port:8091 ssh_username:root, nodes/self 2020-07-16 15:37:09,669 - root - INFO - {'uptime': '252', 'memoryTotal': 4219432960, 'memoryFree': 1153064960, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.107.135:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-2595-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2156, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.135', 'ip': '172.23.107.135', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2020-07-16 15:37:09,677 - root - INFO - pools/default params : memoryQuota=2156 2020-07-16 15:37:09,685 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-16 15:37:09,685 - root - INFO - settings/web params on 172.23.107.135:8091:port=8091&username=Administrator&password=password 2020-07-16 15:37:09,781 - root - INFO - --> status:True 2020-07-16 15:37:09,784 - root - INFO - SSH Connecting to 172.23.107.135 with username:root, attempt#1 of 5 2020-07-16 15:37:09,958 - root - INFO - SSH Connected to 172.23.107.135 as root 2020-07-16 15:37:10,468 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:37:10,899 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:37:10,915 - root - INFO - running command.raw on 172.23.107.135: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:37:10,992 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 332 6817 --:--:-- --:--:-- --:--:-- 8200'] ... 2020-07-16 15:37:10,992 - root - INFO - ['ok'] 2020-07-16 15:37:10,996 - root - INFO - /diag/eval status on 172.23.107.135:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:37:10,996 - root - INFO - settings/indexes params : storageMode=forestdb 2020-07-16 15:37:11,001 - root - ERROR - POST http://172.23.107.135:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:password 2020-07-16 15:37:11,018 - root - INFO - server: ip:172.23.123.102 port:8091 ssh_username:root, nodes/self 2020-07-16 15:37:11,021 - root - INFO - {'uptime': '318', 'memoryTotal': 4201672704, 'memoryFree': 3592650752, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.102:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-2595-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.102', 'ip': '172.23.123.102', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2020-07-16 15:37:11,023 - root - ERROR - GET http://172.23.123.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2020-07-16 15:37:11,024 - root - INFO - pools/default params : memoryQuota=2147 2020-07-16 15:37:11,027 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-16 15:37:11,027 - root - INFO - settings/web params on 172.23.123.102:8091:port=8091&username=Administrator&password=password 2020-07-16 15:37:11,063 - root - INFO - --> status:True 2020-07-16 15:37:11,065 - root - INFO - SSH Connecting to 172.23.123.102 with username:root, attempt#1 of 5 2020-07-16 15:37:11,172 - root - INFO - SSH Connected to 172.23.123.102 as root 2020-07-16 15:37:11,435 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:37:11,743 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-16 15:37:11,754 - root - INFO - running command.raw on 172.23.123.102: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2020-07-16 15:37:11,810 - root - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 292 5998 --:--:-- --:--:-- --:--:-- 6833'] ... 2020-07-16 15:37:11,811 - root - INFO - ['ok'] 2020-07-16 15:37:11,815 - root - INFO - /diag/eval status on 172.23.123.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2020-07-16 15:37:11,815 - root - INFO - settings/indexes params : storageMode=forestdb 2020-07-16 15:37:11,818 - root - ERROR - POST http://172.23.123.102:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:password 2020-07-16 15:37:12,825 - root - INFO - adding node 172.23.123.102:8091 to cluster 2020-07-16 15:37:12,826 - root - INFO - adding remote node @172.23.123.102:8091 to this cluster @172.23.107.135:8091 2020-07-16 15:37:15,237 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.135,ns_1@172.23.123.102', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2020-07-16 15:37:15,245 - root - INFO - rebalance operation started 2020-07-16 15:37:15,248 - root - INFO - rebalance percentage : 0.00 % 2020-07-16 15:37:15,248 - root - INFO - Rebalance - status: running, progress: 0.00% 2020-07-16 15:37:25,272 - root - INFO - Rebalance - status: none, progress: 100.00% 2020-07-16 15:37:25,282 - root - INFO - rebalancing was completed with progress: 100% in 10.037172079086304 sec 2020-07-16 15:37:25,303 - root - ERROR - DELETE http://172.23.123.103:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2020-07-16 15:37:25,304 - root - INFO - Exception while deleting user. Exception is -b'"User was not found."' 2020-07-16 15:37:25,383 - root - ERROR - DELETE http://172.23.107.135:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2020-07-16 15:37:25,384 - root - INFO - Exception while deleting user. Exception is -b'"User was not found."' 2020-07-16 15:37:26,283 - root - INFO - http://172.23.123.103:8091/pools/default/buckets with param: name=default&authType=sasl&saslPassword=None&ramQuotaMB=1647&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2020-07-16 15:37:26,289 - root - INFO - 0.01 seconds to create bucket default 2020-07-16 15:37:26,290 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:37:26,909 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:27,021 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:27,142 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:27,245 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:27,297 - root - WARNING - vbucket map not ready after try 0 2020-07-16 15:37:27,298 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:37:27,394 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:27,494 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:27,592 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:27,682 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:27,737 - root - WARNING - vbucket map not ready after try 1 2020-07-16 15:37:27,738 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:37:27,855 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:27,912 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:28,024 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:28,079 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:28,133 - root - WARNING - vbucket map not ready after try 2 2020-07-16 15:37:28,133 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:37:28,259 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:28,323 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:28,473 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:28,532 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:28,576 - root - WARNING - vbucket map not ready after try 3 2020-07-16 15:37:28,576 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:37:28,701 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:28,761 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:28,895 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:28,959 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:29,045 - root - WARNING - vbucket map not ready after try 4 2020-07-16 15:37:29,046 - root - INFO - waiting for memcached bucket : default in 172.23.123.103 to accept set ops 2020-07-16 15:37:29,128 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:29,185 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:29,307 - root - INFO - creating direct client 172.23.123.101:11210 default 2020-07-16 15:37:29,365 - root - INFO - creating direct client 172.23.123.103:11210 default 2020-07-16 15:37:29,446 - root - WARNING - vbucket map not ready after try 5 2020-07-16 15:37:30,334 - root - INFO - http://172.23.107.135:8091/pools/default/buckets with param: name=default&authType=sasl&saslPassword=None&ramQuotaMB=1647&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2020-07-16 15:37:30,344 - root - INFO - 0.01 seconds to create bucket default 2020-07-16 15:37:30,344 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:37:31,054 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:31,175 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:31,284 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:31,405 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:31,451 - root - WARNING - vbucket map not ready after try 0 2020-07-16 15:37:31,451 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:37:31,591 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:31,721 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:31,842 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:31,976 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:32,035 - root - WARNING - vbucket map not ready after try 1 2020-07-16 15:37:32,036 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:37:32,232 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:32,340 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:32,484 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:32,572 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:32,628 - root - WARNING - vbucket map not ready after try 2 2020-07-16 15:37:32,628 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:37:32,815 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:32,919 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:33,073 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:33,191 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:33,252 - root - WARNING - vbucket map not ready after try 3 2020-07-16 15:37:33,252 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:37:33,491 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:33,589 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:33,724 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:33,839 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:33,929 - root - WARNING - vbucket map not ready after try 4 2020-07-16 15:37:33,930 - root - INFO - waiting for memcached bucket : default in 172.23.107.135 to accept set ops 2020-07-16 15:37:34,076 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:34,167 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:34,321 - root - INFO - creating direct client 172.23.107.135:11210 default 2020-07-16 15:37:34,407 - root - INFO - creating direct client 172.23.123.102:11210 default 2020-07-16 15:37:34,507 - root - WARNING - vbucket map not ready after try 5 2020-07-16 15:37:34,510 - root - INFO - ==== XDCRNewbasetests setup is finished for test #1 test_crud_ops_from_cb_to_es ==== 2020-07-16 15:37:34,520 - root - INFO - adding remote cluster hostname:172.23.107.135:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.123.103:8091 2020-07-16 15:37:34,544 - root - INFO - starting continuous replication type:capi from default to default in the remote cluster remote_cluster_C1-C2 with settings {} 2020-07-16 15:37:34,591 - root - INFO - Replication created with id: a466a3fa5b9b0293471616d5834872fe/default/default 2020-07-16 15:37:34,611 - root - INFO - Updated checkpointInterval=60 on bucket'default' on 172.23.123.103 2020-07-16 15:37:34,645 - root - ERROR - GET http://172.23.123.103:8091/settings/replications/a466a3fa5b9b0293471616d5834872fe%2Fdefault%2Fdefault body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password 2020-07-16 15:37:34,678 - root - INFO - ==== XDCRNewbasetests cleanup is started for test #1 test_crud_ops_from_cb_to_es ==== 2020-07-16 15:37:34,678 - root - INFO - removing xdcr/nodes settings 2020-07-16 15:37:34,690 - root - ERROR - GET http://172.23.123.103:8091/pools/default/remoteClusters/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password 2020-07-16 15:37:34,700 - root - INFO - cleanup [ip:172.23.123.103 port:8091 ssh_username:root, ip:172.23.123.101 port:8091 ssh_username:root] 2020-07-16 15:37:34,717 - root - INFO - deleting existing buckets ['default'] on 172.23.123.103 2020-07-16 15:37:35,713 - root - INFO - waiting for bucket deletion to complete.... 2020-07-16 15:37:35,716 - root - INFO - node 172.23.123.103 existing buckets : [] 2020-07-16 15:37:35,716 - root - INFO - deleted bucket : default from 172.23.123.103 2020-07-16 15:37:35,721 - root - INFO - -->is_ns_server_running? 2020-07-16 15:37:35,731 - root - INFO - rebalancing all nodes in order to remove nodes 2020-07-16 15:37:35,733 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.101,ns_1@172.23.123.103', 'ejectedNodes': 'ns_1@172.23.123.101', 'user': 'Administrator', 'password': 'password'} 2020-07-16 15:37:35,737 - root - INFO - rebalance operation started 2020-07-16 15:37:35,740 - root - INFO - rebalance percentage : 0.00 % 2020-07-16 15:37:55,767 - root - INFO - rebalance progress took 20.03 seconds 2020-07-16 15:37:55,767 - root - INFO - sleep for 10 seconds after rebalance... 2020-07-16 15:38:05,783 - root - INFO - removed all the nodes from cluster associated with ip:172.23.123.103 port:8091 ssh_username:root ? [('ns_1@172.23.123.101', 8091)] 2020-07-16 15:38:05,788 - root - INFO - waiting for ns_server @ 172.23.123.103:8091 2020-07-16 15:38:05,788 - root - INFO - -->is_ns_server_running? 2020-07-16 15:38:05,792 - root - INFO - ns_server @ 172.23.123.103:8091 is running 2020-07-16 15:38:05,797 - root - INFO - Could not find any buckets for node 172.23.123.101, nothing to delete 2020-07-16 15:38:05,801 - root - INFO - -->is_ns_server_running? 2020-07-16 15:38:05,813 - root - INFO - waiting for ns_server @ 172.23.123.101:8091 2020-07-16 15:38:05,813 - root - INFO - -->is_ns_server_running? 2020-07-16 15:38:05,816 - root - INFO - ns_server @ 172.23.123.101:8091 is running Cluster instance shutdown with force 2020-07-16 15:38:05,816 - root - INFO - removing xdcr/nodes settings 2020-07-16 15:38:05,851 - root - INFO - cleanup [ip:172.23.107.135 port:8091 ssh_username:root, ip:172.23.123.102 port:8091 ssh_username:root] 2020-07-16 15:38:05,877 - root - INFO - deleting existing buckets ['default'] on 172.23.107.135 2020-07-16 15:38:06,938 - root - INFO - waiting for bucket deletion to complete.... 2020-07-16 15:38:06,943 - root - INFO - node 172.23.107.135 existing buckets : [] 2020-07-16 15:38:06,943 - root - INFO - deleted bucket : default from 172.23.107.135 2020-07-16 15:38:06,950 - root - INFO - -->is_ns_server_running? 2020-07-16 15:38:06,967 - root - INFO - rebalancing all nodes in order to remove nodes 2020-07-16 15:38:06,971 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.135,ns_1@172.23.123.102', 'ejectedNodes': 'ns_1@172.23.123.102', 'user': 'Administrator', 'password': 'password'} 2020-07-16 15:38:06,980 - root - INFO - rebalance operation started 2020-07-16 15:38:06,984 - root - INFO - rebalance percentage : 0.00 % 2020-07-16 15:38:27,005 - root - INFO - rebalance progress took 20.02 seconds 2020-07-16 15:38:27,005 - root - INFO - sleep for 10 seconds after rebalance... 2020-07-16 15:38:37,023 - root - INFO - removed all the nodes from cluster associated with ip:172.23.107.135 port:8091 ssh_username:root ? [('ns_1@172.23.123.102', 8091)] 2020-07-16 15:38:37,030 - root - INFO - waiting for ns_server @ 172.23.107.135:8091 2020-07-16 15:38:37,030 - root - INFO - -->is_ns_server_running? 2020-07-16 15:38:37,036 - root - INFO - ns_server @ 172.23.107.135:8091 is running 2020-07-16 15:38:37,041 - root - INFO - Could not find any buckets for node 172.23.123.102, nothing to delete 2020-07-16 15:38:37,044 - root - INFO - -->is_ns_server_running? 2020-07-16 15:38:37,056 - root - INFO - waiting for ns_server @ 172.23.123.102:8091 2020-07-16 15:38:37,056 - root - INFO - -->is_ns_server_running? 2020-07-16 15:38:37,060 - root - INFO - ns_server @ 172.23.123.102:8091 is running Cluster instance shutdown with force 2020-07-16 15:38:37,060 - root - INFO - ==== XDCRNewbasetests cleanup is finished for test #1 test_crud_ops_from_cb_to_es === Cluster instance shutdown with force grabbing cbcollect from 172.23.123.103 -->get_cbcollect_info: :ip:172.23.123.103 port:8091 ssh_username:root,:/data/workspace/centos-p0-xdcr-vset00-00-capi/logs/testrunner-20-Jul-16_15-34-39/test_1 2020-07-16 15:38:37,064 - root - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 2020-07-16 15:38:37,165 - root - INFO - SSH Connected to 172.23.123.103 as root 2020-07-16 15:38:37,444 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:38:37,749 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.123.103 2020-07-16 15:38:37,750 - root - INFO - running command.raw on 172.23.123.103: /opt/couchbase/bin/cbcollect_info 172.23.123.103-20200716-1538-diag.zip 2020-07-16 15:39:58,362 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.123.103 2020-07-16 15:39:58,577 - root - INFO - running command.raw on 172.23.123.103: rm -f /root/172.23.123.103-20200716-1538-diag.zip 2020-07-16 15:39:58,634 - root - INFO - command executed successfully with root grabbing cbcollect from 172.23.123.101 -->get_cbcollect_info: :ip:172.23.123.101 port:8091 ssh_username:root,:/data/workspace/centos-p0-xdcr-vset00-00-capi/logs/testrunner-20-Jul-16_15-34-39/test_1 2020-07-16 15:39:58,638 - root - INFO - SSH Connecting to 172.23.123.101 with username:root, attempt#1 of 5 2020-07-16 15:39:58,741 - root - INFO - SSH Connected to 172.23.123.101 as root 2020-07-16 15:39:59,012 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:39:59,321 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.123.101 2020-07-16 15:39:59,322 - root - INFO - running command.raw on 172.23.123.101: /opt/couchbase/bin/cbcollect_info 172.23.123.101-20200716-1539-diag.zip 2020-07-16 15:41:17,272 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.123.101 2020-07-16 15:41:17,480 - root - INFO - running command.raw on 172.23.123.101: rm -f /root/172.23.123.101-20200716-1539-diag.zip 2020-07-16 15:41:17,535 - root - INFO - command executed successfully with root grabbing cbcollect from 172.23.107.135 -->get_cbcollect_info: :ip:172.23.107.135 port:8091 ssh_username:root,:/data/workspace/centos-p0-xdcr-vset00-00-capi/logs/testrunner-20-Jul-16_15-34-39/test_1 2020-07-16 15:41:17,538 - root - INFO - SSH Connecting to 172.23.107.135 with username:root, attempt#1 of 5 2020-07-16 15:41:17,721 - root - INFO - SSH Connected to 172.23.107.135 as root 2020-07-16 15:41:18,265 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:41:18,703 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.135 2020-07-16 15:41:18,704 - root - INFO - running command.raw on 172.23.107.135: /opt/couchbase/bin/cbcollect_info 172.23.107.135-20200716-1541-diag.zip 2020-07-16 15:43:17,274 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.107.135 2020-07-16 15:43:17,731 - root - INFO - running command.raw on 172.23.107.135: rm -f /root/172.23.107.135-20200716-1541-diag.zip 2020-07-16 15:43:17,778 - root - INFO - command executed successfully with root grabbing cbcollect from 172.23.123.102 -->get_cbcollect_info: :ip:172.23.123.102 port:8091 ssh_username:root,:/data/workspace/centos-p0-xdcr-vset00-00-capi/logs/testrunner-20-Jul-16_15-34-39/test_1 2020-07-16 15:43:17,780 - root - INFO - SSH Connecting to 172.23.123.102 with username:root, attempt#1 of 5 2020-07-16 15:43:17,885 - root - INFO - SSH Connected to 172.23.123.102 as root 2020-07-16 15:43:18,151 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:43:18,451 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.123.102 2020-07-16 15:43:18,452 - root - INFO - running command.raw on 172.23.123.102: /opt/couchbase/bin/cbcollect_info 172.23.123.102-20200716-1543-diag.zip 2020-07-16 15:44:37,125 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.123.102 2020-07-16 15:44:37,351 - root - INFO - running command.raw on 172.23.123.102: rm -f /root/172.23.123.102-20200716-1543-diag.zip 2020-07-16 15:44:37,407 - root - INFO - command executed successfully with root grabbing cbcollect from 172.23.123.100 -->get_cbcollect_info: :ip:172.23.123.100 port:8091 ssh_username:root,:/data/workspace/centos-p0-xdcr-vset00-00-capi/logs/testrunner-20-Jul-16_15-34-39/test_1 2020-07-16 15:44:37,410 - root - INFO - SSH Connecting to 172.23.123.100 with username:root, attempt#1 of 5 2020-07-16 15:44:37,515 - root - INFO - SSH Connected to 172.23.123.100 as root 2020-07-16 15:44:37,803 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-16 15:44:38,106 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.123.100 2020-07-16 15:44:38,107 - root - INFO - running command.raw on 172.23.123.100: /opt/couchbase/bin/cbcollect_info 172.23.123.100-20200716-1544-diag.zip 2020-07-16 15:45:58,498 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.123.100 2020-07-16 15:45:58,683 - root - INFO - running command.raw on 172.23.123.100: rm -f /root/172.23.123.100-20200716-1544-diag.zip 2020-07-16 15:45:58,737 - root - INFO - command executed successfully with root summary so far suite xdcr.capiXDCR.Capi , pass 0 , fail 1 failures so far... xdcr.capiXDCR.Capi.test_crud_ops_from_cb_to_es testrunner logs, diags and results are available under /data/workspace/centos-p0-xdcr-vset00-00-capi/logs/testrunner-20-Jul-16_15-34-39/test_1 ERROR ====================================================================== ERROR: test_crud_ops_from_cb_to_es (xdcr.capiXDCR.Capi) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/xdcr/capiXDCR.py", line 59, in test_crud_ops_from_cb_to_es self.src_cluster.pause_all_replications() File "pytests/xdcr/xdcrnewbasetests.py", line 2629, in pause_all_replications remote_cluster_ref.pause_all_replications(verify) File "pytests/xdcr/xdcrnewbasetests.py", line 725, in pause_all_replications [repl.pause(verify=verify) for repl in self.__replications] File "pytests/xdcr/xdcrnewbasetests.py", line 725, in [repl.pause(verify=verify) for repl in self.__replications] File "pytests/xdcr/xdcrnewbasetests.py", line 961, in pause self.__from_bucket.name, self.__to_bucket.name): File "lib/membase/api/rest_client.py", line 2999, in is_replication_paused return self.get_xdcr_param(src_bucket_name, dest_bucket_name, 'pauseRequested') File "lib/membase/api/rest_client.py", line 2981, in get_xdcr_param format(param, src_bucket_name, self.ip)) membase.api.exception.XDCRException: Unable to get replication setting pauseRequested on bucket default on node 172.23.123.103 ---------------------------------------------------------------------- Ran 1 test in 117.187s