./testrunner -i /tmp/testexec.30373.ini -p gsi_type=plasma,get-cbcollect-info=True,reset_services=True,java_sdk_client=True,GROUP=P1,enforce_tls=True,use_https=True -t gsi.collections_index_basics.CollectionsIndexBasics.test_gsi_indexes_with_WITH_clause,default_bucket=false,force_clean=true,defer_build=True,nodes_init=2,services_init=kv:n1ql:index-kv:n1ql:index,GROUP=P1,gsi_type=plasma,get-cbcollect-info=True,reset_services=True,java_sdk_client=True,enforce_tls=True,use_https=True,total_testcases=17 Test Input params: {'default_bucket': 'false', 'force_clean': 'true', 'defer_build': 'True', 'nodes_init': '2', 'services_init': 'kv:n1ql:index-kv:n1ql:index', 'GROUP': 'P1', 'gsi_type': 'plasma', 'get-cbcollect-info': 'True', 'reset_services': 'True', 'java_sdk_client': 'True', 'enforce_tls': 'True', 'use_https': 'True', 'total_testcases': 2, 'ini': '/tmp/testexec.30373.ini', 'cluster_name': 'testexec.30373', 'spec': 'collection-index-basics', 'conf_file': 'gsi/collection-index-basics.conf', 'num_nodes': 4, 'case_number': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/data/workspace/centos-p0-2i-vset00-00-collections_gsi_cbd_sanity_7.0_P1/logs/testrunner-22-Apr-21_03-11-23/test_1'} Run before suite setup for gsi.collections_index_basics.CollectionsIndexBasics.test_gsi_indexes_with_WITH_clause lib/couchbase_helper/tuq_generators.py:113: DeprecationWarning: invalid escape sequence \w regex = re.compile("[\w']+\.[\w']+") lib/couchbase_helper/tuq_generators.py:363: DeprecationWarning: invalid escape sequence \[ diff = set(order_clause.split(',')) - set(re.compile('doc\["[\w\']+"\]').findall(select_clause)) pytests/basetestcase.py:3161: DeprecationWarning: invalid escape sequence \ copy_servers, root_cn='Root\ Authority', type="openssl", lib/couchbase_helper/data_analysis_helper.py:905: DeprecationWarning: invalid escape sequence \{ master_key = ".\{12\}$" ./pytests/security/x509main.py:80: DeprecationWarning: invalid escape sequence \. ''' suite_setUp (gsi.collections_index_basics.CollectionsIndexBasics) ... -->before_suite_name:gsi.collections_index_basics.CollectionsIndexBasics.suite_setUp,suite: ]> 2022-04-21 03:11:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:11:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:11:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:25 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:11:25 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2558-enterprise 2022-04-21 03:11:25 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2022-04-21 03:11:25 | INFO | MainProcess | MainThread | [collections_index_basics.tearDown] ============== CollectionsIndexBasics tearDown has started ============== 2022-04-21 03:11:25 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:11:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:11:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:11:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:26 | ERROR | MainProcess | MainThread | [rest_client._http_request] POST https://172.23.97.145:18091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: zgrep "panic" API is accessible from localhost only/indexer.log* | wc -l 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/indexer.log*.gz: No such file or directory'] ... 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:26 | ERROR | MainProcess | MainThread | [rest_client._http_request] POST https://172.23.97.145:18091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: zgrep "panic" API is accessible from localhost only/projector.log* | wc -l 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/projector.log*.gz: No such file or directory'] ... 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ###################### Disabling n2n encryption 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2022-04-21 03:11:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:11:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:11:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:27 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2022-04-21 03:11:28 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2022-04-21 03:11:28 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 03:11:28 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2022-04-21 03:11:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:11:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:11:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:29 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2022-04-21 03:11:29 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned off encryption for node: http://[::1]:8091', 'SUCCESS: Switched node-to-node encryption off'] 2022-04-21 03:11:29 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 03:11:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 03:11:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.97.145:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 0.5021340697966357, 'mem_free': 3163529216, 'mem_total': 3935289344, 'swap_mem_used': 3932160, 'swap_mem_total': 3758092288} 2022-04-21 03:11:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 03:11:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:11:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:11:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:11:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:11:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:11:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:11:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:11:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:11:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.97.145, nothing to delete 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.97.106, nothing to delete 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.242, nothing to delete 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.24, nothing to delete 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-04-21 03:11:40 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE https://172.23.97.145:18091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."' 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.145:8091 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.145:8091 is running 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.106:8091 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.106:8091 is running 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.242:8091 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.242:8091 is running 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.24:8091 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.24:8091 is running 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [collections_index_basics.tearDown] ============== CollectionsIndexBasics tearDown has completed ============== 2022-04-21 03:11:40 | INFO | MainProcess | MainThread | [basetestcase.setUp] Building docker image with java sdk client OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 2022-04-21 03:11:49 | INFO | MainProcess | MainThread | [basetestcase.setUp] #####Enforcing TLS######## 2022-04-21 03:11:49 | INFO | MainProcess | MainThread | [ntonencryptionBase.setup_nton_cluster] Setting up node to node encryption - status = enable and clusterEncryptionLevel = strict 2022-04-21 03:11:49 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 03:11:49 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to enable 2022-04-21 03:11:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:11:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:11:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:50 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --enable 2022-04-21 03:11:51 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned on encryption for node: http://[::1]:8091', 'SUCCESS: Switched node-to-node encryption on'] 2022-04-21 03:11:51 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 03:11:51 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = strict 2022-04-21 03:11:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:11:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:11:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:52 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level strict 2022-04-21 03:11:52 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ['SUCCESS: Security settings updated'] 2022-04-21 03:11:52 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 03:11:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:11:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:11:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:8091 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:8093 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:9102 2022-04-21 03:11:53 | ERROR | MainProcess | MainThread | [cluster_helper.check_if_services_obey_tls] On Server 172.23.97.145 Expected 127.0.0.1:9102 Actual *:9102 !!!!!!!!!!!!!!!! 2022-04-21 03:11:53 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... 2022-04-21 03:12:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:12:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:12:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:8091 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:8093 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:9102 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :11210 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:11210 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:8092 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:18091 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:18093 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :19102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:19102 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :11207 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:11207 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:18092 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :19000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :19001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :19002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :19003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:12:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:06 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2022-04-21 03:12:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:12:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:12:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:12:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:12:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:07 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.97.145 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:12:07 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 03:12:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: systemctl stop couchbase-server.service 2022-04-21 03:12:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:10 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:10 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 03:12:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 03:12:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 03:12:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:12:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:12:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:11 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.97.145 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:12:11 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:11 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 03:12:11 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 03:12:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: systemctl start couchbase-server.service 2022-04-21 03:12:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:11 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.97.145:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 03:12:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 03:12:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:16 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 113104 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 03:12:16 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:16 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.97.145: with pid 113245 2022-04-21 03:12:16 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 03:12:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:12:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:12:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:12:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:12:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:17 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.97.106 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:12:18 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 03:12:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: systemctl stop couchbase-server.service 2022-04-21 03:12:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:20 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:20 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 03:12:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 03:12:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 03:12:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:12:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:12:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:21 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.97.106 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:12:21 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:21 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 03:12:21 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 03:12:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: systemctl start couchbase-server.service 2022-04-21 03:12:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:21 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.97.106:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 03:12:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 03:12:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:26 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 116339 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 03:12:26 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:26 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.97.106: with pid 116480 2022-04-21 03:12:26 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 03:12:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:12:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:12:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:12:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:12:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:28 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.242 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:12:28 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 03:12:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: systemctl stop couchbase-server.service 2022-04-21 03:12:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:31 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:31 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 03:12:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 03:12:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 03:12:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:12:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:12:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:32 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.242 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:12:32 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:32 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 03:12:32 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 03:12:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: systemctl start couchbase-server.service 2022-04-21 03:12:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:32 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.107.242:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 03:12:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 03:12:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:37 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 64972 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 03:12:37 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:38 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.242: with pid 65115 2022-04-21 03:12:38 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 03:12:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:12:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:12:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:12:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:12:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:40 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.24 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:12:40 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 03:12:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: systemctl stop couchbase-server.service 2022-04-21 03:12:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:43 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:43 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 03:12:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 03:12:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 03:12:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:43 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:12:43 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:12:43 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:12:44 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:12:44 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.24 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:12:44 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:44 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 03:12:44 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 03:12:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: systemctl start couchbase-server.service 2022-04-21 03:12:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:44 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.107.24:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 03:12:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 03:12:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:12:49 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 24309 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 03:12:49 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:12:49 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.24: with pid 24449 2022-04-21 03:12:49 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 03:12:50 | ERROR | MainProcess | MainThread | [rest_client._http_request] socket error while connecting to https://172.23.97.145:18091/node/controller/rename error [Errno 111] Connection refused 2022-04-21 03:13:05 | ERROR | MainProcess | MainThread | [rest_client._http_request] socket error while connecting to https://172.23.107.242:18091/node/controller/rename error [Errno 111] Connection refused 2022-04-21 03:13:08 | ERROR | MainProcess | MainThread | [rest_client._http_request] socket error while connecting to https://172.23.107.24:18091/node/controller/rename error [Errno 111] Connection refused 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.97.145 port:8091 ssh_username:root, nodes/self 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '56', 'memoryTotal': 3935289344, 'memoryFree': 3246784512, 'mcdMemoryReserved': 3002, 'mcdMemoryAllocated': 3002, 'status': 'healthy', 'hostname': '172.23.97.145:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1585, 'memcached': 11207, 'id': 'ns_1@172.23.97.145', 'ip': '172.23.97.145', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3752, 'curr_items': 0} 2022-04-21 03:13:12 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.97.145:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.97.145 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1755 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.97.145,18091,['kv', 'n1ql', 'index']) 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.97.145: 18091:hostname=172.23.97.145&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.97.145:18091:port=8091&username=Administrator&password=password 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:13:12 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.97.106 port:8091 ssh_username:root, nodes/self 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '46', 'memoryTotal': 3935289344, 'memoryFree': 3299323904, 'mcdMemoryReserved': 3002, 'mcdMemoryAllocated': 3002, 'status': 'healthy', 'hostname': '172.23.97.106:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1585, 'memcached': 11207, 'id': 'ns_1@172.23.97.106', 'ip': '172.23.97.106', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3752, 'curr_items': 0} 2022-04-21 03:13:13 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.97.106:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2011 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.97.106:18091:port=8091&username=Administrator&password=password 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:13:13 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.106:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.106:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.242 port:8091 ssh_username:root, nodes/self 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '37', 'memoryTotal': 3945775104, 'memoryFree': 3357982720, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.242:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.242', 'ip': '172.23.107.242', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 03:13:14 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.242:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.242:18091:port=8091&username=Administrator&password=password 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:13:14 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:13:15 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:13:15 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:13:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 03:13:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:13:15 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 03:13:15 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.242:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:13:15 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.242:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:13:15 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-04-21 03:13:15 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.24 port:8091 ssh_username:root, nodes/self 2022-04-21 03:13:15 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '26', 'memoryTotal': 3945775104, 'memoryFree': 3331014656, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.24:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.24', 'ip': '172.23.107.24', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 03:13:15 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.24:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 03:13:15 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 03:13:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 03:13:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.24:18091:port=8091&username=Administrator&password=password 2022-04-21 03:13:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 03:13:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:13:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:13:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:13:17 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:13:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 03:13:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:13:17 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 03:13:17 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.24:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:13:17 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.24:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:13:17 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-04-21 03:13:17 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.97.145 **** 2022-04-21 03:13:17 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE https://172.23.97.145:18091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 03:13:17 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-04-21 03:13:17 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2022-04-21 03:13:22 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-04-21 03:13:22 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2022-04-21 03:13:23 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.97.106:8091 to cluster 2022-04-21 03:13:23 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.97.106:18091 to this cluster @172.23.97.145:18091 2022-04-21 03:13:33 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.06 seconds 2022-04-21 03:13:33 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 03:13:47 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.97.106 not part of cluster inactiveAdded 2022-04-21 03:13:47 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.97.106,ns_1@172.23.97.145', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-04-21 03:13:47 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-04-21 03:13:57 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 36.00 % 2022-04-21 03:13:57 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 36.00% 2022-04-21 03:14:17 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-04-21 03:14:17 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 30.129095792770386 sec 2022-04-21 03:14:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:14:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:14:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:14:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:14:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 03:14:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:14:18 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 03:14:18 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.97.145 2022-04-21 03:14:18 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ============== 2022-04-21 03:14:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:14:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:14:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:14:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:14:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:14:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:14:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:14:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:14:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:14:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:14:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:14:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:14:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:14:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:14:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:14:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.97.106:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.23272178939432, 'mem_free': 3201572864, 'mem_total': 3935289344, 'swap_mem_used': 67633152, 'swap_mem_total': 3758092288} 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.97.145:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.303580433686334, 'mem_free': 3133788160, 'mem_total': 3935289344, 'swap_mem_used': 3932160, 'swap_mem_total': 3758092288} 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 184.77768659591675 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.97.145 cluster is healthy 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.97.145 is healthy 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.97.106 cluster is healthy 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.97.106 is healthy 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.107.242 cluster is healthy 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.107.242 is healthy 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.107.24 cluster is healthy 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.107.24 is healthy 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2022-04-21 03:14:29 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2022-04-21 03:14:34 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2022-04-21 03:14:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:14:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:14:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:14:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:14:49 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED 2022-04-21 03:14:49 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:14:49 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:172.23.97.106 port:8091 ssh_username:root 2022-04-21 03:14:49 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs. ... 2022-04-21 03:15:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:15:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:15:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:15:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:15:20 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:15:20 | INFO | MainProcess | MainThread | [rest_client.urllib_request] Making a rest request api=https://172.23.97.106:19102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False 2022-04-21 03:15:20 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2022-04-21 03:15:20 | INFO | MainProcess | MainThread | [rest_client.urllib_request] Making a rest request api=https://172.23.97.106:19102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False 2022-04-21 03:15:20 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2022-04-21 03:15:20 | INFO | MainProcess | MainThread | [collections_index_basics.setUp] ============== CollectionsIndexBasics setup has started ============== 2022-04-21 03:15:21 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] https://172.23.97.145:18091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=1170&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-04-21 03:15:21 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.05 seconds to create bucket test_bucket 2022-04-21 03:15:21 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 172.23.97.145 to accept set ops 2022-04-21 03:15:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.97.106:11207 test_bucket 2022-04-21 03:15:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.97.145:11207 test_bucket 2022-04-21 03:15:23 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.97.106:11207 test_bucket 2022-04-21 03:15:23 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.97.145:11207 test_bucket 2022-04-21 03:15:24 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'test_bucket' was created with per node RAM quota: 1170 2022-04-21 03:15:24 | INFO | MainProcess | MainThread | [collections_index_basics.setUp] ============== CollectionsIndexBasics setup has completed ============== 2022-04-21 03:15:24 | INFO | MainProcess | MainThread | [collections_index_basics.tearDown] ============== CollectionsIndexBasics tearDown has started ============== 2022-04-21 03:15:24 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:15:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:15:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:15:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:15:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:15:25 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 03:15:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-04-21 03:15:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:15:25 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:15:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:15:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:15:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ###################### Disabling n2n encryption 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:15:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:15:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:15:27 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2022-04-21 03:15:27 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2022-04-21 03:15:27 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 03:15:27 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2022-04-21 03:15:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:15:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:15:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:15:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:15:28 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned off encryption for node: http://172.23.97.106:8091', 'Turned off encryption for node: http://172.23.97.145:8091', 'SUCCESS: Switched node-to-node encryption off'] 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.97.106:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.259464916708733, 'mem_free': 3106050048, 'mem_total': 3935289344, 'swap_mem_used': 67633152, 'swap_mem_total': 3758092288} 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.97.145:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 7.441154138192863, 'mem_free': 3010564096, 'mem_total': 3935289344, 'swap_mem_used': 3932160, 'swap_mem_total': 3758092288} 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:15:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:15:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:15:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:15:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:15:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:15:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:15:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:15:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:15:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:15:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:15:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:15:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2022-04-21 03:15:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket'] on 172.23.97.145 2022-04-21 03:15:40 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.97.145 existing buckets : [] 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket from 172.23.97.145 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.97.106, nothing to delete 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.242, nothing to delete 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.24, nothing to delete 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-04-21 03:15:41 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE https://172.23.97.145:18091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."' 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.97.106,ns_1@172.23.97.145', 'ejectedNodes': 'ns_1@172.23.97.106', 'user': 'Administrator', 'password': 'password'} 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2022-04-21 03:15:41 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-04-21 03:15:51 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 66.00 % 2022-04-21 03:16:01 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 66.00 % 2022-04-21 03:16:21 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 40.16 seconds 2022-04-21 03:16:21 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.97.145 port:8091 ssh_username:root ? [('ns_1@172.23.97.106', 8091)] 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.145:8091 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.145:8091 is running 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.106:8091 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.106:8091 is running 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.242:8091 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.242:8091 is running 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.24:8091 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.24:8091 is running 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2022-04-21 03:16:31 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-04-21 03:16:32 | INFO | MainProcess | MainThread | [collections_index_basics.tearDown] ============== CollectionsIndexBasics tearDown has completed ============== ./lib/log_scanner.py:14: DeprecationWarning: invalid escape sequence \s "Basic\s[a-zA-Z]\{10,\}==", ./lib/log_scanner.py:15: DeprecationWarning: invalid escape sequence \[ "Menelaus-Auth-User:\[", /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.97.106'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.97.106'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, ok ---------------------------------------------------------------------- Ran 1 test in 307.957s OK test_gsi_indexes_with_WITH_clause (gsi.collections_index_basics.CollectionsIndexBasics) ... -->result: 2022-04-21 03:16:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:16:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:16:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2558-enterprise 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_gsi_indexes_with_WITH_clause============== 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [collections_index_basics.tearDown] ============== CollectionsIndexBasics tearDown has started ============== 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-04-21 03:16:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ###################### Disabling n2n encryption 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2022-04-21 03:16:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:16:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:16:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:16:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:16:35 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2022-04-21 03:16:36 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2022-04-21 03:16:36 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 03:16:36 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2022-04-21 03:16:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:16:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:16:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:16:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:16:37 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2022-04-21 03:16:37 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned off encryption for node: http://172.23.97.145:8091', 'SUCCESS: Switched node-to-node encryption off'] 2022-04-21 03:16:37 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 03:16:37 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 03:16:37 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.97.145:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 2.74075936635655, 'mem_free': 3078754304, 'mem_total': 3935289344, 'swap_mem_used': 3932160, 'swap_mem_total': 3758092288} 2022-04-21 03:16:37 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 03:16:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:16:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:16:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:16:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:16:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:16:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:16:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:16:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:16:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:16:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:16:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:16:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:16:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:16:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:16:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:16:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:16:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_gsi_indexes_with_WITH_clause ============== 2022-04-21 03:16:48 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.97.145, nothing to delete 2022-04-21 03:16:48 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.97.106, nothing to delete 2022-04-21 03:16:48 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.242, nothing to delete 2022-04-21 03:16:48 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.24, nothing to delete 2022-04-21 03:16:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-04-21 03:16:48 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE https://172.23.97.145:18091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 03:16:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-04-21 03:16:48 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:16:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.145:8091 2022-04-21 03:16:48 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.145:8091 is running 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.106:8091 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.106:8091 is running 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.242:8091 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.242:8091 is running 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.24:8091 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.24:8091 is running 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_gsi_indexes_with_WITH_clause ============== 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [collections_index_basics.tearDown] ============== CollectionsIndexBasics tearDown has completed ============== 2022-04-21 03:16:49 | INFO | MainProcess | test_thread | [basetestcase.setUp] Building docker image with java sdk client OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 2022-04-21 03:16:58 | INFO | MainProcess | test_thread | [basetestcase.setUp] #####Enforcing TLS######## 2022-04-21 03:16:58 | INFO | MainProcess | test_thread | [ntonencryptionBase.setup_nton_cluster] Setting up node to node encryption - status = enable and clusterEncryptionLevel = strict 2022-04-21 03:16:58 | INFO | MainProcess | test_thread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 03:16:58 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to enable 2022-04-21 03:16:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:16:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:16:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:16:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:16:59 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --enable 2022-04-21 03:16:59 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned on encryption for node: http://172.23.97.145:8091', 'SUCCESS: Switched node-to-node encryption on'] 2022-04-21 03:16:59 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 03:16:59 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = strict 2022-04-21 03:16:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:16:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:17:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:00 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level strict 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ['SUCCESS: Security settings updated'] 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8091 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8093 2022-04-21 03:17:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:02 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:9102 2022-04-21 03:17:02 | ERROR | MainProcess | test_thread | [cluster_helper.check_if_services_obey_tls] On Server 172.23.97.145 Expected 127.0.0.1:9102 Actual *:9102 !!!!!!!!!!!!!!!! 2022-04-21 03:17:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8091 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8093 2022-04-21 03:17:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:9102 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :11210 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:11210 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :8092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8092 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :9003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:18091 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:18093 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :19102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:19102 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :11207 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:11207 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :18092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:18092 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :19000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :19001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :19002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: ss -4anpe | grep :19003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:17:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:17:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:17:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:15 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.145 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:17:16 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 03:17:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: systemctl stop couchbase-server.service 2022-04-21 03:17:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:18 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:18 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 03:17:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 03:17:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 03:17:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:17:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:17:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:19 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.145 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:17:19 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:19 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 03:17:19 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 03:17:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: systemctl start couchbase-server.service 2022-04-21 03:17:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:19 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.97.145:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 03:17:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 03:17:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:24 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 120120 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 03:17:24 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:24 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.97.145: with pid 120260 2022-04-21 03:17:24 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-04-21 03:17:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:17:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:17:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:17:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:17:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:26 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.106 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:17:26 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 03:17:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: systemctl stop couchbase-server.service 2022-04-21 03:17:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:28 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:28 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 03:17:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 03:17:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 03:17:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:17:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:17:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:29 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.106 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:17:29 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:29 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 03:17:29 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 03:17:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: systemctl start couchbase-server.service 2022-04-21 03:17:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:29 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.97.106:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 03:17:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 03:17:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:34 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 122222 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 03:17:34 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:34 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.97.106: with pid 122362 2022-04-21 03:17:34 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-04-21 03:17:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:17:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:17:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:17:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:17:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:36 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.242 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:17:37 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 03:17:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: systemctl stop couchbase-server.service 2022-04-21 03:17:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:39 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:39 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 03:17:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 03:17:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 03:17:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:17:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:17:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:40 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.242 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:17:40 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:41 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 03:17:41 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 03:17:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: systemctl start couchbase-server.service 2022-04-21 03:17:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:41 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.107.242:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 03:17:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 03:17:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:46 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 70280 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 03:17:46 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:46 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.242: with pid 70423 2022-04-21 03:17:46 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-04-21 03:17:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:17:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:17:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:17:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:17:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:48 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.24 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:17:48 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 03:17:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: systemctl stop couchbase-server.service 2022-04-21 03:17:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:51 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:51 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 03:17:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 03:17:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 03:17:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:17:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:17:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:17:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:17:52 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.24 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 03:17:52 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:52 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 03:17:53 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 03:17:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: systemctl start couchbase-server.service 2022-04-21 03:17:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:53 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.107.24:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 03:17:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 03:17:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:17:58 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 29204 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 03:17:58 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 03:17:58 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.24: with pid 29345 2022-04-21 03:17:58 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-04-21 03:17:58 | ERROR | MainProcess | test_thread | [rest_client._http_request] socket error while connecting to https://172.23.107.24:18091/nodes/self error [Errno 111] Connection refused 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.97.145 port:8091 ssh_username:root, nodes/self 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '40', 'memoryTotal': 3935289344, 'memoryFree': 3246313472, 'mcdMemoryReserved': 3002, 'mcdMemoryAllocated': 3002, 'status': 'healthy', 'hostname': '172.23.97.145:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1585, 'memcached': 11207, 'id': 'ns_1@172.23.97.145', 'ip': '172.23.97.145', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3752, 'curr_items': 0} 2022-04-21 03:18:03 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.97.145:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.97.145 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1755 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.97.145,18091,['kv', 'n1ql', 'index']) 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.97.145: 18091:hostname=172.23.97.145&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.97.145:18091:port=8091&username=Administrator&password=password 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:18:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.97.106 port:8091 ssh_username:root, nodes/self 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '31', 'memoryTotal': 3935289344, 'memoryFree': 3296833536, 'mcdMemoryReserved': 3002, 'mcdMemoryAllocated': 3002, 'status': 'healthy', 'hostname': '172.23.97.106:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1585, 'memcached': 11207, 'id': 'ns_1@172.23.97.106', 'ip': '172.23.97.106', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3752, 'curr_items': 0} 2022-04-21 03:18:04 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.97.106:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2011 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.97.106:18091:port=8091&username=Administrator&password=password 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.106:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.106:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.242 port:8091 ssh_username:root, nodes/self 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '17', 'memoryTotal': 3945775104, 'memoryFree': 3336589312, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.242:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.242', 'ip': '172.23.107.242', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 03:18:05 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.242:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.242:18091:port=8091&username=Administrator&password=password 2022-04-21 03:18:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 03:18:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:18:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:18:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:18:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:18:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.242:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.242:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.24 port:8091 ssh_username:root, nodes/self 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '7', 'memoryTotal': 3945775104, 'memoryFree': 3316105216, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.24:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.24', 'ip': '172.23.107.24', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 03:18:07 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.24:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.24:18091:port=8091&username=Administrator&password=password 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:18:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:18:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:18:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:18:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 03:18:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:18:08 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 03:18:08 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.24:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:18:08 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.24:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 03:18:08 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-04-21 03:18:08 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.97.145 **** 2022-04-21 03:18:08 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE https://172.23.97.145:18091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 03:18:08 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-04-21 03:18:09 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-04-21 03:18:14 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-04-21 03:18:14 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2022-04-21 03:18:14 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.97.106:8091 to cluster 2022-04-21 03:18:14 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.97.106:18091 to this cluster @172.23.97.145:18091 2022-04-21 03:18:24 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.06 seconds 2022-04-21 03:18:24 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 03:18:38 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.97.106 not part of cluster inactiveAdded 2022-04-21 03:18:38 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.97.106,ns_1@172.23.97.145', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-04-21 03:18:38 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-04-21 03:18:48 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 36.00 % 2022-04-21 03:18:48 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 36.00% 2022-04-21 03:19:08 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-04-21 03:19:08 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 30.1200954914093 sec 2022-04-21 03:19:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:19:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:19:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:19:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:19:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 03:19:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:19:09 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 03:19:09 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.97.145 2022-04-21 03:19:09 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_gsi_indexes_with_WITH_clause ============== 2022-04-21 03:19:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:19:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:19:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:19:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:19:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:19:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:19:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:19:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:19:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:19:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:19:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:19:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:19:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:19:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:19:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:19:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.97.106:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.406084988684938, 'mem_free': 3201036288, 'mem_total': 3935289344, 'swap_mem_used': 67633152, 'swap_mem_total': 3758092288} 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.97.145:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.779233870967742, 'mem_free': 3139960832, 'mem_total': 3935289344, 'swap_mem_used': 3932160, 'swap_mem_total': 3758092288} 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 168.00722217559814 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.97.145 cluster is healthy 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.97.145 is healthy 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.97.106 cluster is healthy 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.97.106 is healthy 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.107.242 cluster is healthy 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.107.242 is healthy 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.107.24 cluster is healthy 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.107.24 is healthy 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2022-04-21 03:19:20 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-04-21 03:19:25 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2022-04-21 03:19:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:19:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:19:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:19:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:19:43 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2022-04-21 03:19:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:19:43 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:172.23.97.106 port:8091 ssh_username:root 2022-04-21 03:19:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2022-04-21 03:20:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:20:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:20:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:20:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:20:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:20:14 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.97.106:19102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False 2022-04-21 03:20:14 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2022-04-21 03:20:14 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.97.106:19102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False 2022-04-21 03:20:14 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2022-04-21 03:20:14 | INFO | MainProcess | test_thread | [collections_index_basics.setUp] ============== CollectionsIndexBasics setup has started ============== 2022-04-21 03:20:14 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] https://172.23.97.145:18091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=1170&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-04-21 03:20:14 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.05 seconds to create bucket test_bucket 2022-04-21 03:20:14 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 172.23.97.145 to accept set ops 2022-04-21 03:20:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.97.106:11207 test_bucket 2022-04-21 03:20:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.97.145:11207 test_bucket 2022-04-21 03:20:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.97.106:11207 test_bucket 2022-04-21 03:20:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.97.145:11207 test_bucket 2022-04-21 03:20:17 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'test_bucket' was created with per node RAM quota: 1170 2022-04-21 03:20:17 | INFO | MainProcess | test_thread | [collections_index_basics.setUp] ============== CollectionsIndexBasics setup has completed ============== 2022-04-21 03:20:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:20:17 | INFO | MainProcess | test_thread | [rest_client.create_scope] https://172.23.97.145:18091/pools/default/buckets/test_bucket/scopes with params: name=test_scope_1 2022-04-21 03:20:17 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created test_bucket->test_scope_1 {'uid': '1'} 2022-04-21 03:20:17 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] https://172.23.97.145:18091/pools/default/buckets/test_bucket/scopes/test_scope_1/collections with params: name=test_collection_1 2022-04-21 03:20:17 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created test_bucket->test_scope_1->test_collection_1 manifest:{'uid': '2'} 2022-04-21 03:20:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. Allowing time after collection creation ... 2022-04-21 03:20:27 | INFO | MainProcess | test_thread | [basetestcase.data_ops_javasdk_loader_in_batches] Number of batches : 1 2022-04-21 03:20:27 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2022-04-21 03:20:37 | INFO | MainProcess | test_thread | [basetestcase.data_ops_javasdk_loader_in_batches] done 2022-04-21 03:20:38 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.97.145 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False --secure True 2022-04-21 03:20:46 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx` ON default:test_bucket.test_scope_1.test_collection_1(age) USING GSI WITH {'defer_build': True, 'num_replica': 1} 2022-04-21 03:20:46 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28age%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%2C+%27num_replica%27%3A+1%7D 2022-04-21 03:20:46 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 114.999221ms 2022-04-21 03:20:46 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX ON default:test_bucket.test_scope_1.test_collection_1 (idx) USING GSI 2022-04-21 03:20:46 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+ON+default%3Atest_bucket.test_scope_1.test_collection_1+%28idx%29+USING+GSI 2022-04-21 03:20:46 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 222.429078ms 2022-04-21 03:20:46 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.97.145:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 03:20:47 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.97.145:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 03:20:48 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.97.145:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 03:20:49 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.97.145:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 03:20:50 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.97.145:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 03:20:51 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.97.145:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 03:20:51 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2022-04-21 03:21:01 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT COUNT(*) FROM default:test_bucket.test_scope_1.test_collection_1 WHERE age > 65 2022-04-21 03:21:01 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+COUNT%28%2A%29+FROM+default%3Atest_bucket.test_scope_1.test_collection_1+WHERE+age+%3E+65 2022-04-21 03:21:01 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.471755ms 2022-04-21 03:21:01 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx ON default:test_bucket.test_scope_1.test_collection_1 USING GSI 2022-04-21 03:21:01 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx+ON+default%3Atest_bucket.test_scope_1.test_collection_1+USING+GSI 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 54.594299ms 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx` ON default:test_bucket.test_scope_1.test_collection_1(age) USING GSI WITH {'nodes': ['172.23.97.106:18091', '172.23.97.145:18091'], 'defer_build': True} 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28age%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27172.23.97.106%3A18091%27%2C+%27172.23.97.145%3A18091%27%5D%2C+%27defer_build%27%3A+True%7D 2022-04-21 03:21:02 | ERROR | MainProcess | test_thread | [rest_client._http_request] POST https://172.23.97.145:18093/query?statement=CREATE+INDEX+%60idx%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28age%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27172.23.97.106%3A18091%27%2C+%27172.23.97.145%3A18091%27%5D%2C+%27defer_build%27%3A+True%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "28df8c7a-c6a6-4f65-bdb4-deb27ac3d2cb",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: Indexer node (172.23.97.106:18091) not found. The node may be failed or under rebalance or network partitioned from query process."}],\n"status": "errors",\n"metrics": {"elapsedTime": "1.467686ms","executionTime": "1.403428ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx ON default:test_bucket.test_scope_1.test_collection_1 USING GSI 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx+ON+default%3Atest_bucket.test_scope_1.test_collection_1+USING+GSI 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [collections_index_basics.tearDown] ============== CollectionsIndexBasics tearDown has started ============== 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-04-21 03:21:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.145:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.97.106 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ###################### Disabling n2n encryption 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:21:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:21:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:21:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:21:04 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2022-04-21 03:21:05 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2022-04-21 03:21:05 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 03:21:05 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2022-04-21 03:21:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:21:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:21:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:21:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:21:05 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2022-04-21 03:21:06 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned off encryption for node: http://172.23.97.106:8091', 'Turned off encryption for node: http://172.23.97.145:8091', 'SUCCESS: Switched node-to-node encryption off'] 2022-04-21 03:21:06 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 03:21:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 03:21:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.97.106:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.52652005174644, 'mem_free': 2924306432, 'mem_total': 3935289344, 'swap_mem_used': 67633152, 'swap_mem_total': 3758092288} 2022-04-21 03:21:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.97.145:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.70080373347161, 'mem_free': 2860523520, 'mem_total': 3935289344, 'swap_mem_used': 3932160, 'swap_mem_total': 3758092288} 2022-04-21 03:21:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 03:21:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:21:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:21:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:21:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:21:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:21:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:21:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:21:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:21:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:21:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:21:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:21:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:21:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:21:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:21:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:21:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 03:21:25 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_gsi_indexes_with_WITH_clause ============== 2022-04-21 03:21:25 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket'] on 172.23.97.145 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.97.145 existing buckets : [] 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket from 172.23.97.145 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.97.106, nothing to delete 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.242, nothing to delete 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.24, nothing to delete 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-04-21 03:21:27 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE https://172.23.97.145:18091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.97.106,ns_1@172.23.97.145', 'ejectedNodes': 'ns_1@172.23.97.106', 'user': 'Administrator', 'password': 'password'} 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2022-04-21 03:21:27 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-04-21 03:21:37 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 66.00 % 2022-04-21 03:21:47 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 66.00 % 2022-04-21 03:22:07 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 40.16 seconds 2022-04-21 03:22:07 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.97.145 port:8091 ssh_username:root ? [('ns_1@172.23.97.106', 8091)] 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.145:8091 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.145:8091 is running 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.106:8091 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.106:8091 is running 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.242:8091 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.242:8091 is running 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.24:8091 2022-04-21 03:22:17 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 03:22:18 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.24:8091 is running 2022-04-21 03:22:18 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_gsi_indexes_with_WITH_clause ============== 2022-04-21 03:22:18 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-04-21 03:22:18 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-04-21 03:22:18 | INFO | MainProcess | test_thread | [collections_index_basics.tearDown] ============== CollectionsIndexBasics tearDown has completed ============== 2022-04-21 03:22:18 | INFO | MainProcess | Thread-271 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 2022-04-21 03:22:18 | INFO | MainProcess | Thread-272 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.97.106 with username:root, attempt#1 of 5 2022-04-21 03:22:18 | INFO | MainProcess | Thread-273 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.242 with username:root, attempt#1 of 5 2022-04-21 03:22:18 | INFO | MainProcess | Thread-276 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 2022-04-21 03:22:18 | INFO | MainProcess | Thread-271 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.145 as root 2022-04-21 03:22:18 | INFO | MainProcess | Thread-272 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.97.106 as root 2022-04-21 03:22:18 | INFO | MainProcess | Thread-273 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.242 as root 2022-04-21 03:22:18 | INFO | MainProcess | Thread-276 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.24 as root 2022-04-21 03:22:18 | INFO | MainProcess | Thread-271 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:22:18 | INFO | MainProcess | Thread-272 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:22:18 | INFO | MainProcess | Thread-273 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:22:18 | INFO | MainProcess | Thread-276 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 03:22:18 | INFO | MainProcess | Thread-271 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.97.145 2022-04-21 03:22:18 | INFO | MainProcess | Thread-271 | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: /opt/couchbase/bin/cbcollect_info 172.23.97.145-20220421-0322-diag.zip 2022-04-21 03:22:18 | INFO | MainProcess | Thread-272 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.97.106 2022-04-21 03:22:18 | INFO | MainProcess | Thread-272 | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: /opt/couchbase/bin/cbcollect_info 172.23.97.106-20220421-0322-diag.zip 2022-04-21 03:22:19 | INFO | MainProcess | Thread-273 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.242 2022-04-21 03:22:19 | INFO | MainProcess | Thread-273 | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: /opt/couchbase/bin/cbcollect_info 172.23.107.242-20220421-0322-diag.zip 2022-04-21 03:22:19 | INFO | MainProcess | Thread-276 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.24 2022-04-21 03:22:19 | INFO | MainProcess | Thread-276 | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: /opt/couchbase/bin/cbcollect_info 172.23.107.24-20220421-0322-diag.zip 2022-04-21 03:24:11 | INFO | MainProcess | Thread-272 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.97.106 2022-04-21 03:24:11 | INFO | MainProcess | Thread-272 | [remote_util.execute_command_raw] running command.raw on 172.23.97.106: rm -f /root/172.23.97.106-20220421-0322-diag.zip 2022-04-21 03:24:11 | INFO | MainProcess | Thread-272 | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:24:17 | INFO | MainProcess | Thread-276 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.107.24 2022-04-21 03:24:18 | INFO | MainProcess | Thread-276 | [remote_util.execute_command_raw] running command.raw on 172.23.107.24: rm -f /root/172.23.107.24-20220421-0322-diag.zip 2022-04-21 03:24:18 | INFO | MainProcess | Thread-276 | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:24:42 | INFO | MainProcess | Thread-273 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.107.242 2022-04-21 03:24:42 | INFO | MainProcess | Thread-273 | [remote_util.execute_command_raw] running command.raw on 172.23.107.242: rm -f /root/172.23.107.242-20220421-0322-diag.zip 2022-04-21 03:24:42 | INFO | MainProcess | Thread-273 | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 03:25:12 | INFO | MainProcess | Thread-271 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.97.145 2022-04-21 03:25:12 | INFO | MainProcess | Thread-271 | [remote_util.execute_command_raw] running command.raw on 172.23.97.145: rm -f /root/172.23.97.145-20220421-0322-diag.zip 2022-04-21 03:25:12 | INFO | MainProcess | Thread-271 | [remote_util.execute_command_raw] command executed successfully with root summary so far suite gsi.collections_index_basics.CollectionsIndexBasics , pass 0 , fail 1 failures so far... gsi.collections_index_basics.CollectionsIndexBasics.test_gsi_indexes_with_WITH_clause testrunner logs, diags and results are available under /data/workspace/centos-p0-2i-vset00-00-collections_gsi_cbd_sanity_7.0_P1/logs/testrunner-22-Apr-21_03-11-23/test_1 /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.97.106'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.97.106'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.97.145'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.97.145'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.97.145'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.97.145'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.97.145'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.97.145'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, ERROR ====================================================================== ERROR: test_gsi_indexes_with_WITH_clause (gsi.collections_index_basics.CollectionsIndexBasics) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/gsi/collections_index_basics.py", line 329, in test_gsi_indexes_with_WITH_clause self.run_cbq_query(query=query) File "lib/couchbase_helper/tuq_helper.py", line 347, in run_cbq_query raise CBQError(error_result, server.ip) membase.api.exception.CBQError: CBQError: host 172.23.97.145: ERROR:{'requestID': '28df8c7a-c6a6-4f65-bdb4-deb27ac3d2cb', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI CreateIndex() - cause: Indexer node (172.23.97.106:18091) not found. The node may be failed or under rebalance or network partitioned from query process.'}], 'status': 'errors', 'metrics': {'elapsedTime': '1.467686ms', 'executionTime': '1.403428ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6, 'errorCount': 1}} During handling of the above exception, another exception occurred: Traceback (most recent call last): File "pytests/gsi/collections_index_basics.py", line 341, in test_gsi_indexes_with_WITH_clause self.fail(str(err)) File "/usr/local/lib/python3.7/unittest/case.py", line 693, in fail raise self.failureException(msg) AssertionError: CBQError: host 172.23.97.145: ERROR:{'requestID': '28df8c7a-c6a6-4f65-bdb4-deb27ac3d2cb', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI CreateIndex() - cause: Indexer node (172.23.97.106:18091) not found. The node may be failed or under rebalance or network partitioned from query process.'}], 'status': 'errors', 'metrics': {'elapsedTime': '1.467686ms', 'executionTime': '1.403428ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6, 'errorCount': 1}} During handling of the above exception, another exception occurred: Traceback (most recent call last): File "pytests/gsi/collections_index_basics.py", line 344, in test_gsi_indexes_with_WITH_clause self.run_cbq_query(query=query) File "lib/couchbase_helper/tuq_helper.py", line 347, in run_cbq_query raise CBQError(error_result, server.ip) membase.api.exception.CBQError: CBQError: host 172.23.97.145: ERROR:{'requestID': '8e5b3735-e674-44c9-8a26-0c2c8dcbf6ce', 'signature': None, 'results': [], 'errors': [{'code': 12016, 'msg': 'Index Not Found - cause: GSI index idx not found.', 'reason': {'name': 'idx'}}], 'status': 'errors', 'metrics': {'elapsedTime': '616.201µs', 'executionTime': '557.782µs', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6, 'errorCount': 1}} ---------------------------------------------------------------------- Ran 1 test in 345.751s FAILED (errors=1)