./testrunner -i /tmp/testexec.10592.ini -p gsi_type=memory_optimized,get-cbcollect-info=True,reset_services=True,java_sdk_client=True,GROUP=P0,index_quota_percent=50,enforce_tls=True,use_https=True -t gsi.collections_alter_index.CollectionsAlterIndex.test_alter_index_for_collections,default_bucket=false,force_clean=true,defer_build=False,nodes_init=3,services_init=kv:n1ql:index-index-index,GROUP=P0,gsi_type=memory_optimized,get-cbcollect-info=True,reset_services=True,java_sdk_client=True,index_quota_percent=50,enforce_tls=True,use_https=True,total_testcases=1 Test Input params: {'default_bucket': 'false', 'force_clean': 'true', 'defer_build': 'False', 'nodes_init': '3', 'services_init': 'kv:n1ql:index-index-index', 'GROUP': 'P0', 'gsi_type': 'memory_optimized', 'get-cbcollect-info': 'True', 'reset_services': 'True', 'java_sdk_client': 'True', 'index_quota_percent': '50', 'enforce_tls': 'True', 'use_https': 'True', 'total_testcases': 1, 'ini': '/tmp/testexec.10592.ini', 'cluster_name': 'testexec.10592', 'spec': 'collection-alter-index', 'conf_file': 'gsi/collection-alter-index.conf', 'num_nodes': 4, 'case_number': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/data/workspace/centos-p0-2i-vset00-00-collections_gsi_alter_index_moi_7.0_P0/logs/testrunner-22-Apr-21_23-34-17/test_1'} Run before suite setup for gsi.collections_alter_index.CollectionsAlterIndex.test_alter_index_for_collections lib/couchbase_helper/tuq_generators.py:113: DeprecationWarning: invalid escape sequence \w regex = re.compile("[\w']+\.[\w']+") lib/couchbase_helper/tuq_generators.py:363: DeprecationWarning: invalid escape sequence \[ diff = set(order_clause.split(',')) - set(re.compile('doc\["[\w\']+"\]').findall(select_clause)) pytests/basetestcase.py:3161: DeprecationWarning: invalid escape sequence \ copy_servers, root_cn='Root\ Authority', type="openssl", lib/couchbase_helper/data_analysis_helper.py:905: DeprecationWarning: invalid escape sequence \{ master_key = ".\{12\}$" ./pytests/security/x509main.py:80: DeprecationWarning: invalid escape sequence \. ''' suite_setUp (gsi.collections_alter_index.CollectionsAlterIndex) ... -->before_suite_name:gsi.collections_alter_index.CollectionsAlterIndex.suite_setUp,suite: ]> 2022-04-21 23:34:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:18 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:34:18 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2558-enterprise 2022-04-21 23:34:18 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2022-04-21 23:34:18 | INFO | MainProcess | MainThread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has started ============== 2022-04-21 23:34:18 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:34:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:19 | ERROR | MainProcess | MainThread | [rest_client._http_request] POST https://172.23.106.184:18091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2022-04-21 23:34:19 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:34:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" API is accessible from localhost only/indexer.log* | wc -l 2022-04-21 23:34:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/indexer.log*.gz: No such file or directory'] ... 2022-04-21 23:34:19 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:34:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:20 | ERROR | MainProcess | MainThread | [rest_client._http_request] POST https://172.23.106.184:18091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2022-04-21 23:34:20 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:34:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" API is accessible from localhost only/projector.log* | wc -l 2022-04-21 23:34:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/projector.log*.gz: No such file or directory'] ... 2022-04-21 23:34:20 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ###################### Disabling n2n encryption 2022-04-21 23:34:20 | INFO | MainProcess | MainThread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2022-04-21 23:34:21 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 23:34:21 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2022-04-21 23:34:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:22 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2022-04-21 23:34:22 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2022-04-21 23:34:22 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 23:34:22 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2022-04-21 23:34:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:24 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2022-04-21 23:34:24 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned off encryption for node: http://[::1]:8091', 'SUCCESS: Switched node-to-node encryption off'] 2022-04-21 23:34:24 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 23:34:24 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 23:34:24 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.184:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 1.111391765597373, 'mem_free': 3237113856, 'mem_total': 3945775104, 'swap_mem_used': 68943872, 'swap_mem_total': 3758092288} 2022-04-21 23:34:24 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 23:34:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:34:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:34:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:34:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:34:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:34:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:34:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2022-04-21 23:34:38 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.184, nothing to delete 2022-04-21 23:34:38 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.78, nothing to delete 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.58, nothing to delete 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.84, nothing to delete 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-04-21 23:34:39 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE https://172.23.106.184:18091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."' 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.184:8091 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.184:8091 is running 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.78:8091 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.78:8091 is running 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.58:8091 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.58:8091 is running 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.84:8091 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.84:8091 is running 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has completed ============== 2022-04-21 23:34:39 | INFO | MainProcess | MainThread | [basetestcase.setUp] Building docker image with java sdk client Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 2022-04-21 23:34:43 | INFO | MainProcess | MainThread | [basetestcase.setUp] #####Enforcing TLS######## 2022-04-21 23:34:43 | INFO | MainProcess | MainThread | [ntonencryptionBase.setup_nton_cluster] Setting up node to node encryption - status = enable and clusterEncryptionLevel = strict 2022-04-21 23:34:43 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 23:34:43 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to enable 2022-04-21 23:34:43 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:43 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:44 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:44 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:44 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --enable 2022-04-21 23:34:45 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned on encryption for node: http://[::1]:8091', 'SUCCESS: Switched node-to-node encryption on'] 2022-04-21 23:34:45 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 23:34:45 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = strict 2022-04-21 23:34:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:46 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level strict 2022-04-21 23:34:47 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ['SUCCESS: Security settings updated'] 2022-04-21 23:34:47 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 23:34:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:8091 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:8093 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:9102 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :11210 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:11210 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:8092 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:18091 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:18093 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:19102 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :11207 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:11207 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:18092 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:52 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.184 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:34:52 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:34:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: systemctl stop couchbase-server.service 2022-04-21 23:34:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:55 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:34:55 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:34:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:34:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:34:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:34:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:34:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:34:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:34:56 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.184 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:34:56 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:34:56 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:34:57 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:34:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: systemctl start couchbase-server.service 2022-04-21 23:34:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:34:57 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.106.184:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:35:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:35:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:02 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 126955 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:35:02 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:35:02 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.184: with pid 127095 2022-04-21 23:35:02 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 23:35:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:35:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:35:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:35:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:35:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:04 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.78 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:35:04 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:35:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: systemctl stop couchbase-server.service 2022-04-21 23:35:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:07 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:35:07 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:35:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:35:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:35:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:35:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:35:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:08 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.78 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:35:08 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:35:08 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:35:08 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:35:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: systemctl start couchbase-server.service 2022-04-21 23:35:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:08 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.107.78:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:35:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:35:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:13 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 101361 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:35:13 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:35:13 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.78: with pid 101502 2022-04-21 23:35:13 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 23:35:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:35:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:35:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:35:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:35:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:15 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.58 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:35:16 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:35:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: systemctl stop couchbase-server.service 2022-04-21 23:35:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:18 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:35:19 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:35:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:35:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:35:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:35:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:35:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:20 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.58 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:35:20 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:35:20 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:35:20 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:35:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: systemctl start couchbase-server.service 2022-04-21 23:35:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:20 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.107.58:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:35:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:35:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:25 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 99239 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:35:25 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:35:25 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.58: with pid 99379 2022-04-21 23:35:25 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 23:35:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:35:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:35:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:35:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:35:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:27 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.84 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:35:27 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:35:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: systemctl stop couchbase-server.service 2022-04-21 23:35:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:29 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:35:29 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:35:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:35:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:35:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:35:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:35:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:31 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.84 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:35:31 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:35:31 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:35:31 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:35:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: systemctl start couchbase-server.service 2022-04-21 23:35:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:31 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.107.84:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:35:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:35:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:36 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 116689 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:35:36 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:35:36 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.84: with pid 116829 2022-04-21 23:35:36 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 23:35:37 | ERROR | MainProcess | MainThread | [rest_client._http_request] socket error while connecting to https://172.23.106.184:18091/node/controller/rename error [Errno 111] Connection refused 2022-04-21 23:35:40 | ERROR | MainProcess | MainThread | [rest_client._http_request] socket error while connecting to https://172.23.107.78:18091/node/controller/rename error [Errno 111] Connection refused 2022-04-21 23:35:44 | ERROR | MainProcess | MainThread | [rest_client._http_request] socket error while connecting to https://172.23.107.58:18091/node/controller/rename error [Errno 111] Connection refused 2022-04-21 23:35:48 | ERROR | MainProcess | MainThread | [rest_client._http_request] socket error while connecting to https://172.23.107.84:18091/node/controller/rename error [Errno 111] Connection refused 2022-04-21 23:35:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.184 port:8091 ssh_username:root, nodes/self 2022-04-21 23:35:51 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '53', 'memoryTotal': 3945775104, 'memoryFree': 3301384192, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.184:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.106.184', 'ip': '172.23.106.184', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:35:51 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:35:51 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.106.184:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:35:51 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 1008 MB 2022-04-21 23:35:51 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.106.184 2022-04-21 23:35:51 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:35:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1008 2022-04-21 23:35:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.106.184,18091,['kv', 'n1ql', 'index']) 2022-04-21 23:35:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.106.184: 18091:hostname=172.23.106.184&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex 2022-04-21 23:35:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:35:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.184:18091:port=8091&username=Administrator&password=password 2022-04-21 23:35:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:35:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:35:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:35:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.78 port:8091 ssh_username:root, nodes/self 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '38', 'memoryTotal': 3945775104, 'memoryFree': 3324932096, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.78:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.78', 'ip': '172.23.107.78', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:35:53 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.78:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.78:18091:port=8091&username=Administrator&password=password 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:35:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.78:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.58 port:8091 ssh_username:root, nodes/self 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '28', 'memoryTotal': 3945775104, 'memoryFree': 3283013632, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.58:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.58', 'ip': '172.23.107.58', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:35:54 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.58:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.58:18091:port=8091&username=Administrator&password=password 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:35:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:35:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:35:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.58:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.84 port:8091 ssh_username:root, nodes/self 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '18', 'memoryTotal': 3945775104, 'memoryFree': 3305627648, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.84:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.84', 'ip': '172.23.107.84', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:35:56 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.84:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.84:18091:port=8091&username=Administrator&password=password 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:35:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:35:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:35:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:35:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:35:57 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:35:57 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.84:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:35:57 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:35:57 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.106.184 **** 2022-04-21 23:35:57 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE https://172.23.106.184:18091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 23:35:57 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-04-21 23:35:57 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2022-04-21 23:36:02 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-04-21 23:36:02 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2022-04-21 23:36:03 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.78:8091 to cluster 2022-04-21 23:36:03 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.78:18091 to this cluster @172.23.106.184:18091 2022-04-21 23:36:13 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.06 seconds 2022-04-21 23:36:13 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 23:36:28 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.58:8091 to cluster 2022-04-21 23:36:28 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.58:18091 to this cluster @172.23.106.184:18091 2022-04-21 23:36:38 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.05 seconds 2022-04-21 23:36:38 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 23:36:52 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.58 not part of cluster inactiveAdded 2022-04-21 23:36:52 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.78 not part of cluster inactiveAdded 2022-04-21 23:36:52 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.58,ns_1@172.23.107.78', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-04-21 23:36:52 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-04-21 23:37:02 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-04-21 23:37:02 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.071475982666016 sec 2022-04-21 23:37:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:37:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:37:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:37:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:37:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:37:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:37:03 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:37:03 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.106.184 2022-04-21 23:37:03 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ============== 2022-04-21 23:37:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:37:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:37:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:37:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:37:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:37:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:37:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:37:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:37:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:37:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:37:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:37:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:37:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:37:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:37:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:37:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.184:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 8.316430020283976, 'mem_free': 3192422400, 'mem_total': 3945775104, 'swap_mem_used': 68943872, 'swap_mem_total': 3758092288} 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.107.58:8091 => {'services': ['index'], 'cpu_utilization': 9.212857504429259, 'mem_free': 3275157504, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.107.78:8091 => {'services': ['index'], 'cpu_utilization': 1.614530776992936, 'mem_free': 3291193344, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 181.1162567138672 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.106.184 cluster is healthy 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.106.184 is healthy 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.107.78 cluster is healthy 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.107.78 is healthy 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.107.58 cluster is healthy 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.107.58 is healthy 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.107.84 cluster is healthy 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.107.84 is healthy 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.58 port:8091 ssh_username:root, ip:172.23.107.78 port:8091 ssh_username:root] 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2022-04-21 23:37:18 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2022-04-21 23:37:23 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2022-04-21 23:37:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:37:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:37:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:37:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:37:32 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED 2022-04-21 23:37:32 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:37:32 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:172.23.106.184 port:8091 ssh_username:root 2022-04-21 23:37:32 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs. ... 2022-04-21 23:38:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:38:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:38:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:38:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:38:03 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.58 port:8091 ssh_username:root, ip:172.23.107.78 port:8091 ssh_username:root] 2022-04-21 23:38:03 | INFO | MainProcess | MainThread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:19102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False 2022-04-21 23:38:03 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2022-04-21 23:38:03 | INFO | MainProcess | MainThread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:19102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False 2022-04-21 23:38:03 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2022-04-21 23:38:03 | INFO | MainProcess | MainThread | [collections_alter_index.setUp] ============== CollectionsIndexBasics setup has started ============== 2022-04-21 23:38:04 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] https://172.23.106.184:18091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=672&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-04-21 23:38:04 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.06 seconds to create bucket test_bucket 2022-04-21 23:38:04 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 172.23.106.184 to accept set ops 2022-04-21 23:38:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.184:11207 test_bucket 2022-04-21 23:38:06 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.184:11207 test_bucket 2022-04-21 23:38:07 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'test_bucket' was created with per node RAM quota: 672 2022-04-21 23:38:07 | INFO | MainProcess | MainThread | [collections_alter_index.setUp] ============== CollectionsIndexBasics setup has completed ============== 2022-04-21 23:38:07 | INFO | MainProcess | MainThread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has started ============== 2022-04-21 23:38:07 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.58 port:8091 ssh_username:root, ip:172.23.107.78 port:8091 ssh_username:root] 2022-04-21 23:38:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:38:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:38:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:38:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:38:08 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:38:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-04-21 23:38:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:38:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:38:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:38:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:38:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:38:09 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.107.58:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:38:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-04-21 23:38:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:38:09 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:38:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:38:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ###################### Disabling n2n encryption 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:38:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:38:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:38:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:38:12 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2022-04-21 23:38:12 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2022-04-21 23:38:12 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 23:38:12 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2022-04-21 23:38:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:38:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:38:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:38:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:38:13 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2022-04-21 23:38:14 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned off encryption for node: http://172.23.106.184:8091', 'Turned off encryption for node: http://172.23.107.58:8091', 'Turned off encryption for node: http://172.23.107.78:8091', 'SUCCESS: Switched node-to-node encryption off'] 2022-04-21 23:38:14 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 23:38:14 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 23:38:14 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.184:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 14.2638036809816, 'mem_free': 3080794112, 'mem_total': 3945775104, 'swap_mem_used': 68943872, 'swap_mem_total': 3758092288} 2022-04-21 23:38:14 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.107.58:8091 => {'services': ['index'], 'cpu_utilization': 1.98992443324937, 'mem_free': 3264798720, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:38:14 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.107.78:8091 => {'services': ['index'], 'cpu_utilization': 3.063291139240506, 'mem_free': 3264843776, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:38:14 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 23:38:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:38:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:38:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:38:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:38:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:38:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:38:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:38:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:38:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:38:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:38:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:38:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:38:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:38:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:38:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:38:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:38:28 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2022-04-21 23:38:28 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket'] on 172.23.106.184 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.106.184 existing buckets : [] 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket from 172.23.106.184 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.78, nothing to delete 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.58, nothing to delete 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.84, nothing to delete 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-04-21 23:38:29 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE https://172.23.106.184:18091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."' 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.58,ns_1@172.23.107.78', 'ejectedNodes': 'ns_1@172.23.107.58,ns_1@172.23.107.78', 'user': 'Administrator', 'password': 'password'} 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2022-04-21 23:38:29 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-04-21 23:38:49 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 20.09 seconds 2022-04-21 23:38:49 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.106.184 port:8091 ssh_username:root ? [('ns_1@172.23.107.58', 8091), ('ns_1@172.23.107.78', 8091)] 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.184:8091 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.184:8091 is running 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.78:8091 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.78:8091 is running 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.58:8091 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.58:8091 is running 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.84:8091 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.84:8091 is running 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-04-21 23:39:00 | INFO | MainProcess | MainThread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has completed ============== ./lib/log_scanner.py:14: DeprecationWarning: invalid escape sequence \s "Basic\s[a-zA-Z]\{10,\}==", ./lib/log_scanner.py:15: DeprecationWarning: invalid escape sequence \[ "Menelaus-Auth-User:\[", /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.106.184'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.106.184'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, ok ---------------------------------------------------------------------- Ran 1 test in 283.259s OK test_alter_index_for_collections (gsi.collections_alter_index.CollectionsAlterIndex) ... -->result: 2022-04-21 23:39:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:01 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:39:01 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2558-enterprise 2022-04-21 23:39:01 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_alter_index_for_collections============== 2022-04-21 23:39:01 | INFO | MainProcess | test_thread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has started ============== 2022-04-21 23:39:01 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:39:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:02 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:39:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-04-21 23:39:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:02 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:39:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:04 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:39:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-04-21 23:39:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:04 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ###################### Disabling n2n encryption 2022-04-21 23:39:04 | INFO | MainProcess | test_thread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2022-04-21 23:39:04 | INFO | MainProcess | test_thread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 23:39:04 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2022-04-21 23:39:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:05 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2022-04-21 23:39:06 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2022-04-21 23:39:06 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 23:39:06 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2022-04-21 23:39:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:07 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2022-04-21 23:39:07 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned off encryption for node: http://172.23.106.184:8091', 'SUCCESS: Switched node-to-node encryption off'] 2022-04-21 23:39:07 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 23:39:07 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 23:39:07 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.184:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 1.616978271854472, 'mem_free': 3131838464, 'mem_total': 3945775104, 'swap_mem_used': 68943872, 'swap_mem_total': 3758092288} 2022-04-21 23:39:07 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 23:39:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:39:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:39:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:39:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:39:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:39:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:39:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_alter_index_for_collections ============== 2022-04-21 23:39:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.184, nothing to delete 2022-04-21 23:39:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.78, nothing to delete 2022-04-21 23:39:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.58, nothing to delete 2022-04-21 23:39:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.84, nothing to delete 2022-04-21 23:39:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-04-21 23:39:22 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE https://172.23.106.184:18091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 23:39:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-04-21 23:39:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.184:8091 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.184:8091 is running 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.78:8091 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.78:8091 is running 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.58:8091 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.58:8091 is running 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.84:8091 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.84:8091 is running 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_alter_index_for_collections ============== 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has completed ============== 2022-04-21 23:39:23 | INFO | MainProcess | test_thread | [basetestcase.setUp] Building docker image with java sdk client Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 2022-04-21 23:39:27 | INFO | MainProcess | test_thread | [basetestcase.setUp] #####Enforcing TLS######## 2022-04-21 23:39:27 | INFO | MainProcess | test_thread | [ntonencryptionBase.setup_nton_cluster] Setting up node to node encryption - status = enable and clusterEncryptionLevel = strict 2022-04-21 23:39:27 | INFO | MainProcess | test_thread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 23:39:27 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to enable 2022-04-21 23:39:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:28 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --enable 2022-04-21 23:39:29 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned on encryption for node: http://172.23.106.184:8091', 'SUCCESS: Switched node-to-node encryption on'] 2022-04-21 23:39:29 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 23:39:29 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = strict 2022-04-21 23:39:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:30 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level strict 2022-04-21 23:39:31 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ['SUCCESS: Security settings updated'] 2022-04-21 23:39:31 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 23:39:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8091 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8093 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:9102 2022-04-21 23:39:32 | ERROR | MainProcess | test_thread | [cluster_helper.check_if_services_obey_tls] On Server 172.23.106.184 Expected 127.0.0.1:9102 Actual *:9102 !!!!!!!!!!!!!!!! 2022-04-21 23:39:32 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2022-04-21 23:39:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8091 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8093 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:9102 2022-04-21 23:39:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :11210 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:11210 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8092 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:18091 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:18093 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:19102 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :11207 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:11207 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:18092 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:39:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:46 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2022-04-21 23:39:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:48 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.184 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:39:48 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:39:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: systemctl stop couchbase-server.service 2022-04-21 23:39:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:51 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:39:51 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:39:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:39:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:39:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:39:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:39:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:52 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.184 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:39:52 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:39:52 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:39:52 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:39:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: systemctl start couchbase-server.service 2022-04-21 23:39:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:52 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.184:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:39:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:39:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:39:57 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 3278 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:39:57 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:39:58 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.184: with pid 3419 2022-04-21 23:39:58 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-04-21 23:39:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:39:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:39:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:39:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:39:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:39:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:39:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:00 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.78 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:40:00 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:40:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: systemctl stop couchbase-server.service 2022-04-21 23:40:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:03 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:40:03 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:40:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:40:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:40:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:40:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:40:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:04 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.78 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:40:04 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:40:04 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:40:04 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:40:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: systemctl start couchbase-server.service 2022-04-21 23:40:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:04 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.107.78:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:40:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:40:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:09 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 107125 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:40:09 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:40:09 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.78: with pid 107267 2022-04-21 23:40:09 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-04-21 23:40:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:40:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:40:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:40:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:40:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:11 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.58 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:40:12 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:40:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: systemctl stop couchbase-server.service 2022-04-21 23:40:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:40:14 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:40:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:40:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:40:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:40:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:40:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:15 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.58 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:40:15 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:40:15 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:40:15 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:40:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: systemctl start couchbase-server.service 2022-04-21 23:40:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:15 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.107.58:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:40:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:40:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:20 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 104844 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:40:20 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:40:20 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.58: with pid 105029 2022-04-21 23:40:20 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-04-21 23:40:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:40:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:40:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:40:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:40:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:23 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.84 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:40:23 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:40:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: systemctl stop couchbase-server.service 2022-04-21 23:40:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:26 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:40:26 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:40:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:40:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:40:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:40:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:40:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:27 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.84 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:40:27 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:40:27 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:40:27 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:40:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: systemctl start couchbase-server.service 2022-04-21 23:40:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:27 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.107.84:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:40:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:40:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:32 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 121581 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:40:32 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:40:32 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.84: with pid 121721 2022-04-21 23:40:32 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-04-21 23:40:33 | ERROR | MainProcess | test_thread | [rest_client._http_request] socket error while connecting to https://172.23.107.84:18091/nodes/self error [Errno 111] Connection refused 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.184 port:8091 ssh_username:root, nodes/self 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '43', 'memoryTotal': 3945775104, 'memoryFree': 3304865792, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.184:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.106.184', 'ip': '172.23.106.184', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:40:39 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.106.184:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 1008 MB 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.106.184 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1008 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.106.184,18091,['kv', 'n1ql', 'index']) 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.106.184: 18091:hostname=172.23.106.184&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.184:18091:port=8091&username=Administrator&password=password 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:40:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:40:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:40:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:40 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:40:40 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:40:40 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:40:40 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.78 port:8091 ssh_username:root, nodes/self 2022-04-21 23:40:40 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '33', 'memoryTotal': 3945775104, 'memoryFree': 3319926784, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.78:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.78', 'ip': '172.23.107.78', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:40:40 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:40:40 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.78:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:40:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 23:40:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:40:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.78:18091:port=8091&username=Administrator&password=password 2022-04-21 23:40:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:40:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:40:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:40:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.78:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.58 port:8091 ssh_username:root, nodes/self 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '22', 'memoryTotal': 3945775104, 'memoryFree': 3306696704, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.58:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.58', 'ip': '172.23.107.58', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:40:42 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.58:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.58:18091:port=8091&username=Administrator&password=password 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:40:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.58:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.84 port:8091 ssh_username:root, nodes/self 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '12', 'memoryTotal': 3945775104, 'memoryFree': 3292364800, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.84:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.84', 'ip': '172.23.107.84', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:40:43 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.84:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:40:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.84:18091:port=8091&username=Administrator&password=password 2022-04-21 23:40:44 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:40:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:40:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:40:44 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:40:44 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:40:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:40:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:40:45 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:40:45 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.84:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:40:45 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:40:45 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.106.184 **** 2022-04-21 23:40:45 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE https://172.23.106.184:18091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 23:40:45 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-04-21 23:40:45 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-04-21 23:40:50 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-04-21 23:40:50 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2022-04-21 23:40:51 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.78:8091 to cluster 2022-04-21 23:40:51 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.78:18091 to this cluster @172.23.106.184:18091 2022-04-21 23:41:01 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.06 seconds 2022-04-21 23:41:01 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 23:41:15 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.58:8091 to cluster 2022-04-21 23:41:15 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.58:18091 to this cluster @172.23.106.184:18091 2022-04-21 23:41:25 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.06 seconds 2022-04-21 23:41:25 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 23:41:40 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.58 not part of cluster inactiveAdded 2022-04-21 23:41:40 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.78 not part of cluster inactiveAdded 2022-04-21 23:41:40 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.58,ns_1@172.23.107.78', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-04-21 23:41:40 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-04-21 23:41:50 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-04-21 23:41:50 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.07052206993103 sec 2022-04-21 23:41:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:41:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:41:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:41:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:41:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:41:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:41:51 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:41:51 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.106.184 2022-04-21 23:41:51 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_alter_index_for_collections ============== 2022-04-21 23:41:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:41:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:41:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:41:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:41:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:41:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:41:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:41:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:41:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:41:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:41:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:41:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:41:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:41:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:41:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:41:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.184:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.17120325616891, 'mem_free': 3190259712, 'mem_total': 3945775104, 'swap_mem_used': 68943872, 'swap_mem_total': 3758092288} 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.107.58:8091 => {'services': ['index'], 'cpu_utilization': 13.06774930220756, 'mem_free': 3299205120, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.107.78:8091 => {'services': ['index'], 'cpu_utilization': 3.511874684183931, 'mem_free': 3293859840, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 185.61160039901733 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.106.184 cluster is healthy 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.106.184 is healthy 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.107.78 cluster is healthy 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.107.78 is healthy 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.107.58 cluster is healthy 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.107.58 is healthy 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.107.84 cluster is healthy 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.107.84 is healthy 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.58 port:8091 ssh_username:root, ip:172.23.107.78 port:8091 ssh_username:root] 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2022-04-21 23:42:06 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-04-21 23:42:11 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2022-04-21 23:42:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:42:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:42:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:42:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:42:20 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2022-04-21 23:42:20 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:42:20 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:172.23.106.184 port:8091 ssh_username:root 2022-04-21 23:42:20 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2022-04-21 23:42:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:42:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:42:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:42:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:42:51 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.58 port:8091 ssh_username:root, ip:172.23.107.78 port:8091 ssh_username:root] 2022-04-21 23:42:51 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:19102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False 2022-04-21 23:42:51 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2022-04-21 23:42:51 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:19102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False 2022-04-21 23:42:51 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2022-04-21 23:42:51 | INFO | MainProcess | test_thread | [collections_alter_index.setUp] ============== CollectionsIndexBasics setup has started ============== 2022-04-21 23:42:52 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] https://172.23.106.184:18091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=672&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-04-21 23:42:52 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.06 seconds to create bucket test_bucket 2022-04-21 23:42:52 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 172.23.106.184 to accept set ops 2022-04-21 23:42:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.184:11207 test_bucket 2022-04-21 23:42:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.184:11207 test_bucket 2022-04-21 23:42:54 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'test_bucket' was created with per node RAM quota: 672 2022-04-21 23:42:54 | INFO | MainProcess | test_thread | [collections_alter_index.setUp] ============== CollectionsIndexBasics setup has completed ============== 2022-04-21 23:42:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.58 port:8091 ssh_username:root, ip:172.23.107.78 port:8091 ssh_username:root] 2022-04-21 23:42:54 | INFO | MainProcess | test_thread | [rest_client.create_scope] https://172.23.106.184:18091/pools/default/buckets/test_bucket/scopes with params: name=test_scope_1 2022-04-21 23:42:54 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created test_bucket->test_scope_1 {'uid': '1'} 2022-04-21 23:42:54 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] https://172.23.106.184:18091/pools/default/buckets/test_bucket/scopes/test_scope_1/collections with params: name=test_collection_1 2022-04-21 23:42:54 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created test_bucket->test_scope_1->test_collection_1 manifest:{'uid': '2'} 2022-04-21 23:42:54 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. Allowing time after collection creation ... 2022-04-21 23:43:04 | INFO | MainProcess | test_thread | [basetestcase.data_ops_javasdk_loader_in_batches] Number of batches : 1 2022-04-21 23:43:04 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2022-04-21 23:43:14 | INFO | MainProcess | test_thread | [basetestcase.data_ops_javasdk_loader_in_batches] done 2022-04-21 23:43:15 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.106.184 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False --secure True 2022-04-21 23:43:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx1` ON default:test_bucket.test_scope_1.test_collection_1(age) USING GSI WITH {'num_replica': 1} 2022-04-21 23:43:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx1%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28age%29+USING+GSI++WITH+%7B%27num_replica%27%3A+1%7D 2022-04-21 23:43:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.870901144s 2022-04-21 23:43:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx2` ON default:test_bucket.test_scope_1.test_collection_1(city) USING GSI WITH {'num_replica': 1} 2022-04-21 23:43:22 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx2%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28city%29+USING+GSI++WITH+%7B%27num_replica%27%3A+1%7D 2022-04-21 23:43:25 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.438748286s 2022-04-21 23:43:25 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx3` ON default:test_bucket.test_scope_1.test_collection_1(country) USING GSI WITH {'num_replica': 1} 2022-04-21 23:43:25 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx3%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28country%29+USING+GSI++WITH+%7B%27num_replica%27%3A+1%7D 2022-04-21 23:43:29 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.759867492s 2022-04-21 23:43:29 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 23:43:30 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 23:43:31 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 23:43:32 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 23:43:33 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 23:43:33 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:18091/indexStatus verb=GET params= client_cert=None verify=False 2022-04-21 23:43:33 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Giving some time before trying alter index ... 2022-04-21 23:44:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY ALTER INDEX idx2 on default:test_bucket.test_scope_1.test_collection_1 WITH {"action": "move", "nodes":['172.23.107.78:18091', '172.23.107.58:18091']} 2022-04-21 23:44:03 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=ALTER+INDEX+idx2+on+default%3Atest_bucket.test_scope_1.test_collection_1+WITH+%7B%22action%22%3A+%22move%22%2C+%22nodes%22%3A%5B%27172.23.107.78%3A18091%27%2C+%27172.23.107.58%3A18091%27%5D%7D 2022-04-21 23:44:03 | ERROR | MainProcess | test_thread | [rest_client._http_request] POST https://172.23.106.184:18093/query?statement=ALTER+INDEX+idx2+on+default%3Atest_bucket.test_scope_1.test_collection_1+WITH+%7B%22action%22%3A+%22move%22%2C+%22nodes%22%3A%5B%27172.23.107.78%3A18091%27%2C+%27172.23.107.58%3A18091%27%5D%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "1f8e27f3-b99d-40ce-8a6a-66136b1cdded",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI AlterIndex() - cause: Unable to find Index service for destination 172.23.107.78:18091 or desintation is not part of the cluster"}],\n"status": "errors",\n"metrics": {"elapsedTime": "88.24973ms","executionTime": "88.02468ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password 2022-04-21 23:44:03 | ERROR | MainProcess | test_thread | [base_gsi.alter_index_replicas] CBQError: host 172.23.106.184: ERROR:{'requestID': '1f8e27f3-b99d-40ce-8a6a-66136b1cdded', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI AlterIndex() - cause: Unable to find Index service for destination 172.23.107.78:18091 or desintation is not part of the cluster'}], 'status': 'errors', 'metrics': {'elapsedTime': '88.24973ms', 'executionTime': '88.02468ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6, 'errorCount': 1}} 2022-04-21 23:44:03 | INFO | MainProcess | test_thread | [base_gsi.alter_index_replicas] Error : ["CBQError: host 172.23.106.184: ERROR:{'requestID': '1f8e27f3-b99d-40ce-8a6a-66136b1cdded', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI AlterIndex() - cause: Unable to find Index service for destination 172.23.107.78:18091 or desintation is not part of the cluster'}], 'status': 'errors', 'metrics': {'elapsedTime': '88.24973ms', 'executionTime': '88.02468ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6, 'errorCount': 1}}"] 2022-04-21 23:44:03 | INFO | MainProcess | test_thread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has started ============== 2022-04-21 23:44:03 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.58 port:8091 ssh_username:root, ip:172.23.107.78 port:8091 ssh_username:root] 2022-04-21 23:44:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:44:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:44:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:44:04 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:44:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-04-21 23:44:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:44:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:44:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:44:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:44:06 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.58:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:44:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-04-21 23:44:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:44:06 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:44:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:44:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:44:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ###################### Disabling n2n encryption 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:44:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:44:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:44:08 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2022-04-21 23:44:09 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2022-04-21 23:44:09 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 23:44:09 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2022-04-21 23:44:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:44:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:44:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:44:10 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2022-04-21 23:44:11 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned off encryption for node: http://172.23.106.184:8091', 'Turned off encryption for node: http://172.23.107.58:8091', 'Turned off encryption for node: http://172.23.107.78:8091', 'SUCCESS: Switched node-to-node encryption off'] 2022-04-21 23:44:11 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 23:44:11 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 23:44:11 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.184:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 9.179737721779377, 'mem_free': 2806767616, 'mem_total': 3945775104, 'swap_mem_used': 68943872, 'swap_mem_total': 3758092288} 2022-04-21 23:44:11 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.107.58:8091 => {'services': ['index'], 'cpu_utilization': 1.598579040852576, 'mem_free': 3143921664, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:44:11 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.107.78:8091 => {'services': ['index'], 'cpu_utilization': 1.672156067899671, 'mem_free': 3138068480, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:44:11 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 23:44:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:44:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:44:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:44:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:44:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:44:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:44:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:44:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:44:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:44:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:44:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:44:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:44:26 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_alter_index_for_collections ============== 2022-04-21 23:44:26 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket'] on 172.23.106.184 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.184 existing buckets : [] 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket from 172.23.106.184 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.78, nothing to delete 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.58, nothing to delete 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.84, nothing to delete 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-04-21 23:44:27 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE https://172.23.106.184:18091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.58,ns_1@172.23.107.78', 'ejectedNodes': 'ns_1@172.23.107.58,ns_1@172.23.107.78', 'user': 'Administrator', 'password': 'password'} 2022-04-21 23:44:27 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2022-04-21 23:44:28 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-04-21 23:44:48 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 20.09 seconds 2022-04-21 23:44:48 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.106.184 port:8091 ssh_username:root ? [('ns_1@172.23.107.58', 8091), ('ns_1@172.23.107.78', 8091)] 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.184:8091 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.184:8091 is running 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.78:8091 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.78:8091 is running 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.58:8091 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.58:8091 is running 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.84:8091 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.84:8091 is running 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_alter_index_for_collections ============== 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-04-21 23:44:58 | INFO | MainProcess | test_thread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has completed ============== 2022-04-21 23:44:58 | INFO | MainProcess | Thread-274 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:44:58 | INFO | MainProcess | Thread-275 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:44:58 | INFO | MainProcess | Thread-277 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:44:58 | INFO | MainProcess | Thread-276 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:44:58 | INFO | MainProcess | Thread-276 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:44:58 | INFO | MainProcess | Thread-277 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:44:58 | INFO | MainProcess | Thread-274 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:44:58 | INFO | MainProcess | Thread-275 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:44:58 | INFO | MainProcess | Thread-277 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:59 | INFO | MainProcess | Thread-274 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:59 | INFO | MainProcess | Thread-275 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:59 | INFO | MainProcess | Thread-276 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:44:59 | INFO | MainProcess | Thread-277 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.84 2022-04-21 23:44:59 | INFO | MainProcess | Thread-277 | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: /opt/couchbase/bin/cbcollect_info 172.23.107.84-20220421-2344-diag.zip 2022-04-21 23:44:59 | INFO | MainProcess | Thread-274 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.106.184 2022-04-21 23:44:59 | INFO | MainProcess | Thread-274 | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: /opt/couchbase/bin/cbcollect_info 172.23.106.184-20220421-2344-diag.zip 2022-04-21 23:44:59 | INFO | MainProcess | Thread-275 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.78 2022-04-21 23:44:59 | INFO | MainProcess | Thread-275 | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: /opt/couchbase/bin/cbcollect_info 172.23.107.78-20220421-2344-diag.zip 2022-04-21 23:44:59 | INFO | MainProcess | Thread-276 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.58 2022-04-21 23:44:59 | INFO | MainProcess | Thread-276 | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: /opt/couchbase/bin/cbcollect_info 172.23.107.58-20220421-2344-diag.zip 2022-04-21 23:46:57 | INFO | MainProcess | Thread-277 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.107.84 2022-04-21 23:46:57 | INFO | MainProcess | Thread-277 | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: rm -f /root/172.23.107.84-20220421-2344-diag.zip 2022-04-21 23:46:57 | INFO | MainProcess | Thread-277 | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:47:00 | INFO | MainProcess | Thread-276 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.107.58 2022-04-21 23:47:00 | INFO | MainProcess | Thread-276 | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: rm -f /root/172.23.107.58-20220421-2344-diag.zip 2022-04-21 23:47:00 | INFO | MainProcess | Thread-275 | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:47:00 | INFO | MainProcess | Thread-276 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.107.78 2022-04-21 23:47:00 | INFO | MainProcess | Thread-275 | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: rm -f /root/172.23.107.78-20220421-2344-diag.zip 2022-04-21 23:47:00 | INFO | MainProcess | Thread-275 | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:10 | INFO | MainProcess | Thread-274 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.106.184 2022-04-21 23:48:10 | INFO | MainProcess | Thread-274 | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: rm -f /root/172.23.106.184-20220421-2344-diag.zip 2022-04-21 23:48:10 | INFO | MainProcess | Thread-274 | [remote_util.execute_command_raw] command executed successfully with root /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.106.184'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.106.184'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.106.184'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.106.184'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.106.184'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.106.184'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.106.184'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, /usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.23.106.184'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, FAIL ====================================================================== FAIL: test_alter_index_for_collections (gsi.collections_alter_index.CollectionsAlterIndex) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/gsi/collections_alter_index.py", line 89, in test_alter_index_for_collections nodes=[old_index_node, new_index_node]) File "pytests/gsi/base_gsi.py", line 453, in alter_index_replicas self.fail("alter index failed to change the number of replicas") AssertionError: alter index failed to change the number of replicas ---------------------------------------------------------------------- Ran 1 test in 357.869s FAILED (failures=1) suite_tearDown (gsi.collections_alter_index.CollectionsAlterIndex) ... summary so far suite gsi.collections_alter_index.CollectionsAlterIndex , pass 0 , fail 1 failures so far... gsi.collections_alter_index.CollectionsAlterIndex.test_alter_index_for_collections testrunner logs, diags and results are available under /data/workspace/centos-p0-2i-vset00-00-collections_gsi_alter_index_moi_7.0_P0/logs/testrunner-22-Apr-21_23-34-17/test_1 *** Tests executed count: 1 Run after suite setup for gsi.collections_alter_index.CollectionsAlterIndex.test_alter_index_for_collections 2022-04-21 23:48:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:11 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:48:11 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2558-enterprise 2022-04-21 23:48:11 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_tearDown============== 2022-04-21 23:48:11 | INFO | MainProcess | MainThread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has started ============== 2022-04-21 23:48:11 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:48:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:12 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:48:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-04-21 23:48:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:13 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:48:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:14 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:48:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-04-21 23:48:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:14 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ###################### Disabling n2n encryption 2022-04-21 23:48:14 | INFO | MainProcess | MainThread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2022-04-21 23:48:14 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 23:48:14 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2022-04-21 23:48:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:15 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2022-04-21 23:48:16 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2022-04-21 23:48:16 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 23:48:16 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2022-04-21 23:48:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:17 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2022-04-21 23:48:17 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned off encryption for node: http://172.23.106.184:8091', 'SUCCESS: Switched node-to-node encryption off'] 2022-04-21 23:48:17 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 23:48:17 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 23:48:17 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.184:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.78764675855028, 'mem_free': 3044888576, 'mem_total': 3945775104, 'swap_mem_used': 68943872, 'swap_mem_total': 3758092288} 2022-04-21 23:48:17 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 23:48:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:48:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:48:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:48:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:48:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:48:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:48:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:33 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_tearDown ============== 2022-04-21 23:48:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.184, nothing to delete 2022-04-21 23:48:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.78, nothing to delete 2022-04-21 23:48:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.58, nothing to delete 2022-04-21 23:48:33 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.84, nothing to delete 2022-04-21 23:48:33 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-04-21 23:48:33 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE https://172.23.106.184:18091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 23:48:33 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."' 2022-04-21 23:48:33 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:48:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.184:8091 2022-04-21 23:48:33 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:48:33 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.184:8091 is running 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.78:8091 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.78:8091 is running 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.58:8091 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.58:8091 is running 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.84:8091 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.84:8091 is running 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_tearDown ============== 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has completed ============== 2022-04-21 23:48:34 | INFO | MainProcess | MainThread | [basetestcase.setUp] Building docker image with java sdk client Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 2022-04-21 23:48:38 | INFO | MainProcess | MainThread | [basetestcase.setUp] #####Enforcing TLS######## 2022-04-21 23:48:38 | INFO | MainProcess | MainThread | [ntonencryptionBase.setup_nton_cluster] Setting up node to node encryption - status = enable and clusterEncryptionLevel = strict 2022-04-21 23:48:38 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 23:48:38 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to enable 2022-04-21 23:48:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:39 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --enable 2022-04-21 23:48:40 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned on encryption for node: http://172.23.106.184:8091', 'SUCCESS: Switched node-to-node encryption on'] 2022-04-21 23:48:40 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 23:48:40 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = strict 2022-04-21 23:48:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:41 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:41 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level strict 2022-04-21 23:48:42 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ['SUCCESS: Security settings updated'] 2022-04-21 23:48:42 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 23:48:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:8091 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:8093 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:9102 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :11210 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:11210 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :8092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.log_command_output] 127.0.0.1:8092 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :9003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:18091 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:18093 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:19102 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :11207 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:11207 2022-04-21 23:48:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :18092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.log_command_output] *:18092 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: ss -4anpe | grep :19003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:47 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.184 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:48:47 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:48:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: systemctl stop couchbase-server.service 2022-04-21 23:48:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:49 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:48:49 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:48:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:48:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:48:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:48:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:48:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:51 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.184 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:48:51 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:48:51 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:48:51 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:48:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: systemctl start couchbase-server.service 2022-04-21 23:48:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:51 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.106.184:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:48:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:48:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:48:56 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 12958 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:48:56 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:48:56 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.184: with pid 13099 2022-04-21 23:48:56 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 23:48:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:48:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:48:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:48:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:48:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:48:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:48:58 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.78 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:48:58 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:48:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: systemctl stop couchbase-server.service 2022-04-21 23:49:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:01 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:49:01 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:49:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:49:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:49:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:49:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:49:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:49:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:49:03 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.78 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:49:03 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:49:03 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:49:03 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:49:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: systemctl start couchbase-server.service 2022-04-21 23:49:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:03 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.107.78:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:49:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:49:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:08 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 115287 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:49:08 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:49:08 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.78: with pid 115429 2022-04-21 23:49:08 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 23:49:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:49:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:49:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:49:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:49:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:49:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:49:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:49:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:49:10 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.58 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:49:10 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:49:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: systemctl stop couchbase-server.service 2022-04-21 23:49:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:13 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:49:13 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:49:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:49:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:49:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:49:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:49:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:49:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:49:15 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.58 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:49:15 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:49:15 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:49:15 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:49:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: systemctl start couchbase-server.service 2022-04-21 23:49:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:15 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.107.58:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:49:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:49:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:20 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 112907 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:49:20 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:49:20 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.58: with pid 113078 2022-04-21 23:49:20 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 23:49:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:49:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:49:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:49:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:49:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:49:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:49:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:49:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:49:22 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.84 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:49:22 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2022-04-21 23:49:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: systemctl stop couchbase-server.service 2022-04-21 23:49:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:25 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:49:25 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2022-04-21 23:49:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-04-21 23:49:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-04-21 23:49:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:49:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:49:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:49:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:49:26 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.84 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-04-21 23:49:26 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:49:26 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2022-04-21 23:49:26 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2022-04-21 23:49:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: systemctl start couchbase-server.service 2022-04-21 23:49:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:26 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.107.84:sleep for 5 secs. waiting for couchbase server to come up ... 2022-04-21 23:49:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-04-21 23:49:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:32 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 128889 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-04-21 23:49:32 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-04-21 23:49:32 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.107.84: with pid 129031 2022-04-21 23:49:32 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2022-04-21 23:49:32 | ERROR | MainProcess | MainThread | [rest_client._http_request] socket error while connecting to https://172.23.107.84:18091/nodes/self error [Errno 111] Connection refused 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.184 port:8091 ssh_username:root, nodes/self 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '43', 'memoryTotal': 3945775104, 'memoryFree': 3303223296, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.184:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.106.184', 'ip': '172.23.106.184', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:49:38 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.106.184:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 1008 MB 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.106.184 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1008 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.106.184,18091,['kv', 'n1ql', 'index']) 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.106.184: 18091:hostname=172.23.106.184&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.184:18091:port=8091&username=Administrator&password=password 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:49:38 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.78 port:8091 ssh_username:root, nodes/self 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '32', 'memoryTotal': 3945775104, 'memoryFree': 3326001152, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.78:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.78', 'ip': '172.23.107.78', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:49:39 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.78:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:49:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.78:18091:port=8091&username=Administrator&password=password 2022-04-21 23:49:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:49:40 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:49:40 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:49:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:49:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:49:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.78: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.78:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.58 port:8091 ssh_username:root, nodes/self 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '22', 'memoryTotal': 3945775104, 'memoryFree': 3308937216, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.58:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.58', 'ip': '172.23.107.58', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:49:41 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.58:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.58:18091:port=8091&username=Administrator&password=password 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:49:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.58:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.84 port:8091 ssh_username:root, nodes/self 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '12', 'memoryTotal': 3945775104, 'memoryFree': 3302219776, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.84:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2558-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11207, 'id': 'ns_1@172.23.107.84', 'ip': '172.23.107.84', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=1008 2022-04-21 23:49:42 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://172.23.107.84:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.84:18091:port=8091&username=Administrator&password=password 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:49:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:49:43 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:49:43 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:49:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.84: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:49:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:49:43 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:49:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.84:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-04-21 23:49:43 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-04-21 23:49:43 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.106.184 **** 2022-04-21 23:49:44 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE https://172.23.106.184:18091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 23:49:44 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-04-21 23:49:44 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2022-04-21 23:49:49 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-04-21 23:49:49 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2022-04-21 23:49:49 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.78:8091 to cluster 2022-04-21 23:49:50 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.78:18091 to this cluster @172.23.106.184:18091 2022-04-21 23:50:00 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.06 seconds 2022-04-21 23:50:00 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 23:50:14 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.58:8091 to cluster 2022-04-21 23:50:14 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.58:18091 to this cluster @172.23.106.184:18091 2022-04-21 23:50:24 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.05 seconds 2022-04-21 23:50:24 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 23:50:38 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.58 not part of cluster inactiveAdded 2022-04-21 23:50:38 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.78 not part of cluster inactiveAdded 2022-04-21 23:50:38 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.58,ns_1@172.23.107.78', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-04-21 23:50:38 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-04-21 23:50:48 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-04-21 23:50:48 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.07340669631958 sec 2022-04-21 23:50:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:50:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:50:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:50:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:50:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-04-21 23:50:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:50:49 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-04-21 23:50:49 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.106.184 2022-04-21 23:50:49 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_tearDown ============== 2022-04-21 23:50:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:50:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:50:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:50:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:50:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:50:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:50:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:50:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:50:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:50:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:50:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:50:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:50:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:50:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:50:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:50:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.184:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.71508522004579, 'mem_free': 3191021568, 'mem_total': 3945775104, 'swap_mem_used': 68943872, 'swap_mem_total': 3758092288} 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.107.58:8091 => {'services': ['index'], 'cpu_utilization': 3.661616161616162, 'mem_free': 3295080448, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.107.78:8091 => {'services': ['index'], 'cpu_utilization': 12.65243902439024, 'mem_free': 3299938304, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 175.05167722702026 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.106.184 cluster is healthy 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.106.184 is healthy 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.107.78 cluster is healthy 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.107.78 is healthy 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.107.58 cluster is healthy 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.107.58 is healthy 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.107.84 cluster is healthy 2022-04-21 23:51:05 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.107.84 is healthy 2022-04-21 23:51:06 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.58 port:8091 ssh_username:root, ip:172.23.107.78 port:8091 ssh_username:root] 2022-04-21 23:51:06 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set 2022-04-21 23:51:06 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2022-04-21 23:51:06 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2022-04-21 23:51:11 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2022-04-21 23:51:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:51:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:51:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:51:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:51:20 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED 2022-04-21 23:51:20 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:51:20 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:172.23.106.184 port:8091 ssh_username:root 2022-04-21 23:51:20 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs. ... 2022-04-21 23:51:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:51:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:51:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:51:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:51:51 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.58 port:8091 ssh_username:root, ip:172.23.107.78 port:8091 ssh_username:root] 2022-04-21 23:51:51 | INFO | MainProcess | MainThread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:19102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False 2022-04-21 23:51:51 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2022-04-21 23:51:51 | INFO | MainProcess | MainThread | [rest_client.urllib_request] Making a rest request api=https://172.23.106.184:19102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False 2022-04-21 23:51:51 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2022-04-21 23:51:51 | INFO | MainProcess | MainThread | [collections_alter_index.setUp] ============== CollectionsIndexBasics setup has started ============== 2022-04-21 23:51:52 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] https://172.23.106.184:18091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=672&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-04-21 23:51:52 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.06 seconds to create bucket test_bucket 2022-04-21 23:51:52 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 172.23.106.184 to accept set ops 2022-04-21 23:51:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.184:11207 test_bucket 2022-04-21 23:51:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.184:11207 test_bucket 2022-04-21 23:51:55 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'test_bucket' was created with per node RAM quota: 672 2022-04-21 23:51:55 | INFO | MainProcess | MainThread | [collections_alter_index.setUp] ============== CollectionsIndexBasics setup has completed ============== 2022-04-21 23:51:55 | INFO | MainProcess | MainThread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has started ============== 2022-04-21 23:51:55 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.58 port:8091 ssh_username:root, ip:172.23.107.78 port:8091 ssh_username:root] 2022-04-21 23:51:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:51:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:51:55 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:51:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:51:56 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:51:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-04-21 23:51:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:51:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:51:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:51:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:51:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:51:57 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.107.58:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:51:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.58: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-04-21 23:51:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:51:57 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:51:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:51:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:51:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.106.184:18091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.184: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root] 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ###################### Disabling n2n encryption 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:51:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:51:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:51:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:51:59 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2022-04-21 23:52:00 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2022-04-21 23:52:00 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-04-21 23:52:00 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2022-04-21 23:52:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:52:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:52:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:52:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:52:01 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2022-04-21 23:52:02 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned off encryption for node: http://172.23.106.184:8091', 'Turned off encryption for node: http://172.23.107.58:8091', 'Turned off encryption for node: http://172.23.107.78:8091', 'SUCCESS: Switched node-to-node encryption off'] 2022-04-21 23:52:02 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-04-21 23:52:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-04-21 23:52:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.184:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.92114695340502, 'mem_free': 3086336000, 'mem_total': 3945775104, 'swap_mem_used': 68943872, 'swap_mem_total': 3758092288} 2022-04-21 23:52:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.107.58:8091 => {'services': ['index'], 'cpu_utilization': 2.878061095682908, 'mem_free': 3266289664, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:52:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.107.78:8091 => {'services': ['index'], 'cpu_utilization': 3.141626551811502, 'mem_free': 3274625024, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-04-21 23:52:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-04-21 23:52:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 2022-04-21 23:52:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.184 as root 2022-04-21 23:52:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:52:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:52:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.78 with username:root, attempt#1 of 5 2022-04-21 23:52:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.78 as root 2022-04-21 23:52:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:52:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:52:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.58 with username:root, attempt#1 of 5 2022-04-21 23:52:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.58 as root 2022-04-21 23:52:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:52:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:52:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 2022-04-21 23:52:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.84 as root 2022-04-21 23:52:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-04-21 23:52:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-04-21 23:52:17 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_tearDown ============== 2022-04-21 23:52:17 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket'] on 172.23.106.184 2022-04-21 23:52:18 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-04-21 23:52:18 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.106.184 existing buckets : [] 2022-04-21 23:52:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket from 172.23.106.184 2022-04-21 23:52:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.78, nothing to delete 2022-04-21 23:52:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.58, nothing to delete 2022-04-21 23:52:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.84, nothing to delete 2022-04-21 23:52:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-04-21 23:52:18 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE https://172.23.106.184:18091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-04-21 23:52:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."' 2022-04-21 23:52:18 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:52:18 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-04-21 23:52:18 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.58,ns_1@172.23.107.78', 'ejectedNodes': 'ns_1@172.23.107.58,ns_1@172.23.107.78', 'user': 'Administrator', 'password': 'password'} 2022-04-21 23:52:19 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2022-04-21 23:52:19 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-04-21 23:52:39 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 20.09 seconds 2022-04-21 23:52:39 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.106.184 port:8091 ssh_username:root ? [('ns_1@172.23.107.58', 8091), ('ns_1@172.23.107.78', 8091)] 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.184:8091 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.184:8091 is running 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.78:8091 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.78:8091 is running 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.58:8091 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.58:8091 is running 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.84:8091 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.84:8091 is running 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_tearDown ============== 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-04-21 23:52:49 | INFO | MainProcess | MainThread | [collections_alter_index.tearDown] ============== CollectionsIndexBasics tearDown has completed ============== *** TestRunner *** workspace is /data/workspace/centos-p0-2i-vset00-00-collections_gsi_alter_index_moi_7.0_P0 fails is 1 1 Desc1: 7.1.0-2558 - 2i collections_gsi_alter_index_moi_7.0_P0 - centos (0/1) python3 scripts/rerun_jobs.py 7.1.0-2558 --executor_jenkins_job --run_params=gsi_type=memory_optimized,get-cbcollect-info=True,reset_services=True,java_sdk_client=True,GROUP=P0,index_quota_percent=50,enforce_tls=True,use_https=True INFO:merge_reports:Merging of report files from ['Old_Report_mergedreport-22-Apr-21_23-31-01-gsi.collections_alter_index.CollectionsAlterIndex.xml', 'logs/**/*.xml'] INFO:merge_reports:-- Old_Report_mergedreport-22-Apr-21_23-31-01-gsi.collections_alter_index.CollectionsAlterIndex.xml -- INFO:merge_reports:-- logs/testrunner-22-Apr-21_23-34-17/report-22-Apr-21_23-34-17-gsi.collections_alter_index.CollectionsAlterIndex.xml -- INFO:merge_reports: Number of TestSuites=1 INFO:merge_reports: TestSuite#1) gsi.collections_alter_index.CollectionsAlterIndex, Number of Tests=1 INFO:merge_reports:Summary file is at /data/workspace/centos-p0-2i-vset00-00-collections_gsi_alter_index_moi_7.0_P0/logs/testrunner-22-Apr-21_23-52-50/merged_summary/mergedreport-22-Apr-21_23-52-50-gsi.collections_alter_index.CollectionsAlterIndex.xml Merging xmls summary so far suite gsi.collections_alter_index.CollectionsAlterIndex , pass 0 , fail 1 failures so far... gsi.collections_alter_index.CollectionsAlterIndex.test_alter_index_for_collections merged xmls http://qa.sc.couchbase.com/job/test_suite_dispatcher/buildWithParameters?token=extended_sanity&OS=centos&version_number=7.1.0-2558&suite=12hr_weekly&component=2i&subcomponent=collections_gsi_alter_index_moi_7.0_P0&subcomponent_regex=None&serverPoolId=regression&addPoolId=None&extraParameters=enforce_tls%3DTrue%2Cuse_https%3DTrue&Test=False&url=&branch=master&cherrypick=&retries=0&fresh_run=false&rerun_params=-d+failed%3Dhttp%3A%2F%2Fqa.sc.couchbase.com%2Fjob%2Ftest_suite_executor%2F462998%2F&executor_suffix=&executor_job_parameters=&check_vm=False&initial_version=&capella_mock=False&serverType=CAPELLA_LOCAL [description-setter] Description set: 7.1.0-2558 - 2i collections_gsi_alter_index_moi_7.0_P0 - centos (0/1) [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) #462996 is still in progress; ignoring for purposes of comparison Sending email to: mihir.kamdar@couchbase.com hemant.rajput@couchbase.com Triggering a new build of savejoblogs Triggering a new build of test-executor-cleanup Triggering a new build of post-to-gerrit Triggering a new build of test-executor-cleanup-aws Finished: UNSTABLE