/Users/lfc/venv/bin/python "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 58669 --file /Users/lfc/testrunner/testrunner.py -i b/resources/tmp.ini -p gsi_type=memory_optimized,reset_services=False,enforce_tls=True,use_https=True -t gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_alter_index_with_prepared_statements,doc-per-day=1000,dataset=bigdata,nodes_init=4,services_init=kv:n1ql:index-index-index-index,groups=simple,reset_services=False,nodes=0:1,dest_node=2:3,create_primary_index=False,GROUP=P1,gsi_type=memory_optimized,enforce_tls=True,use_https=True Connected to pydev debugger (build 212.4746.96) /Users/lfc/testrunner/lib/builds/build_query.py:818: SyntaxWarning: "is not" with a literal. Did you mean "!="? if toy is not "": /Users/lfc/testrunner/lib/builds/build_query.py:938: SyntaxWarning: "is not" with a literal. Did you mean "!="? elif toy is not "": /Users/lfc/testrunner/./lib/remote/remote_util.py:4250: SyntaxWarning: "is not" with a literal. Did you mean "!="? if command_options is not '': /Users/lfc/testrunner/lib/remote/remote_util.py:4250: SyntaxWarning: "is not" with a literal. Did you mean "!="? if command_options is not '': INFO:root:__main__ *** TestRunner *** INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: {'cluster_name': 'tmp', 'enforce_tls': 'True', 'gsi_type': 'memory_optimized', 'ini': 'b/resources/tmp.ini', 'num_nodes': 4, 'reset_services': 'False', 'use_https': 'True'} INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_alter_index_with_prepared_statements,doc-per-day=1000,dataset=bigdata,nodes_init=4,services_init=kv:n1ql:index-index-index-index,groups=simple,reset_services=False,nodes=0:1,dest_node=2:3,create_primary_index=False,GROUP=P1,gsi_type=memory_optimized,enforce_tls=True,use_https=True INFO:__main__:Logs folder: /Users/lfc/testrunner/logs/testrunner-22-May-17_12-43-46/test_1 Logs will be stored at /Users/lfc/testrunner/logs/testrunner-22-May-17_12-43-46/test_1 ./testrunner -i b/resources/tmp.ini -p gsi_type=memory_optimized,reset_services=False,enforce_tls=True,use_https=True -t gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_alter_index_with_prepared_statements,doc-per-day=1000,dataset=bigdata,nodes_init=4,services_init=kv:n1ql:index-index-index-index,groups=simple,reset_services=False,nodes=0:1,dest_node=2:3,create_primary_index=False,GROUP=P1,gsi_type=memory_optimized,enforce_tls=True,use_https=True Test Input params: {'doc-per-day': '1000', 'dataset': 'bigdata', 'nodes_init': '4', 'services_init': 'kv:n1ql:index-index-index-index', 'groups': 'simple', 'reset_services': 'False', 'nodes': '0:1', 'dest_node': '2:3', 'create_primary_index': 'False', 'GROUP': 'P1', 'gsi_type': 'memory_optimized', 'enforce_tls': 'True', 'use_https': 'True', 'ini': 'b/resources/tmp.ini', 'cluster_name': 'tmp', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/Users/lfc/testrunner/logs/testrunner-22-May-17_12-43-46/test_1'} Run before suite setup for gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_alter_index_with_prepared_statements /Users/lfc/testrunner/lib/tasks/task.py:302: SyntaxWarning: "is" with a literal. Did you mean "=="? if bucket_params['bucket_priority'] is None or bucket_params['bucket_priority'].lower() is 'low': /Users/lfc/testrunner/./lib/membase/helper/cluster_helper.py:382: SyntaxWarning: "is" with a literal. Did you mean "=="? if index_path is '': /Users/lfc/testrunner/pytests/basetestcase.py:1494: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.stats_monitor is not '': /Users/lfc/testrunner/pytests/basetestcase.py:1496: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.warmup_stats_monitor is not '': /Users/lfc/testrunner/lib/membase/helper/cluster_helper.py:382: SyntaxWarning: "is" with a literal. Did you mean "=="? if index_path is '': /Users/lfc/testrunner/lib/couchbase_helper/tuq_helper.py:219: SyntaxWarning: "is" with a literal. Did you mean "=="? if status is "any": /Users/lfc/testrunner/lib/couchbase_helper/tuq_helper.py:229: SyntaxWarning: "is" with a literal. Did you mean "=="? if status is "any": -->before_suite_name:gsi.gsi_replica_indexes.GSIReplicaIndexesTests.suite_setUp,suite: ]> suite_setUp (unittest.loader._FailedTest) ... ERROR ====================================================================== ERROR: suite_setUp (unittest.loader._FailedTest) ---------------------------------------------------------------------- AttributeError: type object 'GSIReplicaIndexesTests' has no attribute 'suite_setUp' ---------------------------------------------------------------------- Ran 1 test in 0.001s FAILED (errors=1) -->result: test_alter_index_with_prepared_statements (gsi.gsi_replica_indexes.GSIReplicaIndexesTests) ... 2022-05-17 12:43:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.1.1-3027-enterprise 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_alter_index_with_prepared_statements============== 2022-05-17 12:43:47 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET https://10.112.205.101:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-05-17 12:43:47 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET https://10.112.205.101:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ###################### Disabling n2n encryption 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:43:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:43:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:43:48 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2022-05-17 12:43:48 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2022-05-17 12:43:48 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-05-17 12:43:48 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2022-05-17 12:43:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:43:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:43:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:43:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:43:49 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2022-05-17 12:43:49 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['SUCCESS: Switched node-to-node encryption off'] 2022-05-17 12:43:49 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-05-17 12:43:49 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET https://10.112.205.101:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-05-17 12:43:49 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-05-17 12:43:49 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-05-17 12:43:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:43:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:43:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:43:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:43:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-05-17 12:43:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2022-05-17 12:43:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:43:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:43:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-05-17 12:43:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2022-05-17 12:43:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:43:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:43:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-05-17 12:43:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2022-05-17 12:43:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:43:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:43:56 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_alter_index_with_prepared_statements ============== 2022-05-17 12:43:56 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET https://10.112.205.101:18091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-05-17 12:43:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.101, nothing to delete 2022-05-17 12:43:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2022-05-17 12:43:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2022-05-17 12:43:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2022-05-17 12:43:56 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-05-17 12:43:56 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET https://10.112.205.101:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-05-17 12:43:56 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-05-17 12:43:56 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET https://10.112.205.101:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_alter_index_with_prepared_statements ============== 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-05-17 12:43:57 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.101 port:8091 ssh_username:root, nodes/self 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '474', 'memoryTotal': 1930829824, 'memoryFree': 1502814208, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.101:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.1-3027-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'memcached': 11207, 'id': 'ns_1@cb.local', 'ip': '10.112.205.101', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2022-05-17 12:43:58 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://10.112.205.101:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 10.112.205.101 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=730 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.205.101,18091,['kv', 'n1ql', 'index']) 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.205.101: 18091:hostname=10.112.205.101&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.101:18091:port=8091&username=Administrator&password=password 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:43:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.102 port:8091 ssh_username:root, nodes/self 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '474', 'memoryTotal': 1930829824, 'memoryFree': 1522257920, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.102:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.1-3027-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'memcached': 11207, 'id': 'ns_1@cb.local', 'ip': '10.112.205.102', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2022-05-17 12:43:59 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://10.112.205.102:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.102:18091:port=8091&username=Administrator&password=password 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-05-17 12:43:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.102:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.103 port:8091 ssh_username:root, nodes/self 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '474', 'memoryTotal': 2033594368, 'memoryFree': 1619640320, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.103:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.1-3027-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'memcached': 11207, 'id': 'ns_1@cb.local', 'ip': '10.112.205.103', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2022-05-17 12:44:00 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://10.112.205.103:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1039 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.103:18091:port=8091&username=Administrator&password=password 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-05-17 12:44:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.104 port:8091 ssh_username:root, nodes/self 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '474', 'memoryTotal': 2033594368, 'memoryFree': 1607434240, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.104:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.1-3027-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'memcached': 11207, 'id': 'ns_1@cb.local', 'ip': '10.112.205.104', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2022-05-17 12:44:01 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET https://10.112.205.104:18091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1039 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.104:18091:port=8091&username=Administrator&password=password 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-05-17 12:44:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-05-17 12:44:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2022-05-17 12:44:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:44:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:44:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-05-17 12:44:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:02 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-05-17 12:44:02 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:18091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-05-17 12:44:02 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized 2022-05-17 12:44:02 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.205.101 **** 2022-05-17 12:44:02 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE https://10.112.205.101:18091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-05-17 12:44:02 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-05-17 12:44:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-05-17 12:44:08 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-05-17 12:44:08 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2022-05-17 12:44:08 | INFO | MainProcess | test_thread | [basetestcase.setUp] #####Enforcing TLS######## 2022-05-17 12:44:08 | INFO | MainProcess | test_thread | [ntonencryptionBase.setup_nton_cluster] Setting up node to node encryption - status = enable and clusterEncryptionLevel = strict 2022-05-17 12:44:08 | INFO | MainProcess | test_thread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-05-17 12:44:08 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to enable 2022-05-17 12:44:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:44:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:44:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:44:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:44:08 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --enable 2022-05-17 12:44:09 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned on encryption for node: http://[::1]:8091', 'SUCCESS: Switched node-to-node encryption on'] 2022-05-17 12:44:09 | INFO | MainProcess | test_thread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2022-05-17 12:44:09 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = strict 2022-05-17 12:44:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:44:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:44:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:44:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:44:09 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level strict 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ['SUCCESS: Security settings updated'] 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :8091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8091 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :8094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :8093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8093 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :9102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:9102 2022-05-17 12:44:10 | ERROR | MainProcess | test_thread | [cluster_helper.check_if_services_obey_tls] On Server 10.112.205.101 Expected 127.0.0.1:9102 Actual *:9102 !!!!!!!!!!!!!!!! 2022-05-17 12:44:10 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2022-05-17 12:44:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:44:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :8091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8091 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :8094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :8093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8093 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :9102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:9102 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :8096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :8097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :8095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :11210 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:11210 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :8092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 127.0.0.1:8092 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :9000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :9001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :9002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :9003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :18091 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:18091 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :18094 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :18093 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:18093 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :19102 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:19102 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :18096 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :18097 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :18095 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :11207 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:11207 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :18092 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.log_command_output] *:18092 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :19000 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :19001 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :19002 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: ss -4anpe | grep :19003 | grep 'LISTEN' | awk -F ' ' '{print $5}' 2022-05-17 12:44:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:44:22 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.102:8091 to cluster 2022-05-17 12:44:22 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.102:18091 to this cluster @10.112.205.101:18091 2022-05-17 12:44:33 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.04 seconds 2022-05-17 12:44:33 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-05-17 12:44:54 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.103:8091 to cluster 2022-05-17 12:44:54 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.103:18091 to this cluster @10.112.205.101:18091 2022-05-17 12:45:04 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.04 seconds 2022-05-17 12:45:04 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-05-17 12:45:19 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.104:8091 to cluster 2022-05-17 12:45:19 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.104:18091 to this cluster @10.112.205.101:18091 2022-05-17 12:45:29 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.05 seconds 2022-05-17 12:45:29 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-05-17 12:45:43 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 10.112.205.102 not part of cluster inactiveAdded 2022-05-17 12:45:43 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 10.112.205.103 not part of cluster inactiveAdded 2022-05-17 12:45:43 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 10.112.205.104 not part of cluster inactiveAdded 2022-05-17 12:45:43 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103,ns_1@10.112.205.104', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-05-17 12:45:43 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-05-17 12:45:53 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-05-17 12:45:53 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.07779312133789 sec 2022-05-17 12:45:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:45:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:45:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:45:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:45:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-05-17 12:45:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-05-17 12:45:54 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-05-17 12:45:54 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.205.101 2022-05-17 12:45:54 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] https://10.112.205.101:18091/pools/default/buckets with param: name=default&ramQuotaMB=730&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-05-17 12:45:54 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.05 seconds to create bucket default 2022-05-17 12:45:54 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 10.112.205.101 to accept set ops 2022-05-17 12:45:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11207 default 2022-05-17 12:45:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11207 default 2022-05-17 12:45:57 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 730 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_alter_index_with_prepared_statements ============== 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2022-05-17 12:45:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:45:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:45:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-05-17 12:45:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2022-05-17 12:45:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:45:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:46:04 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-05-17 12:46:04 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 42.10526315789474, 'mem_free': 1300344832, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2022-05-17 12:46:04 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.102:8091 => {'services': ['index'], 'cpu_utilization': 9.18580375782881, 'mem_free': 1487343616, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2022-05-17 12:46:04 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.103:8091 => {'services': ['index'], 'cpu_utilization': 3.278688524590164, 'mem_free': 1586008064, 'mem_total': 2033594368, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2022-05-17 12:46:04 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.104:8091 => {'services': ['index'], 'cpu_utilization': 3.995901639344262, 'mem_free': 1576079360, 'mem_total': 2033594368, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2022-05-17 12:46:04 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-05-17 12:46:04 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 137.72880816459656 2022-05-17 12:46:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:46:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:46:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.101 cluster is healthy 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.101 is healthy 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.102 cluster is healthy 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.102 is healthy 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.103 cluster is healthy 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.103 is healthy 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.104 cluster is healthy 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.104 is healthy 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root, ip:10.112.205.104 port:8091 ssh_username:root] 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2022-05-17 12:46:05 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-05-17 12:46:10 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2022-05-17 12:46:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:46:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:46:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:46:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:46:11 | INFO | MainProcess | test_thread | [basetestcase.load] create 1000.0 to default documents... 2022-05-17 12:46:11 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.112.205.101:11207 default 2022-05-17 12:46:14 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2022-05-17 12:46:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2022-05-17 12:46:14 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:10.112.205.101 port:8091 ssh_username:root 2022-05-17 12:46:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root, ip:10.112.205.104 port:8091 ssh_username:root] 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://10.112.205.101:19102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False /Users/lfc/venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.112.205.101'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings warnings.warn( 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://10.112.205.101:19102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False /Users/lfc/venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.112.205.101'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings warnings.warn( 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 12:46:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 12:46:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 12:46:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 12:46:45 | INFO | MainProcess | test_thread | [gsi_replica_indexes.test_alter_index_with_prepared_statements] CREATE INDEX random_index_708809 ON default(age) USING GSI WITH {'nodes': ['10.112.205.101:18091', '10.112.205.102:18091']}; 2022-05-17 12:46:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX random_index_708809 ON default(age) USING GSI WITH {'nodes': ['10.112.205.101:18091', '10.112.205.102:18091']}; 2022-05-17 12:46:45 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+random_index_708809+ON+default%28age%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.205.101%3A18091%27%2C+%2710.112.205.102%3A18091%27%5D%7D%3B 2022-05-17 12:46:49 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.229210488s 2022-05-17 12:46:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY PREPARE prep_stmt AS SELECT count(age) from default USE INDEX (random_index_708809 USING GSI) where age > 10 2022-05-17 12:46:53 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=PREPARE+prep_stmt+AS+SELECT+count%28age%29+from+default+USE+INDEX+%28random_index_708809+USING+GSI%29+where+age+%3E+10 2022-05-17 12:46:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 241.222794ms 2022-05-17 12:46:55 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2022-05-17 13:00:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY prep_stmt 2022-05-17 13:00:45 | INFO | MainProcess | test_thread | [rest_client.query_tool] https://10.112.205.101:18093/query/service?prepared=%22prep_stmt%22 2022-05-17 13:00:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.450934ms 2022-05-17 13:00:45 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=https://10.112.205.101:18091/indexStatus verb=GET params= client_cert=None verify=False /Users/lfc/venv/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.112.205.101'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings warnings.warn( 2022-05-17 13:00:46 | INFO | MainProcess | test_thread | [gsi_replica_indexes.test_alter_index_with_prepared_statements] {'default': {'random_index_708809': {'status': 'Ready', 'progress': '100', 'definition': 'CREATE INDEX `random_index_708809` ON `default`(`age`) WITH { "nodes":[ "10.112.205.101:18091","10.112.205.102:18091" ], "num_replica":1 }', 'hosts': '10.112.205.102:18091', 'id': 17531439742419453994}, 'random_index_708809 (replica 1)': {'status': 'Ready', 'progress': '100', 'definition': 'CREATE INDEX `random_index_708809` ON `default`(`age`) WITH { "nodes":[ "10.112.205.101:18091","10.112.205.102:18091" ], "num_replica":1 }', 'hosts': '10.112.205.101:18091', 'id': 17531439742419453994}}} 2022-05-17 13:00:46 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes] Hostnames : 10.112.205.102:18091 , 10.112.205.101:18091 2022-05-17 13:00:46 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes] Index IDs : 17531439742419453994, 17531439742419453994 2022-05-17 13:00:46 | INFO | MainProcess | test_thread | [gsi_replica_indexes.test_alter_index_with_prepared_statements] ['10.112.205.103:18091', '10.112.205.104:18091'] 2022-05-17 13:00:50 | INFO | MainProcess | test_thread | [gsi_replica_indexes._cbindex_move] cbindex -type move -server '10.112.205.101:18091' -auth 'Administrator:password' -index 'random_index_708809' -bucket default -with '{"nodes":["10.112.205.103:18091", "10.112.205.104:18091"]}' 2022-05-17 13:00:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-05-17 13:00:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-05-17 13:00:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-05-17 13:00:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-05-17 13:00:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: /opt/couchbase/bin//cbindex -type move -server '10.112.205.101:18091' -auth 'Administrator:password' -index 'random_index_708809' -bucket default -with '{"nodes":["10.112.205.103:18091", "10.112.205.104:18091"]}' 2022-05-17 13:01:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with root but got an error ['2022/05/17 00:30:51 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:51 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:52 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:52 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:53 revrpc: Got error (unexpected EOF) and will retry in 1s', '202 ... 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:51 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:51 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:52 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:52 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:53 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:53 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:54 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:54 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:55 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:55 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:56 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:56 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:57 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:57 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:58 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:58 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:59 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:30:59 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:31:00 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:31:00 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:31:01 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:31:01 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] 2022/05/17 00:31:02 revrpc: Got error (unexpected EOF) and will retry in 1s 2022-05-17 13:01:02 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2022-05-17T00:31:02.293-07:00 [Fatal] CBAuth database is stale: last reason: unexpected EOF 2022-05-17 13:01:02 | INFO | MainProcess | test_thread | [gsi_replica_indexes._cbindex_move] Output : ['2022-05-17T00:31:02.293-07:00 [Fatal] CBAuth database is stale: last reason: unexpected EOF'] 2022-05-17 13:01:02 | INFO | MainProcess | test_thread | [gsi_replica_indexes._cbindex_move] Error : ['2022/05/17 00:30:51 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:51 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:52 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:52 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:53 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:53 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:54 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:54 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:55 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:55 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:56 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:56 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:57 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:57 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:58 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:58 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:59 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:30:59 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:31:00 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:31:00 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:31:01 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:31:01 revrpc: Got error (unexpected EOF) and will retry in 1s', '2022/05/17 00:31:02 revrpc: Got error (unexpected EOF) and will retry in 1s']