lfc/venv/bin/python "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 60922 --file /Users/lfc/testrunner/testrunner.py -i b/resources/tmp.ini -p gsi_type=plasma,bucket_storage=couchstore -t gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_build_index_with_replica_failover_addback,doc-per-day=1000,dataset=bigdata,nodes_init=4,services_init=kv:index:n1ql-index-index-index,groups=simple,create_primary_index=False,reset_services=True,nodes=0:1:2,node_out=2,recovery_type=full Connected to pydev debugger (build 212.4746.96) /Users/lfc/testrunner/lib/builds/build_query.py:818: SyntaxWarning: "is not" with a literal. Did you mean "!="? if toy is not "": /Users/lfc/testrunner/lib/builds/build_query.py:938: SyntaxWarning: "is not" with a literal. Did you mean "!="? elif toy is not "": /Users/lfc/testrunner/./lib/remote/remote_util.py:4227: SyntaxWarning: "is not" with a literal. Did you mean "!="? if command_options is not '': /Users/lfc/testrunner/lib/remote/remote_util.py:4227: SyntaxWarning: "is not" with a literal. Did you mean "!="? if command_options is not '': INFO:root:__main__ *** TestRunner *** INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: {'bucket_storage': 'couchstore', 'cluster_name': 'tmp', 'gsi_type': 'plasma', 'ini': 'b/resources/tmp.ini', 'num_nodes': 5} INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_build_index_with_replica_failover_addback,doc-per-day=1000,dataset=bigdata,nodes_init=4,services_init=kv:index:n1ql-index-index-index,groups=simple,create_primary_index=False,reset_services=True,nodes=0:1:2,node_out=2,recovery_type=full INFO:__main__:Logs folder: /Users/lfc/testrunner/logs/testrunner-22-Feb-17_12-19-56/test_1 Logs will be stored at /Users/lfc/testrunner/logs/testrunner-22-Feb-17_12-19-56/test_1 ./testrunner -i b/resources/tmp.ini -p gsi_type=plasma,bucket_storage=couchstore -t gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_build_index_with_replica_failover_addback,doc-per-day=1000,dataset=bigdata,nodes_init=4,services_init=kv:index:n1ql-index-index-index,groups=simple,create_primary_index=False,reset_services=True,nodes=0:1:2,node_out=2,recovery_type=full Test Input params: {'doc-per-day': '1000', 'dataset': 'bigdata', 'nodes_init': '4', 'services_init': 'kv:index:n1ql-index-index-index', 'groups': 'simple', 'create_primary_index': 'False', 'reset_services': 'True', 'nodes': '0:1:2', 'node_out': '2', 'recovery_type': 'full', 'ini': 'b/resources/tmp.ini', 'cluster_name': 'tmp', 'gsi_type': 'plasma', 'bucket_storage': 'couchstore', 'num_nodes': 5, 'case_number': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/Users/lfc/testrunner/logs/testrunner-22-Feb-17_12-19-56/test_1'} Run before suite setup for gsi.gsi_replica_indexes.GSIReplicaIndexesTests.test_build_index_with_replica_failover_addback /Users/lfc/testrunner/lib/tasks/task.py:302: SyntaxWarning: "is" with a literal. Did you mean "=="? if bucket_params['bucket_priority'] is None or bucket_params['bucket_priority'].lower() is 'low': /Users/lfc/testrunner/./lib/membase/helper/cluster_helper.py:382: SyntaxWarning: "is" with a literal. Did you mean "=="? if index_path is '': /Users/lfc/testrunner/pytests/basetestcase.py:1490: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.stats_monitor is not '': /Users/lfc/testrunner/pytests/basetestcase.py:1492: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.warmup_stats_monitor is not '': /Users/lfc/testrunner/lib/membase/helper/cluster_helper.py:382: SyntaxWarning: "is" with a literal. Did you mean "=="? if index_path is '': /Users/lfc/testrunner/lib/couchbase_helper/tuq_helper.py:219: SyntaxWarning: "is" with a literal. Did you mean "=="? if status is "any": /Users/lfc/testrunner/lib/couchbase_helper/tuq_helper.py:229: SyntaxWarning: "is" with a literal. Did you mean "=="? if status is "any": -->before_suite_name:gsi.gsi_replica_indexes.GSIReplicaIndexesTests.suite_setUp,suite: ]> suite_setUp (unittest.loader._FailedTest) ... ERROR ====================================================================== ERROR: suite_setUp (unittest.loader._FailedTest) ---------------------------------------------------------------------- AttributeError: type object 'GSIReplicaIndexesTests' has no attribute 'suite_setUp' ---------------------------------------------------------------------- Ran 1 test in 0.001s FAILED (errors=1) -->result: test_build_index_with_replica_failover_addback (gsi.gsi_replica_indexes.GSIReplicaIndexesTests) ... 2022-02-17 12:19:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:19:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:19:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:19:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:19:57 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 12:19:57 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2318-enterprise 2022-02-17 12:19:57 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_build_index_with_replica_failover_addback============== 2022-02-17 12:19:57 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root, ip:10.112.205.104 port:8091 ssh_username:root] 2022-02-17 12:19:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:19:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:19:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2022-02-17 12:19:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.008583690987124, 'mem_free': 1207226368, 'mem_total': 1930829824, 'swap_mem_used': 22384640, 'swap_mem_total': 1107292160} 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.102:8091 => {'services': ['index'], 'cpu_utilization': 4.393305439330544, 'mem_free': 1478139904, 'mem_total': 2033594368, 'swap_mem_used': 225280, 'swap_mem_total': 1107292160} 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.103:8091 => {'services': ['index'], 'cpu_utilization': 3.611971104231166, 'mem_free': 1516048384, 'mem_total': 2033594368, 'swap_mem_used': 8192, 'swap_mem_total': 1107292160} 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.104:8091 => {'services': ['index'], 'cpu_utilization': 3.186022610483042, 'mem_free': 1524453376, 'mem_total': 2033594368, 'swap_mem_used': 163840, 'swap_mem_total': 1107292160} 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:19:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:20:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:20:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-17 12:20:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2022-02-17 12:20:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:20:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:20:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-17 12:20:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2022-02-17 12:20:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:20:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:20:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-17 12:20:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2022-02-17 12:20:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:20:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:20:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.105 with username:root, attempt#1 of 5 2022-02-17 12:20:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.105 as root 2022-02-17 12:20:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:20:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:20:11 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_build_index_with_replica_failover_addback ============== 2022-02-17 12:20:11 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 10.112.205.101 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 10.112.205.101 existing buckets : [] 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 10.112.205.101 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.105, nothing to delete 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-02-17 12:20:12 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://10.112.205.101:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103,ns_1@10.112.205.104', 'ejectedNodes': 'ns_1@10.112.205.102,ns_1@10.112.205.103,ns_1@10.112.205.104', 'user': 'Administrator', 'password': 'password'} 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2022-02-17 12:20:12 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-02-17 12:20:22 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 16.00 % 2022-02-17 12:20:42 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 30.03 seconds 2022-02-17 12:20:42 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:10.112.205.101 port:8091 ssh_username:root ? [('ns_1@10.112.205.102', 8091), ('ns_1@10.112.205.103', 8091), ('ns_1@10.112.205.104', 8091)] 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.105:8091 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.105:8091 is running 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_build_index_with_replica_failover_addback ============== 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:20:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:20:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:20:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:20:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:20:53 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.205.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 12:20:53 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-02-17 12:20:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: systemctl stop couchbase-server.service 2022-02-17 12:20:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:20:55 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:20:55 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-02-17 12:20:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-02-17 12:20:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:20:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-02-17 12:20:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:20:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:20:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:20:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:20:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:20:56 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.205.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 12:20:56 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:20:56 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-02-17 12:20:56 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-02-17 12:20:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: systemctl start couchbase-server.service 2022-02-17 12:20:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:20:56 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.205.101:sleep for 5 secs. waiting for couchbase server to come up ... 2022-02-17 12:21:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-02-17 12:21:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:01 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 22359 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-02-17 12:21:01 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:01 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.205.101: with pid 22416 2022-02-17 12:21:01 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-02-17 12:21:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-17 12:21:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2022-02-17 12:21:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-17 12:21:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2022-02-17 12:21:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:02 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.205.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 12:21:02 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-02-17 12:21:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.102: systemctl stop couchbase-server.service 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.205.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.102: systemctl start couchbase-server.service 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:04 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.205.102:sleep for 5 secs. waiting for couchbase server to come up ... 2022-02-17 12:21:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.102: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-02-17 12:21:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:09 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 6538 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-02-17 12:21:09 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:10 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.205.102: with pid 6595 2022-02-17 12:21:10 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-02-17 12:21:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-17 12:21:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2022-02-17 12:21:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-17 12:21:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2022-02-17 12:21:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:10 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.205.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 12:21:11 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-02-17 12:21:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: systemctl stop couchbase-server.service 2022-02-17 12:21:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:12 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:12 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-02-17 12:21:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-02-17 12:21:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-02-17 12:21:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-17 12:21:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2022-02-17 12:21:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:13 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.205.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 12:21:13 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:13 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-02-17 12:21:13 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-02-17 12:21:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: systemctl start couchbase-server.service 2022-02-17 12:21:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:13 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.205.103:sleep for 5 secs. waiting for couchbase server to come up ... 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 18374 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.205.103: with pid 18431 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2022-02-17 12:21:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:19 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.205.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 12:21:19 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-02-17 12:21:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: systemctl stop couchbase-server.service 2022-02-17 12:21:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:20 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.205.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: systemctl start couchbase-server.service 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:21 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.205.104:sleep for 5 secs. waiting for couchbase server to come up ... 2022-02-17 12:21:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-02-17 12:21:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:26 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 16838 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-02-17 12:21:26 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:26 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.205.104: with pid 16896 2022-02-17 12:21:26 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-02-17 12:21:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.105 with username:root, attempt#1 of 5 2022-02-17 12:21:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.105 as root 2022-02-17 12:21:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.105 with username:root, attempt#1 of 5 2022-02-17 12:21:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.105 as root 2022-02-17 12:21:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:27 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.205.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 12:21:27 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-02-17 12:21:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.105: systemctl stop couchbase-server.service 2022-02-17 12:21:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:29 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:29 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-02-17 12:21:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.105: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-02-17 12:21:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.105: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-02-17 12:21:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.105 with username:root, attempt#1 of 5 2022-02-17 12:21:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.105 as root 2022-02-17 12:21:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:30 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.205.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-17 12:21:30 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:30 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-02-17 12:21:30 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-02-17 12:21:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.105: systemctl start couchbase-server.service 2022-02-17 12:21:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:30 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.205.105:sleep for 5 secs. waiting for couchbase server to come up ... 2022-02-17 12:21:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.105: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-02-17 12:21:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:35 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 30871 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-02-17 12:21:35 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-17 12:21:35 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.205.105: with pid 30928 2022-02-17 12:21:35 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-02-17 12:21:35 | ERROR | MainProcess | test_thread | [rest_client._http_request] socket error while connecting to http://10.112.205.105:8091/nodes/self error [Errno 61] Connection refused 2022-02-17 12:21:40 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.101 port:8091 ssh_username:root, nodes/self 2022-02-17 12:21:40 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '41', 'memoryTotal': 1930829824, 'memoryFree': 1539268608, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.101:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2318-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'memcached': 11210, 'id': 'ns_1@10.112.205.101', 'ip': '10.112.205.101', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2022-02-17 12:21:40 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-17 12:21:40 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2022-02-17 12:21:40 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 10.112.205.101 2022-02-17 12:21:40 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2022-02-17 12:21:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=730 2022-02-17 12:21:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.205.101,8091,['kv', 'index', 'n1ql']) 2022-02-17 12:21:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.205.101: 8091:hostname=10.112.205.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql 2022-02-17 12:21:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-17 12:21:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.101:8091:port=8091&username=Administrator&password=password 2022-02-17 12:21:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-17 12:21:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:21:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:21:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-17 12:21:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:41 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-17 12:21:41 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.102 port:8091 ssh_username:root, nodes/self 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '31', 'memoryTotal': 2033594368, 'memoryFree': 1612623872, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.102:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2318-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'memcached': 11210, 'id': 'ns_1@10.112.205.102', 'ip': '10.112.205.102', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2022-02-17 12:21:42 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1039 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.102:8091:port=8091&username=Administrator&password=password 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.102:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.102:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.103 port:8091 ssh_username:root, nodes/self 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '26', 'memoryTotal': 2033594368, 'memoryFree': 1594937344, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.103:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2318-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'memcached': 11210, 'id': 'ns_1@10.112.205.103', 'ip': '10.112.205.103', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2022-02-17 12:21:43 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1039 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.103:8091:port=8091&username=Administrator&password=password 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2022-02-17 12:21:43 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.104 port:8091 ssh_username:root, nodes/self 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '20', 'memoryTotal': 2033594368, 'memoryFree': 1598738432, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.104:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2318-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'memcached': 11210, 'id': 'ns_1@10.112.205.104', 'ip': '10.112.205.104', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2022-02-17 12:21:44 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.104:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1039 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.104:8091:port=8091&username=Administrator&password=password 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-17 12:21:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.105 port:8091 ssh_username:root, nodes/self 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '10', 'memoryTotal': 2033594368, 'memoryFree': 1592348672, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.105:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2318-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'memcached': 11210, 'id': 'ns_1@10.112.205.105', 'ip': '10.112.205.105', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2022-02-17 12:21:45 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.105:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1039 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.105:8091:port=8091&username=Administrator&password=password 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.105 with username:root, attempt#1 of 5 2022-02-17 12:21:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.105 as root 2022-02-17 12:21:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:21:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:21:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.105: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-17 12:21:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:21:46 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-17 12:21:46 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.105:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 12:21:46 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.105:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-17 12:21:46 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-17 12:21:46 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.205.101 **** 2022-02-17 12:21:46 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://10.112.205.101:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-02-17 12:21:46 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-02-17 12:21:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-02-17 12:21:52 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-02-17 12:21:52 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2022-02-17 12:21:52 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.102:8091 to cluster 2022-02-17 12:21:52 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.102:18091 to this cluster @10.112.205.101:8091 2022-02-17 12:22:02 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2022-02-17 12:22:02 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-02-17 12:22:16 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.103:8091 to cluster 2022-02-17 12:22:16 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.103:18091 to this cluster @10.112.205.101:8091 2022-02-17 12:22:26 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2022-02-17 12:22:26 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-02-17 12:22:40 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.104:8091 to cluster 2022-02-17 12:22:40 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.104:18091 to this cluster @10.112.205.101:8091 2022-02-17 12:22:50 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2022-02-17 12:22:50 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-02-17 12:23:03 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 10.112.205.102 not part of cluster inactiveAdded 2022-02-17 12:23:03 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 10.112.205.103 not part of cluster inactiveAdded 2022-02-17 12:23:03 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 10.112.205.104 not part of cluster inactiveAdded 2022-02-17 12:23:03 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103,ns_1@10.112.205.104', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-02-17 12:23:03 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-02-17 12:23:13 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-02-17 12:23:13 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.012335300445557 sec 2022-02-17 12:23:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:23:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:23:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:23:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:23:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-17 12:23:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-02-17 12:23:14 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-17 12:23:14 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.205.101 2022-02-17 12:23:14 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.205.101:8091/pools/default/buckets with param: name=default&ramQuotaMB=730&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-02-17 12:23:15 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket default 2022-02-17 12:23:15 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 10.112.205.101 to accept set ops 2022-02-17 12:23:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 default 2022-02-17 12:23:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 default 2022-02-17 12:23:17 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 730 2022-02-17 12:23:17 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_build_index_with_replica_failover_addback ============== 2022-02-17 12:23:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:23:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:23:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:23:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:23:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-17 12:23:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2022-02-17 12:23:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.105 with username:root, attempt#1 of 5 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.105 as root 2022-02-17 12:23:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:23:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 48.40909090909091, 'mem_free': 1372876800, 'mem_total': 1930829824, 'swap_mem_used': 22368256, 'swap_mem_total': 1107292160} 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.102:8091 => {'services': ['index'], 'cpu_utilization': 3.696098562628337, 'mem_free': 1568747520, 'mem_total': 2033594368, 'swap_mem_used': 221184, 'swap_mem_total': 1107292160} 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.103:8091 => {'services': ['index'], 'cpu_utilization': 7.604166666666667, 'mem_free': 1575030784, 'mem_total': 2033594368, 'swap_mem_used': 8192, 'swap_mem_total': 1107292160} 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.104:8091 => {'services': ['index'], 'cpu_utilization': 3.381147540983607, 'mem_free': 1588101120, 'mem_total': 2033594368, 'swap_mem_used': 155648, 'swap_mem_total': 1107292160} 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 211.37273502349854 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:23:28 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.101 cluster is healthy 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.101 is healthy 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.102 cluster is healthy 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.102 is healthy 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.103 cluster is healthy 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.103 is healthy 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.104 cluster is healthy 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.104 is healthy 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.105 cluster is healthy 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.105 is healthy 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root, ip:10.112.205.104 port:8091 ssh_username:root] 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2022-02-17 12:23:29 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-02-17 12:23:34 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2022-02-17 12:23:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:23:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:23:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:23:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:23:34 | INFO | MainProcess | test_thread | [basetestcase.load] create 1000.0 to default documents... 2022-02-17 12:23:34 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 default 2022-02-17 12:23:37 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2022-02-17 12:23:37 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2022-02-17 12:23:37 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:10.112.205.101 port:8091 ssh_username:root 2022-02-17 12:23:37 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2022-02-17 12:24:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:24:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:24:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:24:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:24:08 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root, ip:10.112.205.104 port:8091 ssh_username:root] 2022-02-17 12:24:08 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=http://10.112.205.101:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False 2022-02-17 12:24:08 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2022-02-17 12:24:08 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=http://10.112.205.101:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False 2022-02-17 12:24:08 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2022-02-17 12:24:08 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2022-02-17 12:24:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-17 12:24:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2022-02-17 12:24:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-17 12:24:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-17 12:24:13 | INFO | MainProcess | test_thread | [gsi_replica_indexes.test_build_index_with_replica_failover_addback] ['10.112.205.101:8091', '10.112.205.102:8091', '10.112.205.103:8091'] 2022-02-17 12:24:18 | INFO | MainProcess | test_thread | [gsi_replica_indexes.test_build_index_with_replica_failover_addback] CREATE INDEX random_index_270666 ON default(age) USING GSI WITH {'nodes': ['10.112.205.101:8091', '10.112.205.102:8091', '10.112.205.103:8091'], 'defer_build': true}; 2022-02-17 12:24:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX random_index_270666 ON default(age) USING GSI WITH {'nodes': ['10.112.205.101:8091', '10.112.205.102:8091', '10.112.205.103:8091'], 'defer_build': true}; 2022-02-17 12:24:22 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+random_index_270666+ON+default%28age%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.205.101%3A8091%27%2C+%2710.112.205.102%3A8091%27%2C+%2710.112.205.103%3A8091%27%5D%2C+%27defer_build%27%3A+true%7D%3B 2022-02-17 12:24:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 579.142576ms 2022-02-17 12:24:26 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2022-02-17 12:25:04 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=http://10.112.205.101:8091/indexStatus verb=GET params= client_cert=None verify=False 2022-02-17 12:25:12 | INFO | MainProcess | test_thread | [gsi_replica_indexes.test_build_index_with_replica_failover_addback] {'default': {'random_index_270666': {'status': 'Created', 'progress': '0', 'definition': 'CREATE INDEX `random_index_270666` ON `default`(`age`) WITH { "defer_build":true, "nodes":[ "10.112.205.101:8091","10.112.205.102:8091","10.112.205.103:8091" ], "num_replica":2 }', 'hosts': '10.112.205.103:8091', 'id': 5571552157608950059}, 'random_index_270666 (replica 1)': {'status': 'Created', 'progress': '0', 'definition': 'CREATE INDEX `random_index_270666` ON `default`(`age`) WITH { "defer_build":true, "nodes":[ "10.112.205.101:8091","10.112.205.102:8091","10.112.205.103:8091" ], "num_replica":2 }', 'hosts': '10.112.205.102:8091', 'id': 5571552157608950059}, 'random_index_270666 (replica 2)': {'status': 'Created', 'progress': '0', 'definition': 'CREATE INDEX `random_index_270666` ON `default`(`age`) WITH { "defer_build":true, "nodes":[ "10.112.205.101:8091","10.112.205.102:8091","10.112.205.103:8091" ], "num_replica":2 }', 'hosts': '10.112.205.101:8091', 'id': 5571552157608950059}}} 2022-02-17 12:25:47 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes] Hostnames : 10.112.205.103:8091 , 10.112.205.102:8091 2022-02-17 12:25:47 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes] Index IDs : 5571552157608950059, 5571552157608950059 2022-02-17 12:25:47 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes] Hostnames : 10.112.205.103:8091 , 10.112.205.101:8091 2022-02-17 12:25:47 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes] Index IDs : 5571552157608950059, 5571552157608950059 2022-02-17 12:25:56 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY select distinct(name) from system:indexes where `using`='gsi' 2022-02-17 12:25:56 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+distinct%28name%29+from+system%3Aindexes+where+%60using%60%3D%27gsi%27 2022-02-17 12:25:56 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 11.939444ms 2022-02-17 12:25:56 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes_build_status] index_name = random_index_270666, defer_build = True, index_status = Created 2022-02-17 12:25:56 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes_build_status] index_name = random_index_270666 (replica 1), defer_build = True, index_replica_status = Created 2022-02-17 12:25:56 | INFO | MainProcess | test_thread | [tuq_helper.verify_replica_indexes_build_status] index_name = random_index_270666 (replica 2), defer_build = True, index_replica_status = Created 2022-02-17 12:26:04 | INFO | MainProcess | Cluster_Thread | [task._failover_nodes] Failing over 10.112.205.103:8091 with graceful=False 2022-02-17 12:26:04 | INFO | MainProcess | Cluster_Thread | [rest_client.fail_over] fail_over node ns_1@10.112.205.103 successful 2022-02-17 12:26:04 | INFO | MainProcess | Cluster_Thread | [task.execute] 180 seconds sleep after failover, for nodes to go pending.... 2022-02-17 12:29:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on `default`(random_index_270666) 2022-02-17 12:29:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+%60default%60%28random_index_270666%29 2022-02-17 12:29:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 139.656339ms 2022-02-17 12:29:50 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2022-02-17 12:31:17 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 10.112.205.103 not part of cluster inactiveFailed 2022-02-17 12:31:39 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] Going to set recoveryType=full for node :: ns_1@10.112.205.103 2022-02-17 12:31:39 | INFO | MainProcess | test_thread | [rest_client.set_recovery_type] recoveryType for node ns_1@10.112.205.103 set successful 2022-02-17 12:31:40 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@10.112.205.103 successful 2022-02-17 12:31:43 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 10.112.205.103 not part of cluster inactiveAdded 2022-02-17 12:31:43 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103,ns_1@10.112.205.104', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-02-17 12:31:43 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-02-17 12:31:43 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-02-17 12:31:49 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 6.021121025085449 seconds 2022-02-17 12:31:53 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-02-17 12:31:53 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.01401400566101 sec 2022-02-17 12:34:58 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 180 secs. ... 2022-02-17 12:40:34 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=http://10.112.205.101:8091/indexStatus verb=GET params= client_cert=None verify=False Process finished with exit code 129 (interrupted by signal 1: SIGHUP)