tions/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 63546 --file /Users/hemantrajput/testrunner/testrunner.py -i b/resources/tmp.ini -p java_sdk_client=True -c conf/gsi/collection-planner-gsi.conf Connected to pydev debugger (build 212.5080.64) /Users/hemantrajput/testrunner/./lib/remote/remote_util.py:4273: SyntaxWarning: "is not" with a literal. Did you mean "!="? if command_options is not '': /Users/hemantrajput/testrunner/lib/remote/remote_util.py:4273: SyntaxWarning: "is not" with a literal. Did you mean "!="? if command_options is not '': INFO:root:__main__ *** TestRunner *** INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:Conf filename: conf/gsi/collection-planner-gsi.conf INFO:__main__:Test prefix: gsi.planner_gsi.PlannerGSI INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: {'cluster_name': 'tmp', 'conf_file': 'conf/gsi/collection-planner-gsi.conf', 'ini': 'b/resources/tmp.ini', 'java_sdk_client': 'True', 'num_nodes': 4, 'spec': 'collection-planner-gsi'} INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: gsi.planner_gsi.PlannerGSI.test_index_placements_with_skewed_load,default_bucket=false,force_clean=true,defer_build=False,nodes_init=3,services_init=kv:index:n1ql-index-index,initial_index_num=5,GROUP=P0 INFO:__main__:Logs folder: /Users/hemantrajput/testrunner/logs/testrunner-21-Sep-13_14-39-08/test_1 Logs will be stored at /Users/hemantrajput/testrunner/logs/testrunner-21-Sep-13_14-39-08/test_1 ./testrunner -i b/resources/tmp.ini -p java_sdk_client=True -t gsi.planner_gsi.PlannerGSI.test_index_placements_with_skewed_load,default_bucket=false,force_clean=true,defer_build=False,nodes_init=3,services_init=kv:index:n1ql-index-index,initial_index_num=5,GROUP=P0 Test Input params: {'default_bucket': 'false', 'force_clean': 'true', 'defer_build': 'False', 'nodes_init': '3', 'services_init': 'kv:index:n1ql-index-index', 'initial_index_num': '5', 'GROUP': 'P0', 'ini': 'b/resources/tmp.ini', 'cluster_name': 'tmp', 'spec': 'collection-planner-gsi', 'conf_file': 'conf/gsi/collection-planner-gsi.conf', 'java_sdk_client': 'True', 'num_nodes': 4, 'case_number': 1, 'logs_folder': '/Users/hemantrajput/testrunner/logs/testrunner-21-Sep-13_14-39-08/test_1'} Run before suite setup for gsi.planner_gsi.PlannerGSI.test_index_placements_with_skewed_load /Users/hemantrajput/testrunner/lib/tasks/task.py:299: SyntaxWarning: "is" with a literal. Did you mean "=="? if bucket_params['bucket_priority'] is None or bucket_params['bucket_priority'].lower() is 'low': /Users/hemantrajput/testrunner/./lib/membase/helper/cluster_helper.py:376: SyntaxWarning: "is" with a literal. Did you mean "=="? if index_path is '': /Users/hemantrajput/testrunner/pytests/basetestcase.py:1515: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.stats_monitor is not '': /Users/hemantrajput/testrunner/pytests/basetestcase.py:1517: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.warmup_stats_monitor is not '': /Users/hemantrajput/testrunner/lib/membase/helper/cluster_helper.py:376: SyntaxWarning: "is" with a literal. Did you mean "=="? if index_path is '': /Users/hemantrajput/testrunner/lib/couchbase_helper/tuq_helper.py:219: SyntaxWarning: "is" with a literal. Did you mean "=="? if status is "any": /Users/hemantrajput/testrunner/lib/couchbase_helper/tuq_helper.py:229: SyntaxWarning: "is" with a literal. Did you mean "=="? if status is "any": -->before_suite_name:gsi.planner_gsi.PlannerGSI.suite_setUp,suite: ]> suite_setUp (gsi.planner_gsi.PlannerGSI) ... 2021-09-13 14:39:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:39:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:39:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:39:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:39:09 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:39:09 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.0.2-6653-enterprise 2021-09-13 14:39:09 | INFO | MainProcess | MainThread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.2-6653-enterprise', '7.0.2-6653-enterprise', '7.0.2-6653-enterprise'] 2021-09-13 14:39:09 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2021-09-13 14:39:09 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket'] on 10.112.205.101 2021-09-13 14:39:11 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-09-13 14:39:11 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.205.101 existing buckets : [] 2021-09-13 14:39:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket from 10.112.205.101 2021-09-13 14:39:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2021-09-13 14:39:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2021-09-13 14:39:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2021-09-13 14:39:11 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:39:11 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-09-13 14:39:11 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103', 'ejectedNodes': 'ns_1@10.112.205.102,ns_1@10.112.205.103', 'user': 'Administrator', 'password': 'password'} 2021-09-13 14:39:11 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2021-09-13 14:39:11 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-09-13 14:39:31 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 20.01 seconds 2021-09-13 14:39:31 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:10.112.205.101 port:8091 ssh_username:root ? [('ns_1@10.112.205.102', 8091), ('ns_1@10.112.205.103', 8091)] 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2021-09-13 14:39:41 | INFO | MainProcess | MainThread | [basetestcase.setUp] Building docker image with java sdk client 2021-09-13 14:39:45 | INFO | MainProcess | MainThread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has started ============== 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-09-13 14:39:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 1.950718685831622, 'mem_free': 1252155392, 'mem_total': 1930829824, 'swap_mem_used': 26312704, 'swap_mem_total': 1107292160} 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:39:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-09-13 14:39:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-09-13 14:39:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:39:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:39:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-09-13 14:39:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-09-13 14:39:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:39:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.101, nothing to delete 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has completed ============== 2021-09-13 14:39:55 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2021-09-13 14:39:56 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.101 port:8091 ssh_username:root, nodes/self 2021-09-13 14:39:56 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '684', 'memoryTotal': 1930829824, 'memoryFree': 1251549184, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6653-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 730, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.101', 'ip': '10.112.205.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 1841, 'curr_items': 1000} 2021-09-13 14:39:56 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-09-13 14:39:56 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 10.112.205.101 2021-09-13 14:39:56 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-09-13 14:39:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=730 2021-09-13 14:39:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.205.101,8091,['kv', 'index', 'n1ql']) 2021-09-13 14:39:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.205.101: 8091:hostname=10.112.205.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql 2021-09-13 14:39:56 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://10.112.205.101:8091//node/controller/setupServices body: hostname=10.112.205.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password 2021-09-13 14:39:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.101:8091:port=8091&username=Administrator&password=password 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.102 port:8091 ssh_username:root, nodes/self 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '684', 'memoryTotal': 1930829824, 'memoryFree': 1540571136, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.102:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6653-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.102', 'ip': '10.112.205.102', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-09-13 14:39:57 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.102:8091:port=8091&username=Administrator&password=password 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-09-13 14:39:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-09-13 14:39:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-09-13 14:39:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:39:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:39:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.103 port:8091 ssh_username:root, nodes/self 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '684', 'memoryTotal': 2033594368, 'memoryFree': 1631014912, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.103:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6653-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.103', 'ip': '10.112.205.103', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2021-09-13 14:39:59 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1039 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.103:8091:port=8091&username=Administrator&password=password 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-09-13 14:39:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.104 port:8091 ssh_username:root, nodes/self 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '685', 'memoryTotal': 1930829824, 'memoryFree': 1526239232, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.104:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6653-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 986, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.205.104', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.104:8091:port=8091&username=Administrator&password=password 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-09-13 14:40:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-09-13 14:40:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-09-13 14:40:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:40:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:40:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-09-13 14:40:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:40:02 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-09-13 14:40:02 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:40:02 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:40:02 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-09-13 14:40:02 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.205.101 **** 2021-09-13 14:40:02 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-09-13 14:40:07 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-09-13 14:40:07 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2021-09-13 14:40:08 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.102:8091 to cluster 2021-09-13 14:40:08 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.102:8091 to this cluster @10.112.205.101:8091 2021-09-13 14:40:18 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-09-13 14:40:18 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-09-13 14:40:31 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.103:8091 to cluster 2021-09-13 14:40:31 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.103:8091 to this cluster @10.112.205.101:8091 2021-09-13 14:40:41 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-09-13 14:40:41 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-09-13 14:40:55 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-09-13 14:40:55 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-09-13 14:41:05 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-09-13 14:41:05 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.014174938201904 sec 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.205.101 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ============== 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:41:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:41:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:41:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-09-13 14:41:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-09-13 14:41:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:41:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:41:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-09-13 14:41:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-09-13 14:41:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:41:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:41:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-09-13 14:41:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-09-13 14:41:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:41:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 10.7103825136612, 'mem_free': 1294815232, 'mem_total': 1930829824, 'swap_mem_used': 26312704, 'swap_mem_total': 1107292160} 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.102:8091 => {'services': ['index'], 'cpu_utilization': 2.256410256410256, 'mem_free': 1496313856, 'mem_total': 1930829824, 'swap_mem_used': 167936, 'swap_mem_total': 1107292160} 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.103:8091 => {'services': ['index'], 'cpu_utilization': 1.221995926680244, 'mem_free': 1622806528, 'mem_total': 2033594368, 'swap_mem_used': 253952, 'swap_mem_total': 1107292160} 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.205.101 cluster is healthy 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.205.101 is healthy 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.205.102 cluster is healthy 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.205.102 is healthy 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.205.103 cluster is healthy 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.205.103 is healthy 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.205.104 cluster is healthy 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.205.104 is healthy 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2021-09-13 14:41:13 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-09-13 14:41:18 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2021-09-13 14:41:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:41:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:41:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:41:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:41:26 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED 2021-09-13 14:41:26 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-09-13 14:41:26 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:10.112.205.101 port:8091 ssh_username:root 2021-09-13 14:41:26 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs. ... 2021-09-13 14:41:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:41:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:41:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:41:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:41:56 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-09-13 14:41:56 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2021-09-13 14:41:56 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2021-09-13 14:41:56 | INFO | MainProcess | MainThread | [planner_gsi.setUp] ============== PlannerGSI setup has started ============== 2021-09-13 14:41:56 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.205.101:8091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=486&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-09-13 14:41:56 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket test_bucket 2021-09-13 14:41:56 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:41:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:57 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-09-13 14:41:57 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:41:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:57 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-09-13 14:41:57 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:41:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:58 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:58 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-09-13 14:41:58 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:41:58 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:58 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:58 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-09-13 14:41:58 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:41:58 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:58 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:58 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-09-13 14:41:58 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:41:58 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:59 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:41:59 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [planner_gsi.setUp] ============== PlannerGSI setup has completed ============== 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has started ============== 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-09-13 14:41:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 2.059732234809475, 'mem_free': 1297948672, 'mem_total': 1930829824, 'swap_mem_used': 26312704, 'swap_mem_total': 1107292160} 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.102:8091 => {'services': ['index'], 'cpu_utilization': 1.333333333333333, 'mem_free': 1498427392, 'mem_total': 1930829824, 'swap_mem_used': 167936, 'swap_mem_total': 1107292160} 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.103:8091 => {'services': ['index'], 'cpu_utilization': 1.642710472279261, 'mem_free': 1598365696, 'mem_total': 2033594368, 'swap_mem_used': 253952, 'swap_mem_total': 1107292160} 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-09-13 14:42:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:42:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:42:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-09-13 14:42:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-09-13 14:42:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-09-13 14:42:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-09-13 14:42:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-09-13 14:42:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-09-13 14:42:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket'] on 10.112.205.101 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.205.101 existing buckets : [] 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket from 10.112.205.101 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103', 'ejectedNodes': 'ns_1@10.112.205.102,ns_1@10.112.205.103', 'user': 'Administrator', 'password': 'password'} 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2021-09-13 14:42:09 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-09-13 14:42:29 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 20.04 seconds 2021-09-13 14:42:29 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:10.112.205.101 port:8091 ssh_username:root ? [('ns_1@10.112.205.102', 8091), ('ns_1@10.112.205.103', 8091)] 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-09-13 14:42:39 | INFO | MainProcess | MainThread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has completed ============== ok ---------------------------------------------------------------------- Ran 1 test in 211.154s OK -->result: test_index_placements_with_skewed_load (gsi.planner_gsi.PlannerGSI) ... 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.0.2-6653-enterprise 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.2-6653-enterprise'] 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_index_placements_with_skewed_load============== 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.101, nothing to delete 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2021-09-13 14:42:40 | INFO | MainProcess | test_thread | [basetestcase.setUp] Building docker image with java sdk client 2021-09-13 14:42:45 | INFO | MainProcess | test_thread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has started ============== 2021-09-13 14:42:45 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-09-13 14:42:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:42:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:42:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 1.750772399588054, 'mem_free': 1296252928, 'mem_total': 1930829824, 'swap_mem_used': 26312704, 'swap_mem_total': 1107292160} 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:42:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-09-13 14:42:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-09-13 14:42:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-09-13 14:42:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-09-13 14:42:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-09-13 14:42:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-09-13 14:42:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_index_placements_with_skewed_load ============== 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.101, nothing to delete 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_index_placements_with_skewed_load ============== 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has completed ============== 2021-09-13 14:42:53 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.101 port:8091 ssh_username:root, nodes/self 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '858', 'memoryTotal': 1930829824, 'memoryFree': 1309118464, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6653-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 730, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.101', 'ip': '10.112.205.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 10.112.205.101 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=730 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.205.101,8091,['kv', 'index', 'n1ql']) 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.205.101: 8091:hostname=10.112.205.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql 2021-09-13 14:42:54 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://10.112.205.101:8091//node/controller/setupServices body: hostname=10.112.205.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.101:8091:port=8091&username=Administrator&password=password 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-09-13 14:42:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.102 port:8091 ssh_username:root, nodes/self 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '863', 'memoryTotal': 1930829824, 'memoryFree': 1544986624, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.102:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6653-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.102', 'ip': '10.112.205.102', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-09-13 14:42:55 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.102:8091:port=8091&username=Administrator&password=password 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-09-13 14:42:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.103 port:8091 ssh_username:root, nodes/self 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '863', 'memoryTotal': 2033594368, 'memoryFree': 1631535104, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.103:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6653-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.103', 'ip': '10.112.205.103', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2021-09-13 14:42:56 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1039 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.103:8091:port=8091&username=Administrator&password=password 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-09-13 14:42:56 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.104 port:8091 ssh_username:root, nodes/self 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '867', 'memoryTotal': 1930829824, 'memoryFree': 1525469184, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.104:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6653-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 986, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.205.104', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.104:8091:port=8091&username=Administrator&password=password 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-09-13 14:42:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-09-13 14:42:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-09-13 14:42:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:42:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:42:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-09-13 14:42:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:42:58 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-09-13 14:42:58 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:42:58 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-09-13 14:42:58 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-09-13 14:42:58 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.205.101 **** 2021-09-13 14:42:58 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-09-13 14:43:03 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-09-13 14:43:03 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2021-09-13 14:43:04 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.102:8091 to cluster 2021-09-13 14:43:04 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.102:8091 to this cluster @10.112.205.101:8091 2021-09-13 14:43:14 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-09-13 14:43:14 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-09-13 14:43:27 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.103:8091 to cluster 2021-09-13 14:43:27 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.103:8091 to this cluster @10.112.205.101:8091 2021-09-13 14:43:37 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-09-13 14:43:37 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-09-13 14:43:50 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-09-13 14:43:50 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-09-13 14:44:00 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-09-13 14:44:00 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.011132955551147 sec 2021-09-13 14:44:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:44:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:44:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.205.101 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_index_placements_with_skewed_load ============== 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-09-13 14:44:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:44:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:44:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-09-13 14:44:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-09-13 14:44:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:44:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:44:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-09-13 14:44:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-09-13 14:44:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:44:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 8.945686900958467, 'mem_free': 1306001408, 'mem_total': 1930829824, 'swap_mem_used': 26312704, 'swap_mem_total': 1107292160} 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.102:8091 => {'services': ['index'], 'cpu_utilization': 7.790821771611526, 'mem_free': 1495846912, 'mem_total': 1930829824, 'swap_mem_used': 167936, 'swap_mem_total': 1107292160} 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.103:8091 => {'services': ['index'], 'cpu_utilization': 1.217038539553753, 'mem_free': 1623404544, 'mem_total': 2033594368, 'swap_mem_used': 245760, 'swap_mem_total': 1107292160} 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.101 cluster is healthy 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.101 is healthy 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.102 cluster is healthy 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.102 is healthy 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.103 cluster is healthy 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.103 is healthy 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.104 cluster is healthy 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.104 is healthy 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2021-09-13 14:44:08 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-09-13 14:44:13 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2021-09-13 14:44:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:44:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:44:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:44:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:44:21 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2021-09-13 14:44:21 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-09-13 14:44:21 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:10.112.205.101 port:8091 ssh_username:root 2021-09-13 14:44:21 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2021-09-13 14:44:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:44:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:44:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:44:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:44:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-09-13 14:44:52 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2021-09-13 14:44:52 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2021-09-13 14:44:52 | INFO | MainProcess | test_thread | [planner_gsi.setUp] ============== PlannerGSI setup has started ============== 2021-09-13 14:44:52 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.205.101:8091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=486&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-09-13 14:44:52 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.04 seconds to create bucket test_bucket 2021-09-13 14:44:52 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:44:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:53 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-09-13 14:44:53 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:44:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:54 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-09-13 14:44:54 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:44:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:54 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-09-13 14:44:54 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:44:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:55 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-09-13 14:44:55 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:44:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:55 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-09-13 14:44:55 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-09-13 14:44:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-09-13 14:44:55 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-09-13 14:44:55 | INFO | MainProcess | test_thread | [planner_gsi.setUp] ============== PlannerGSI setup has completed ============== 2021-09-13 14:44:55 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-09-13 14:44:55 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.205.101:8091/pools/default/buckets/test_bucket/scopes with params: name=test_scope_1 2021-09-13 14:44:55 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created test_bucket->test_scope_1 {'uid': '1'} 2021-09-13 14:44:56 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.205.101:8091/pools/default/buckets/test_bucket/scopes/test_scope_1/collections with params: name=test_collection_1 2021-09-13 14:44:56 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created test_bucket->test_scope_1->test_collection_1 manifest:{'uid': '2'} 2021-09-13 14:44:56 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. Allowing time after collection creation ... 2021-09-13 14:45:06 | INFO | MainProcess | test_thread | [basetestcase.data_ops_javasdk_loader_in_batches] Number of batches : 1 2021-09-13 14:45:06 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2021-09-13 14:45:16 | INFO | MainProcess | test_thread | [basetestcase.data_ops_javasdk_loader_in_batches] done 2021-09-13 14:45:16 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.205.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False 2021-09-13 14:45:21 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx_0` ON default:test_bucket.test_scope_1.test_collection_1(age) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-09-13 14:45:21 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx_0%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28age%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-09-13 14:45:25 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.751905369s 2021-09-13 14:45:30 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx_1` ON default:test_bucket.test_scope_1.test_collection_1(city) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-09-13 14:45:30 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx_1%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28city%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-09-13 14:45:32 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.623951886s 2021-09-13 14:45:32 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx_2` ON default:test_bucket.test_scope_1.test_collection_1(country) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-09-13 14:45:32 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx_2%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28country%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-09-13 14:45:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.905674234s 2021-09-13 14:45:40 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx_3` ON default:test_bucket.test_scope_1.test_collection_1(title) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-09-13 14:45:40 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx_3%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-09-13 14:45:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.644152313s 2021-09-13 14:45:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx_4` ON default:test_bucket.test_scope_1.test_collection_1(firstName) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-09-13 14:45:44 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx_4%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-09-13 14:45:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.352432801s 2021-09-13 14:45:51 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2021-09-13 14:46:01 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY ALTER INDEX idx_1 ON default:test_bucket.test_scope_1.test_collection_1 WITH {"action": "drop_replica", "replicaId": 0}; 2021-09-13 14:46:01 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=ALTER+INDEX+idx_1+ON+default%3Atest_bucket.test_scope_1.test_collection_1+WITH+%7B%22action%22%3A+%22drop_replica%22%2C+%22replicaId%22%3A+0%7D%3B 2021-09-13 14:46:01 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 96.776561ms 2021-09-13 14:46:01 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY ALTER INDEX idx_2 ON default:test_bucket.test_scope_1.test_collection_1 WITH {"action": "drop_replica", "replicaId": 0}; 2021-09-13 14:46:01 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=ALTER+INDEX+idx_2+ON+default%3Atest_bucket.test_scope_1.test_collection_1+WITH+%7B%22action%22%3A+%22drop_replica%22%2C+%22replicaId%22%3A+0%7D%3B 2021-09-13 14:46:01 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 154.238975ms 2021-09-13 14:46:01 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY ALTER INDEX idx_3 ON default:test_bucket.test_scope_1.test_collection_1 WITH {"action": "drop_replica", "replicaId": 1}; 2021-09-13 14:46:01 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=ALTER+INDEX+idx_3+ON+default%3Atest_bucket.test_scope_1.test_collection_1+WITH+%7B%22action%22%3A+%22drop_replica%22%2C+%22replicaId%22%3A+1%7D%3B 2021-09-13 14:46:01 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 132.373545ms 2021-09-13 14:46:01 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 60 secs. Giving some time for clean-up after replicas drop ... 2021-09-13 14:47:01 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-09-13 14:47:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:47:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:47:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:47:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:47:01 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loaded_index_node] Deleting index_plan.log from Remote host 2021-09-13 14:47:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: rm -rf /tmp/index_plan.log 2021-09-13 14:47:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:47:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: /opt/couchbase/bin/cbindexplan -command=retrieve -cluster="127.0.0.1:8091" -username="Administrator" -password="password" -getUsage -numNewReplica 1 -output /tmp/index_plan.log 2021-09-13 14:47:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:47:02 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loaded_index_node] Deleting index_plan.log from slave 2021-09-13 14:47:02 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loaded_index_node] Deleting index_plan.log from Remote host after copying 2021-09-13 14:47:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: rm -rf /tmp/index_plan.log 2021-09-13 14:47:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:47:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-09-13 14:47:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-09-13 14:47:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-09-13 14:47:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-13 14:47:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-09-13 14:47:14 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loaded_index_node] Deleting index_plan.log from Remote host 2021-09-13 14:47:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: rm -rf /tmp/index_plan.log 2021-09-13 14:47:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:47:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: /opt/couchbase/bin/cbindexplan -command=retrieve -cluster="127.0.0.1:8091" -username="Administrator" -password="password" -getUsage -numNewReplica 2 -output /tmp/index_plan.log 2021-09-13 14:47:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:47:14 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loaded_index_node] Deleting index_plan.log from slave 2021-09-13 14:47:14 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loaded_index_node] Deleting index_plan.log from Remote host after copying 2021-09-13 14:47:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: rm -rf /tmp/index_plan.log 2021-09-13 14:47:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-09-13 14:49:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `new_idx_0` ON default:test_bucket.test_scope_1.test_collection_1(age) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-09-13 14:49:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60new_idx_0%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28age%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-09-13 14:49:24 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.600406609s Process finished with exit code 129 (interrupted by signal 1: SIGHUP)