/Users/hemantrajput/venv3/bin/python "/private/var/folders/kc/nmmr85kn0856922bf9hb66vc0000gn/T/AppTranslocation/8C9D545D-67B3-491D-B988-AD4FF4205F6B/d/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py" --cmd-line --multiproc --qt-support=auto --client 127.0.0.1 --port 59450 --file /Users/hemantrajput/testrunner/testrunner.py -i b/resources/tmp.ini -p java_sdk_client=True -c conf/gsi/collection-planner-gsi.conf Connected to pydev debugger (build 212.4746.96) /Users/hemantrajput/testrunner/./lib/remote/remote_util.py:4242: SyntaxWarning: "is not" with a literal. Did you mean "!="? if command_options is not '': /Users/hemantrajput/testrunner/lib/remote/remote_util.py:4242: SyntaxWarning: "is not" with a literal. Did you mean "!="? if command_options is not '': INFO:root:__main__ *** TestRunner *** INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:Conf filename: conf/gsi/collection-planner-gsi.conf INFO:__main__:Test prefix: gsi.planner_gsi.PlannerGSI INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: gsi.planner_gsi.PlannerGSI.test_index_placements_with_skewed_load,default_bucket=false,force_clean=true,defer_build=False,nodes_init=3,services_init=kv:index:n1ql-index-index,initial_index_num=5,num_replica=2,GROUP=P0 INFO:__main__:Logs folder: /Users/hemantrajput/testrunner/logs/testrunner-21-Aug-13_14-22-51/test_1 {'cluster_name': 'tmp', 'conf_file': 'conf/gsi/collection-planner-gsi.conf', 'ini': 'b/resources/tmp.ini', 'java_sdk_client': 'True', 'num_nodes': 4, 'spec': 'collection-planner-gsi'} Logs will be stored at /Users/hemantrajput/testrunner/logs/testrunner-21-Aug-13_14-22-51/test_1 ./testrunner -i b/resources/tmp.ini -p java_sdk_client=True -t gsi.planner_gsi.PlannerGSI.test_index_placements_with_skewed_load,default_bucket=false,force_clean=true,defer_build=False,nodes_init=3,services_init=kv:index:n1ql-index-index,initial_index_num=5,num_replica=2,GROUP=P0 Test Input params: {'default_bucket': 'false', 'force_clean': 'true', 'defer_build': 'False', 'nodes_init': '3', 'services_init': 'kv:index:n1ql-index-index', 'initial_index_num': '5', 'num_replica': '2', 'GROUP': 'P0', 'ini': 'b/resources/tmp.ini', 'cluster_name': 'tmp', 'spec': 'collection-planner-gsi', 'conf_file': 'conf/gsi/collection-planner-gsi.conf', 'java_sdk_client': 'True', 'num_nodes': 4, 'case_number': 1, 'logs_folder': '/Users/hemantrajput/testrunner/logs/testrunner-21-Aug-13_14-22-51/test_1'} Run before suite setup for gsi.planner_gsi.PlannerGSI.test_index_placements_with_skewed_load /Users/hemantrajput/testrunner/pytests/basetestcase.py:1495: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.stats_monitor is not '': /Users/hemantrajput/testrunner/pytests/basetestcase.py:1497: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.warmup_stats_monitor is not '': -->before_suite_name:gsi.planner_gsi.PlannerGSI.suite_setUp,suite: ]> 2021-08-13 14:22:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 suite_setUp (gsi.planner_gsi.PlannerGSI) ... 2021-08-13 14:22:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:22:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:22:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:22:52 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:22:52 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.0.2-6503-enterprise 2021-08-13 14:22:52 | INFO | MainProcess | MainThread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.2-6503-enterprise', '7.0.2-6503-enterprise', '7.0.2-6503-enterprise'] 2021-08-13 14:22:52 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2021-08-13 14:22:52 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket'] on 10.112.205.101 2021-08-13 14:22:53 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-08-13 14:22:53 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.205.101 existing buckets : [] 2021-08-13 14:22:53 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket from 10.112.205.101 2021-08-13 14:22:53 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2021-08-13 14:22:53 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2021-08-13 14:22:53 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2021-08-13 14:22:53 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:22:53 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-08-13 14:22:53 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103', 'ejectedNodes': 'ns_1@10.112.205.102,ns_1@10.112.205.103', 'user': 'Administrator', 'password': 'password'} 2021-08-13 14:22:53 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2021-08-13 14:22:53 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-08-13 14:23:13 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 20.02 seconds 2021-08-13 14:23:13 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-08-13 14:23:23 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:10.112.205.101 port:8091 ssh_username:root ? [('ns_1@10.112.205.102', 8091), ('ns_1@10.112.205.103', 8091)] 2021-08-13 14:23:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2021-08-13 14:23:23 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:23:24 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2021-08-13 14:23:24 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2021-08-13 14:23:24 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:23:24 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2021-08-13 14:23:24 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2021-08-13 14:23:24 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:23:24 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2021-08-13 14:23:24 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2021-08-13 14:23:24 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:23:24 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2021-08-13 14:23:24 | INFO | MainProcess | MainThread | [basetestcase.setUp] Building docker image with java sdk client 2021-08-13 14:23:29 | INFO | MainProcess | MainThread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has started ============== 2021-08-13 14:23:29 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-08-13 14:23:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:23:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:23:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:23:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:23:30 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-08-13 14:23:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-08-13 14:23:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:23:30 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-08-13 14:23:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:23:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:23:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:23:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:23:30 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-08-13 14:23:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-08-13 14:23:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:23:31 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-08-13 14:23:31 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 2.040816326530612, 'mem_free': 1159458816, 'mem_total': 1930829824, 'swap_mem_used': 35528704, 'swap_mem_total': 1107292160} 2021-08-13 14:23:31 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-08-13 14:23:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:23:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:23:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:23:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:23:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-08-13 14:23:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-08-13 14:23:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:23:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:23:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-08-13 14:23:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-08-13 14:23:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:23:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:23:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-08-13 14:23:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-08-13 14:23:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:23:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.101, nothing to delete 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has completed ============== 2021-08-13 14:23:45 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.101 port:8091 ssh_username:root, nodes/self 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '7851', 'memoryTotal': 1930829824, 'memoryFree': 1163599872, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6503-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 730, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.101', 'ip': '10.112.205.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 1841, 'curr_items': 1000} 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 10.112.205.101 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=730 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.205.101,8091,['kv', 'index', 'n1ql']) 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.205.101: 8091:hostname=10.112.205.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql 2021-08-13 14:23:46 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://10.112.205.101:8091//node/controller/setupServices body: hostname=10.112.205.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.101:8091:port=8091&username=Administrator&password=password 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:23:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.102 port:8091 ssh_username:root, nodes/self 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '7852', 'memoryTotal': 1930829824, 'memoryFree': 1533079552, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.102:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6503-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.102', 'ip': '10.112.205.102', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-08-13 14:23:47 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.102:8091:port=8091&username=Administrator&password=password 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:23:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.103 port:8091 ssh_username:root, nodes/self 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '7852', 'memoryTotal': 1930829824, 'memoryFree': 1531764736, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.103:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6503-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.103', 'ip': '10.112.205.103', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-08-13 14:23:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.103:8091:port=8091&username=Administrator&password=password 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.104 port:8091 ssh_username:root, nodes/self 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '7854', 'memoryTotal': 1930829824, 'memoryFree': 1502019584, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.104:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6503-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 986, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.205.104', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.104:8091:port=8091&username=Administrator&password=password 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:23:49 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-08-13 14:23:49 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.205.101 **** 2021-08-13 14:23:50 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-08-13 14:23:55 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-08-13 14:23:55 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2021-08-13 14:23:56 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.102:8091 to cluster 2021-08-13 14:23:56 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.102:8091 to this cluster @10.112.205.101:8091 2021-08-13 14:24:06 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-08-13 14:24:06 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-08-13 14:24:19 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.103:8091 to cluster 2021-08-13 14:24:19 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.103:8091 to this cluster @10.112.205.101:8091 2021-08-13 14:24:29 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-08-13 14:24:29 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-08-13 14:24:42 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-08-13 14:24:43 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-08-13 14:24:53 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-08-13 14:24:53 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.013327836990356 sec 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.205.101 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ============== 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:24:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:24:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:24:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-08-13 14:24:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-08-13 14:24:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:24:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:24:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-08-13 14:24:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-08-13 14:24:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:24:55 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:24:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-08-13 14:24:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-08-13 14:24:55 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:24:55 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 66.16541353383458, 'mem_free': 1238212608, 'mem_total': 1930829824, 'swap_mem_used': 35487744, 'swap_mem_total': 1107292160} 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.102:8091 => {'services': ['index'], 'cpu_utilization': 4.396728016359918, 'mem_free': 1485697024, 'mem_total': 1930829824, 'swap_mem_used': 282624, 'swap_mem_total': 1107292160} 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.103:8091 => {'services': ['index'], 'cpu_utilization': 4.285714285714286, 'mem_free': 1488158720, 'mem_total': 1930829824, 'swap_mem_used': 380928, 'swap_mem_total': 1107292160} 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.205.101 cluster is healthy 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.205.101 is healthy 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.205.102 cluster is healthy 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.205.102 is healthy 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.205.103 cluster is healthy 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.205.103 is healthy 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.205.104 cluster is healthy 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.205.104 is healthy 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2021-08-13 14:25:09 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-08-13 14:25:14 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2021-08-13 14:25:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:25:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:25:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:25:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:25:23 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED 2021-08-13 14:25:23 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-08-13 14:25:23 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:10.112.205.101 port:8091 ssh_username:root 2021-08-13 14:25:23 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs. ... 2021-08-13 14:25:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:25:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:25:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:25:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:25:53 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-08-13 14:25:53 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2021-08-13 14:25:53 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2021-08-13 14:25:53 | INFO | MainProcess | MainThread | [planner_gsi.setUp] ============== PlannerGSI setup has started ============== 2021-08-13 14:25:53 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.205.101:8091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=486&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-08-13 14:25:53 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket test_bucket 2021-08-13 14:25:53 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:25:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:54 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-08-13 14:25:54 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:25:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:55 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-08-13 14:25:55 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:25:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:55 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-08-13 14:25:55 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:25:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:55 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-08-13 14:25:55 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:25:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:56 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-08-13 14:25:56 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:25:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:25:56 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-08-13 14:25:56 | INFO | MainProcess | MainThread | [planner_gsi.setUp] ============== PlannerGSI setup has completed ============== 2021-08-13 14:25:56 | INFO | MainProcess | MainThread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has started ============== 2021-08-13 14:25:56 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-08-13 14:25:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:25:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:25:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:25:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:25:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:25:58 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-08-13 14:25:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-08-13 14:25:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:25:58 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-08-13 14:25:58 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 2.371134020618557, 'mem_free': 1263788032, 'mem_total': 1930829824, 'swap_mem_used': 35479552, 'swap_mem_total': 1107292160} 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.102:8091 => {'services': ['index'], 'cpu_utilization': 1.538461538461539, 'mem_free': 1485578240, 'mem_total': 1930829824, 'swap_mem_used': 282624, 'swap_mem_total': 1107292160} 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.205.103:8091 => {'services': ['index'], 'cpu_utilization': 1.846153846153846, 'mem_free': 1488773120, 'mem_total': 1930829824, 'swap_mem_used': 380928, 'swap_mem_total': 1107292160} 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-08-13 14:25:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-08-13 14:26:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:26:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:26:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-08-13 14:26:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-08-13 14:26:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:26:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:26:11 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-08-13 14:26:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket'] on 10.112.205.101 2021-08-13 14:26:12 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-08-13 14:26:12 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.205.101 existing buckets : [] 2021-08-13 14:26:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket from 10.112.205.101 2021-08-13 14:26:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2021-08-13 14:26:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2021-08-13 14:26:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2021-08-13 14:26:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:26:12 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-08-13 14:26:12 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103', 'ejectedNodes': 'ns_1@10.112.205.102,ns_1@10.112.205.103', 'user': 'Administrator', 'password': 'password'} 2021-08-13 14:26:12 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2021-08-13 14:26:12 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-08-13 14:26:32 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 20.02 seconds 2021-08-13 14:26:32 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:10.112.205.101 port:8091 ssh_username:root ? [('ns_1@10.112.205.102', 8091), ('ns_1@10.112.205.103', 8091)] 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-08-13 14:26:42 | INFO | MainProcess | MainThread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has completed ============== ok ---------------------------------------------------------------------- Ran 1 test in 230.742s OK test_index_placements_with_skewed_load (gsi.planner_gsi.PlannerGSI) ... -->result: 2021-08-13 14:26:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:26:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:26:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.0.2-6503-enterprise 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.2-6503-enterprise'] 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_index_placements_with_skewed_load============== 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.101, nothing to delete 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2021-08-13 14:26:43 | INFO | MainProcess | test_thread | [basetestcase.setUp] Building docker image with java sdk client 2021-08-13 14:26:48 | INFO | MainProcess | test_thread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has started ============== 2021-08-13 14:26:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-08-13 14:26:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:26:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:26:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:26:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:26:49 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-08-13 14:26:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-08-13 14:26:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:26:49 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-08-13 14:26:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:26:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:26:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:26:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:26:49 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-08-13 14:26:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-08-13 14:26:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:26:50 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-08-13 14:26:50 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 1.637666325486182, 'mem_free': 1252925440, 'mem_total': 1930829824, 'swap_mem_used': 35446784, 'swap_mem_total': 1107292160} 2021-08-13 14:26:50 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-08-13 14:26:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:26:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:26:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:26:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:26:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-08-13 14:26:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-08-13 14:26:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:26:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:26:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-08-13 14:26:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-08-13 14:26:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:26:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:26:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-08-13 14:26:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-08-13 14:26:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:26:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_index_placements_with_skewed_load ============== 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.101, nothing to delete 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.102, nothing to delete 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.103, nothing to delete 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.205.104, nothing to delete 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.101:8091 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.101:8091 is running 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.102:8091 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.102:8091 is running 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.103:8091 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.103:8091 is running 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.205.104:8091 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.205.104:8091 is running 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_index_placements_with_skewed_load ============== 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [planner_gsi.tearDown] ============== PlannerGSI tearDown has completed ============== 2021-08-13 14:27:03 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.101 port:8091 ssh_username:root, nodes/self 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '8050', 'memoryTotal': 1930829824, 'memoryFree': 1270984704, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6503-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 730, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.101', 'ip': '10.112.205.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 10.112.205.101 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=730 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.205.101,8091,['kv', 'index', 'n1ql']) 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.205.101: 8091:hostname=10.112.205.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql 2021-08-13 14:27:04 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://10.112.205.101:8091//node/controller/setupServices body: hostname=10.112.205.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.101:8091:port=8091&username=Administrator&password=password 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:27:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.102 port:8091 ssh_username:root, nodes/self 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '8049', 'memoryTotal': 1930829824, 'memoryFree': 1529495552, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.102:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6503-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.102', 'ip': '10.112.205.102', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-08-13 14:27:05 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.102:8091:port=8091&username=Administrator&password=password 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-08-13 14:27:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.103 port:8091 ssh_username:root, nodes/self 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '8054', 'memoryTotal': 1930829824, 'memoryFree': 1527132160, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.103:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6503-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.205.103', 'ip': '10.112.205.103', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-08-13 14:27:06 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.205.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.103:8091:port=8091&username=Administrator&password=password 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-08-13 14:27:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.205.104 port:8091 ssh_username:root, nodes/self 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '8055', 'memoryTotal': 1930829824, 'memoryFree': 1502130176, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.104:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.2-6503-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 986, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.205.104', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.205.104:8091:port=8091&username=Administrator&password=password 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-08-13 14:27:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-08-13 14:27:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-08-13 14:27:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:27:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:27:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.104: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-08-13 14:27:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:27:08 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-08-13 14:27:08 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:27:08 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.205.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-08-13 14:27:08 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-08-13 14:27:08 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.205.101 **** 2021-08-13 14:27:08 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-08-13 14:27:13 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-08-13 14:27:13 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2021-08-13 14:27:14 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.102:8091 to cluster 2021-08-13 14:27:14 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.102:8091 to this cluster @10.112.205.101:8091 2021-08-13 14:27:24 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-08-13 14:27:24 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-08-13 14:27:38 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.205.103:8091 to cluster 2021-08-13 14:27:38 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.205.103:8091 to this cluster @10.112.205.101:8091 2021-08-13 14:27:48 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-08-13 14:27:48 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-08-13 14:28:02 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-08-13 14:28:02 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-08-13 14:28:12 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-08-13 14:28:12 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.010992765426636 sec 2021-08-13 14:28:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:28:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:28:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:28:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:28:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-08-13 14:28:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:28:12 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-08-13 14:28:12 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.205.101 2021-08-13 14:28:12 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_index_placements_with_skewed_load ============== 2021-08-13 14:28:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:28:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:28:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:28:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:28:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2021-08-13 14:28:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.102 as root 2021-08-13 14:28:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:28:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:28:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2021-08-13 14:28:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.103 as root 2021-08-13 14:28:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:28:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:28:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2021-08-13 14:28:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.104 as root 2021-08-13 14:28:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:28:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 67.40638002773925, 'mem_free': 1262411776, 'mem_total': 1930829824, 'swap_mem_used': 35385344, 'swap_mem_total': 1107292160} 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.102:8091 => {'services': ['index'], 'cpu_utilization': 2.769230769230769, 'mem_free': 1496846336, 'mem_total': 1930829824, 'swap_mem_used': 282624, 'swap_mem_total': 1107292160} 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.205.103:8091 => {'services': ['index'], 'cpu_utilization': 3.070624360286592, 'mem_free': 1492938752, 'mem_total': 1930829824, 'swap_mem_used': 380928, 'swap_mem_total': 1107292160} 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.101 cluster is healthy 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.101 is healthy 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.102 cluster is healthy 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.102 is healthy 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.103 cluster is healthy 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.103 is healthy 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.205.104 cluster is healthy 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.205.104 is healthy 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2021-08-13 14:28:27 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-08-13 14:28:32 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2021-08-13 14:28:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:28:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:28:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:28:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:28:41 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2021-08-13 14:28:41 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root] 2021-08-13 14:28:41 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:10.112.205.101 port:8091 ssh_username:root 2021-08-13 14:28:41 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2021-08-13 14:29:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:29:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:29:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:29:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:29:11 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-08-13 14:29:11 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set 2021-08-13 14:29:11 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set 2021-08-13 14:29:11 | INFO | MainProcess | test_thread | [planner_gsi.setUp] ============== PlannerGSI setup has started ============== 2021-08-13 14:29:12 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.205.101:8091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=486&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-08-13 14:29:12 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket test_bucket 2021-08-13 14:29:12 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:29:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:13 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-08-13 14:29:13 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:29:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:13 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-08-13 14:29:13 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:29:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:14 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-08-13 14:29:14 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:29:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:14 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-08-13 14:29:14 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:29:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:14 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-08-13 14:29:14 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : test_bucket in 10.112.205.101 to accept set ops 2021-08-13 14:29:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.205.101:11210 test_bucket 2021-08-13 14:29:15 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-08-13 14:29:15 | INFO | MainProcess | test_thread | [planner_gsi.setUp] ============== PlannerGSI setup has completed ============== 2021-08-13 14:29:15 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-08-13 14:29:15 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.205.101:8091/pools/default/buckets/test_bucket/scopes with params: name=test_scope_1 2021-08-13 14:29:15 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created test_bucket->test_scope_1 {'uid': '1'} 2021-08-13 14:29:15 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.205.101:8091/pools/default/buckets/test_bucket/scopes/test_scope_1/collections with params: name=test_collection_1 2021-08-13 14:29:15 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created test_bucket->test_scope_1->test_collection_1 manifest:{'uid': '2'} 2021-08-13 14:29:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. Allowing time after collection creation ... 2021-08-13 14:29:25 | INFO | MainProcess | test_thread | [basetestcase.data_ops_javasdk_loader_in_batches] Number of batches : 1 2021-08-13 14:29:25 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2021-08-13 14:29:35 | INFO | MainProcess | test_thread | [basetestcase.data_ops_javasdk_loader_in_batches] done 2021-08-13 14:29:36 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.205.101 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False 2021-08-13 14:29:41 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx_0` ON default:test_bucket.test_scope_1.test_collection_1(age) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-08-13 14:29:41 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx_0%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28age%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-08-13 14:29:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.710525195s 2021-08-13 14:29:45 | ERROR | MainProcess | test_thread | [base_gsi.wait_until_indexes_online] No indexes are present, this check does not apply! 2021-08-13 14:29:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx_1` ON default:test_bucket.test_scope_1.test_collection_1(city) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-08-13 14:29:45 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx_1%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28city%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-08-13 14:29:50 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.55095806s 2021-08-13 14:29:55 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx_2` ON default:test_bucket.test_scope_1.test_collection_1(country) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-08-13 14:29:55 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx_2%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28country%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-08-13 14:30:01 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.726215652s 2021-08-13 14:30:06 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx_3` ON default:test_bucket.test_scope_1.test_collection_1(title) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-08-13 14:30:06 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx_3%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-08-13 14:30:11 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.952380394s 2021-08-13 14:30:16 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `idx_4` ON default:test_bucket.test_scope_1.test_collection_1(firstName) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-08-13 14:30:16 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60idx_4%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-08-13 14:30:21 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.765325839s 2021-08-13 14:30:26 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2021-08-13 14:30:36 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY ALTER INDEX idx_0 ON default:test_bucket.test_scope_1.test_collection_1 WITH {"action": "drop_replica", "replicaId": 1}; 2021-08-13 14:30:36 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=ALTER+INDEX+idx_0+ON+default%3Atest_bucket.test_scope_1.test_collection_1+WITH+%7B%22action%22%3A+%22drop_replica%22%2C+%22replicaId%22%3A+1%7D%3B 2021-08-13 14:30:36 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 96.229867ms 2021-08-13 14:30:36 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY ALTER INDEX idx_2 ON default:test_bucket.test_scope_1.test_collection_1 WITH {"action": "drop_replica", "replicaId": 1}; 2021-08-13 14:30:36 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=ALTER+INDEX+idx_2+ON+default%3Atest_bucket.test_scope_1.test_collection_1+WITH+%7B%22action%22%3A+%22drop_replica%22%2C+%22replicaId%22%3A+1%7D%3B 2021-08-13 14:30:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 224.803204ms 2021-08-13 14:30:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY ALTER INDEX idx_4 ON default:test_bucket.test_scope_1.test_collection_1 WITH {"action": "drop_replica", "replicaId": 0}; 2021-08-13 14:30:37 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=ALTER+INDEX+idx_4+ON+default%3Atest_bucket.test_scope_1.test_collection_1+WITH+%7B%22action%22%3A+%22drop_replica%22%2C+%22replicaId%22%3A+0%7D%3B 2021-08-13 14:30:37 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 223.022671ms 2021-08-13 14:30:37 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 60 secs. Giving some time for clean-up after replicas drop ... 2021-08-13 14:31:40 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-08-13 14:31:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:31:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:31:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:31:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:31:41 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loded_index_node] Deleting index_plan.log from Remote host 2021-08-13 14:31:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: rm -rf /tmp/index_plan.log 2021-08-13 14:31:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:31:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: /opt/couchbase/bin/cbindexplan -command=retrieve -cluster="127.0.0.1:8091" -username="Administrator" -password="password" -output /tmp/index_plan.log 2021-08-13 14:31:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:31:42 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loded_index_node] Deleting index_plan.log from slave 2021-08-13 14:31:42 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loded_index_node] Deleting index_plan.log from Remote host after copying 2021-08-13 14:31:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: rm -rf /tmp/index_plan.log 2021-08-13 14:31:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:32:09 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.205.101 port:8091 ssh_username:root, ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2021-08-13 14:32:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2021-08-13 14:32:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.205.101 as root 2021-08-13 14:32:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-08-13 14:32:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-08-13 14:32:10 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loded_index_node] Deleting index_plan.log from Remote host 2021-08-13 14:32:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: rm -rf /tmp/index_plan.log 2021-08-13 14:32:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:32:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: /opt/couchbase/bin/cbindexplan -command=retrieve -cluster="127.0.0.1:8091" -username="Administrator" -password="password" -output /tmp/index_plan.log 2021-08-13 14:32:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:32:11 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loded_index_node] Deleting index_plan.log from slave 2021-08-13 14:32:11 | INFO | MainProcess | test_thread | [planner_gsi._find_least_loded_index_node] Deleting index_plan.log from Remote host after copying 2021-08-13 14:32:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.205.101: rm -rf /tmp/index_plan.log 2021-08-13 14:32:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-08-13 14:32:24 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `new_idx_0` ON default:test_bucket.test_scope_1.test_collection_1(age) USING GSI WITH {'defer_build': False, 'num_replica': 1} 2021-08-13 14:32:24 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60new_idx_0%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28age%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D 2021-08-13 14:32:29 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.878193602s