Starting Test: test_eventing_rebalance_swap_delete_recreate_collections 2022-03-01 20:58:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 20:58:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 20:58:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:58:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:58:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 20:58:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2335-enterprise 2022-03-01 20:58:58 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_eventing_rebalance_swap_delete_recreate_collections============== 2022-03-01 20:58:58 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2022-03-01 20:58:58 | INFO | MainProcess | test_thread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has started ============== 2022-03-01 20:58:59 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2022-03-01 20:58:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 20:58:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.64:8091 => {'services': ['kv'], 'cpu_utilization': 0.5263157894736842, 'mem_free': 3219210240, 'mem_total': 3945984000, 'swap_mem_used': 61775872, 'swap_mem_total': 2147479552} 2022-03-01 20:58:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 20:58:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 20:58:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 20:58:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:58:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:58:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 20:58:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 20:59:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 20:59:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 20:59:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 20:59:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 20:59:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 20:59:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 20:59:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 20:59:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 20:59:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 20:59:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 20:59:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 20:59:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 20:59:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_eventing_rebalance_swap_delete_recreate_collections ============== 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.64, nothing to delete 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.67, nothing to delete 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.73, nothing to delete 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.74, nothing to delete 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.75, nothing to delete 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.76, nothing to delete 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.77, nothing to delete 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.78, nothing to delete 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-03-01 20:59:10 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.106.64:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.64:8091 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.64:8091 is running 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.67:8091 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.67:8091 is running 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.73:8091 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.73:8091 is running 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.74:8091 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.74:8091 is running 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.75:8091 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.75:8091 is running 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.76:8091 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.76:8091 is running 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.77:8091 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.77:8091 is running 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.78:8091 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.78:8091 is running 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_eventing_rebalance_swap_delete_recreate_collections ============== 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has completed ============== 2022-03-01 20:59:10 | INFO | MainProcess | test_thread | [basetestcase.setUp] Building docker image with java sdk client OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 2022-03-01 20:59:14 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2022-03-01 20:59:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 20:59:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 20:59:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 20:59:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 20:59:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:16 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.64 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 20:59:16 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 20:59:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: systemctl stop couchbase-server.service 2022-03-01 20:59:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:18 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:18 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 20:59:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 20:59:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 20:59:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 20:59:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 20:59:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:19 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.64 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 20:59:19 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:19 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 20:59:19 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 20:59:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: systemctl start couchbase-server.service 2022-03-01 20:59:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:19 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.64:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 20:59:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 20:59:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:24 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 130200 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 20:59:24 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:24 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.64: with pid 130378 2022-03-01 20:59:24 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 20:59:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 20:59:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 20:59:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 20:59:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 20:59:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:26 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.67 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 20:59:26 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 20:59:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: systemctl stop couchbase-server.service 2022-03-01 20:59:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:28 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:28 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 20:59:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 20:59:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 20:59:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 20:59:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 20:59:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:29 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.67 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 20:59:29 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:29 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 20:59:29 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 20:59:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: systemctl start couchbase-server.service 2022-03-01 20:59:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:29 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.67:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 20:59:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 20:59:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:34 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 35744 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 20:59:34 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:34 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.67: with pid 35893 2022-03-01 20:59:34 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 20:59:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 20:59:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 20:59:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 20:59:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 20:59:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:36 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.73 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 20:59:36 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 20:59:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: systemctl stop couchbase-server.service 2022-03-01 20:59:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:37 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.73 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: systemctl start couchbase-server.service 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:38 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.73:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 20:59:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 20:59:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:44 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 28241 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 20:59:44 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:44 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.73: with pid 28420 2022-03-01 20:59:44 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 20:59:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 20:59:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 20:59:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 20:59:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 20:59:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:45 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.74 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 20:59:45 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 20:59:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: systemctl stop couchbase-server.service 2022-03-01 20:59:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:47 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:47 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 20:59:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 20:59:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 20:59:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 20:59:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 20:59:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:48 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.74 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 20:59:48 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:48 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 20:59:48 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 20:59:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: systemctl start couchbase-server.service 2022-03-01 20:59:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:48 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.74:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 20:59:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 20:59:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:53 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 53427 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 20:59:53 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:53 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.74: with pid 53569 2022-03-01 20:59:53 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 20:59:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 20:59:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 20:59:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 20:59:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 20:59:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:55 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.75 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 20:59:55 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 20:59:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: systemctl stop couchbase-server.service 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.75 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 20:59:57 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 20:59:58 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 20:59:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: systemctl start couchbase-server.service 2022-03-01 20:59:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 20:59:58 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.75:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 42199 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.75: with pid 42342 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 21:00:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 21:00:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:04 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.76 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 21:00:04 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 21:00:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: systemctl stop couchbase-server.service 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.76 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: systemctl start couchbase-server.service 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:07 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.76:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 21:00:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 21:00:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:12 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 129188 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 21:00:12 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 21:00:13 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.76: with pid 129330 2022-03-01 21:00:13 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 21:00:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 21:00:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 21:00:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 21:00:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 21:00:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:14 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.77 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 21:00:14 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 21:00:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: systemctl stop couchbase-server.service 2022-03-01 21:00:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:16 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.77 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 21:00:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: systemctl start couchbase-server.service 2022-03-01 21:00:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:18 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.77:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 27430 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.77: with pid 27573 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 21:00:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 21:00:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:24 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.78 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 21:00:24 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 21:00:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: systemctl stop couchbase-server.service 2022-03-01 21:00:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:26 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 21:00:26 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 21:00:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 21:00:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 21:00:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 21:00:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 21:00:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:27 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.78 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 21:00:27 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 21:00:27 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 21:00:27 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 21:00:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: systemctl start couchbase-server.service 2022-03-01 21:00:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:27 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.78:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 21:00:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 21:00:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:32 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 88307 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 21:00:32 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 21:00:32 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.78: with pid 88451 2022-03-01 21:00:32 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 21:00:32 | ERROR | MainProcess | test_thread | [rest_client._http_request] socket error while connecting to http://172.23.106.78:8091/nodes/self error [Errno 111] Connection refused 2022-03-01 21:00:36 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.64 port:8091 ssh_username:root, nodes/self 2022-03-01 21:00:36 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '71', 'memoryTotal': 3945984000, 'memoryFree': 3327848448, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.64:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.64', 'ip': '172.23.106.64', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 21:00:36 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.64:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 21:00:36 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 21:00:36 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.106.64,8091,['kv']) 2022-03-01 21:00:36 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.106.64: 8091:hostname=172.23.106.64&user=Administrator&password=password&services=kv 2022-03-01 21:00:36 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 21:00:36 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.64:8091:port=8091&username=Administrator&password=password 2022-03-01 21:00:36 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 21:00:36 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 21:00:36 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 21:00:36 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.64:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.64:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.67 port:8091 ssh_username:root, nodes/self 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '61', 'memoryTotal': 3945984000, 'memoryFree': 3366178816, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.67:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.67', 'ip': '172.23.106.67', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 21:00:37 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.67:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.67:8091:port=8091&username=Administrator&password=password 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 21:00:37 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.67:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.67:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.73 port:8091 ssh_username:root, nodes/self 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '56', 'memoryTotal': 3945984000, 'memoryFree': 3324637184, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.73:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.73', 'ip': '172.23.106.73', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 21:00:38 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.73:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.73:8091:port=8091&username=Administrator&password=password 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.73:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.73:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.74 port:8091 ssh_username:root, nodes/self 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '46', 'memoryTotal': 3945984000, 'memoryFree': 3358404608, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.74:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.74', 'ip': '172.23.106.74', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 21:00:38 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.74:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 21:00:38 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.74:8091:port=8091&username=Administrator&password=password 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.74:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.74:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.75 port:8091 ssh_username:root, nodes/self 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '36', 'memoryTotal': 3945984000, 'memoryFree': 3312099328, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.75:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.75', 'ip': '172.23.106.75', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 21:00:39 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.75:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.75:8091:port=8091&username=Administrator&password=password 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 21:00:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.75:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.75:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.76 port:8091 ssh_username:root, nodes/self 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '25', 'memoryTotal': 3945984000, 'memoryFree': 3312918528, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.76:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.76', 'ip': '172.23.106.76', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 21:00:40 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.76:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.76:8091:port=8091&username=Administrator&password=password 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 21:00:40 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.76:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.76:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.77 port:8091 ssh_username:root, nodes/self 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '21', 'memoryTotal': 0, 'memoryFree': 0, 'mcdMemoryReserved': 3763, 'mcdMemoryAllocated': 0, 'status': 'healthy', 'hostname': '172.23.106.77:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.77', 'ip': '172.23.106.77', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 21:00:41 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.77:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2521 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.77:8091:port=8091&username=Administrator&password=password 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 21:00:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.77:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.77:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.78 port:8091 ssh_username:root, nodes/self 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '10', 'memoryTotal': 3945984000, 'memoryFree': 3259736064, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.78:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2335-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.78', 'ip': '172.23.106.78', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 21:00:42 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.78:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.78:8091:port=8091&username=Administrator&password=password 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 21:00:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:00:43 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:00:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 21:00:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:00:43 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 21:00:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.78:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.78:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 21:00:43 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 21:00:43 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.106.64 **** 2022-03-01 21:00:43 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.106.64:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-01 21:00:43 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-03-01 21:00:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-03-01 21:00:48 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-03-01 21:00:48 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2022-03-01 21:00:49 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.67:8091 to cluster 2022-03-01 21:00:49 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.67:18091 to this cluster @172.23.106.64:8091 2022-03-01 21:00:59 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 21:00:59 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 21:01:12 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.73:8091 to cluster 2022-03-01 21:01:12 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.73:18091 to this cluster @172.23.106.64:8091 2022-03-01 21:01:22 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 21:01:22 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 21:01:36 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.74:8091 to cluster 2022-03-01 21:01:36 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.74:18091 to this cluster @172.23.106.64:8091 2022-03-01 21:01:46 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2022-03-01 21:01:46 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 21:01:59 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.75:8091 to cluster 2022-03-01 21:01:59 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.75:18091 to this cluster @172.23.106.64:8091 2022-03-01 21:02:09 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2022-03-01 21:02:09 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 21:02:22 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.67 not part of cluster inactiveAdded 2022-03-01 21:02:22 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.73 not part of cluster inactiveAdded 2022-03-01 21:02:22 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.74 not part of cluster inactiveAdded 2022-03-01 21:02:22 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.75 not part of cluster inactiveAdded 2022-03-01 21:02:22 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.64,ns_1@172.23.106.67,ns_1@172.23.106.73,ns_1@172.23.106.74,ns_1@172.23.106.75', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-03-01 21:02:22 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-03-01 21:02:32 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-03-01 21:02:32 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.019862651824951 sec 2022-03-01 21:02:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 21:02:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 21:02:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:02:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:02:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 21:02:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:02:33 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 21:02:33 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.106.64 2022-03-01 21:02:33 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_eventing_rebalance_swap_delete_recreate_collections ============== 2022-03-01 21:02:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 21:02:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 21:02:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:02:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:02:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 21:02:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 21:02:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:02:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:02:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 21:02:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 21:02:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:02:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:02:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 21:02:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 21:02:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:02:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:02:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 21:02:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 21:02:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:02:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:02:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 21:02:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 21:02:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:02:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:02:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 21:02:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 21:02:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:02:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:02:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 21:02:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 21:02:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:02:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.64:8091 => {'services': ['kv'], 'cpu_utilization': 4.933299773470929, 'mem_free': 3282030592, 'mem_total': 3945984000, 'swap_mem_used': 61775872, 'swap_mem_total': 2147479552} 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.67:8091 => {'services': ['eventing'], 'cpu_utilization': 0.9789156626506024, 'mem_free': 3317379072, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.73:8091 => {'services': ['eventing'], 'cpu_utilization': 2.206619859578736, 'mem_free': 3281334272, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.74:8091 => {'services': ['index'], 'cpu_utilization': 5.363888189372954, 'mem_free': 3302330368, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.75:8091 => {'services': ['n1ql'], 'cpu_utilization': 4.298642533936651, 'mem_free': 3237728256, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 227.04290342330933 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:02:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:02:49 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:02:49 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 21:02:49 | INFO | MainProcess | test_thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 21:02:49 | INFO | MainProcess | test_thread | [eventing_base.setUp] Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. 2022-03-01 21:02:49 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=330 2022-03-01 21:02:49 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2022-03-01 21:02:49 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : eventingMemoryQuota=512 2022-03-01 21:02:49 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:02:49 | INFO | MainProcess | test_thread | [eventing_base.setUp] ============== EventingBaseTest setup has completed ============== 2022-03-01 21:02:49 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=700 2022-03-01 21:02:49 | INFO | MainProcess | test_thread | [eventing_rebalance_collection.setUp] 200 2022-03-01 21:02:50 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.64:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-03-01 21:02:50 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket src_bucket 2022-03-01 21:02:50 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.106.64 to accept set ops 2022-03-01 21:02:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2022-03-01 21:02:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2022-03-01 21:02:52 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'src_bucket' was created with per node RAM quota: 200 2022-03-01 21:02:53 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.64:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-03-01 21:02:53 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket dst_bucket 2022-03-01 21:02:53 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.106.64 to accept set ops 2022-03-01 21:02:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2022-03-01 21:02:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2022-03-01 21:02:55 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'dst_bucket' was created with per node RAM quota: 200 2022-03-01 21:02:56 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.64:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-03-01 21:02:56 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket metadata 2022-03-01 21:02:56 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.106.64 to accept set ops 2022-03-01 21:02:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2022-03-01 21:02:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2022-03-01 21:02:57 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'metadata' was created with per node RAM quota: 200 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_0 {'uid': '1'} 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_1 {'uid': '2'} 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_2 {'uid': '3'} 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_3 {'uid': '4'} 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_4 {'uid': '5'} 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_0 {'uid': '1'} 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_1 {'uid': '2'} 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 2022-03-01 21:02:57 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_2 {'uid': '3'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_3 {'uid': '4'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_4 {'uid': '5'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 2022-03-01 21:02:58 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} 2022-03-01 21:02:59 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function bucket_op 2022-03-01 21:03:00 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function timers 2022-03-01 21:03:01 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function sbm 2022-03-01 21:03:03 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function curl 2022-03-01 21:03:04 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function n1ql 2022-03-01 21:03:04 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 21:03:05 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2022-03-01 21:03:10 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2022-03-01 21:03:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2022-03-01 21:03:20 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2022-03-01 21:03:25 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 21:03:26 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2022-03-01 21:03:31 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2022-03-01 21:03:36 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2022-03-01 21:03:41 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2022-03-01 21:03:46 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 21:03:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2022-03-01 21:03:52 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2022-03-01 21:03:57 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2022-03-01 21:04:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2022-03-01 21:04:07 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 21:04:08 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2022-03-01 21:04:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2022-03-01 21:04:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2022-03-01 21:04:23 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2022-03-01 21:04:28 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 21:04:30 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2022-03-01 21:04:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2022-03-01 21:04:40 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2022-03-01 21:04:45 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2022-03-01 21:04:50 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 21:04:50 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.106.64 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False --secure False 2022-03-01 21:04:53 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.005934000015259 seconds 2022-03-01 21:05:33 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.106.64 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False --secure False 2022-03-01 21:06:16 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.76:8091 to cluster 2022-03-01 21:06:16 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.76:18091 to this cluster @172.23.106.64:8091 2022-03-01 21:06:26 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 21:06:26 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 21:06:40 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.76 not part of cluster inactiveAdded 2022-03-01 21:06:40 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.64,ns_1@172.23.106.67,ns_1@172.23.106.73,ns_1@172.23.106.74,ns_1@172.23.106.75,ns_1@172.23.106.76', 'ejectedNodes': 'ns_1@172.23.106.67,ns_1@172.23.106.73', 'user': 'Administrator', 'password': 'password'} 2022-03-01 21:06:40 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-03-01 21:06:50 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 33.00 % 2022-03-01 21:06:50 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 33.00% 2022-03-01 21:07:10 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 33.00 % 2022-03-01 21:07:10 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 33.00% 2022-03-01 21:07:30 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 83.00 % 2022-03-01 21:07:30 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 83.00% 2022-03-01 21:07:50 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 83.00 % 2022-03-01 21:07:50 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 83.00% 2022-03-01 21:08:10 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-03-01 21:08:10 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 90.16064071655273 sec 2022-03-01 21:08:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.76 port:8091 ssh_username:root] 2022-03-01 21:08:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:08:10 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on dst_bucket.scope_1.coll_3 2022-03-01 21:08:10 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 2022-03-01 21:08:12 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.155428964s 2022-03-01 21:08:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:08:12 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 2022-03-01 21:08:12 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 40320 Expected : 40320 2022-03-01 21:08:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.76 port:8091 ssh_username:root] 2022-03-01 21:08:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:08:12 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on dst_bucket.scope_1.coll_4 2022-03-01 21:08:12 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 2022-03-01 21:08:15 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.113596068s 2022-03-01 21:08:15 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:08:15 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 2022-03-01 21:08:15 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 40320 Expected : 40320 2022-03-01 21:08:15 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.76 port:8091 ssh_username:root] 2022-03-01 21:08:15 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:08:15 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on src_bucket.scope_1.coll_1 2022-03-01 21:08:15 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+src_bucket.scope_1.coll_1 2022-03-01 21:08:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.198427378s 2022-03-01 21:08:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:08:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 2022-03-01 21:08:18 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 80640 Expected : 80640 2022-03-01 21:08:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.76 port:8091 ssh_username:root] 2022-03-01 21:08:18 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.64:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 2022-03-01 21:08:18 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': 'e'} 2022-03-01 21:08:18 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.64:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 2022-03-01 21:08:18 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': 'f'} 2022-03-01 21:08:18 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.64:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 2022-03-01 21:08:18 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '10'} 2022-03-01 21:08:18 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2335-enterprise', '7.1.0-2335-enterprise', '7.1.0-2335-enterprise', '7.1.0-2335-enterprise'] 2022-03-01 21:08:18 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.77:8091 to cluster 2022-03-01 21:08:18 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.77:18091 to this cluster @172.23.106.64:8091 2022-03-01 21:08:28 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 21:08:28 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 21:08:41 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.77 not part of cluster inactiveAdded 2022-03-01 21:08:41 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.64,ns_1@172.23.106.74,ns_1@172.23.106.75,ns_1@172.23.106.76,ns_1@172.23.106.77', 'ejectedNodes': 'ns_1@172.23.106.76', 'user': 'Administrator', 'password': 'password'} 2022-03-01 21:08:41 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-03-01 21:08:41 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.106.64 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 0 -pu 0 -pd 100 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False --secure False 2022-03-01 21:09:25 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 52.00 % 2022-03-01 21:09:25 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 52.00% 2022-03-01 21:09:25 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.106.64 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 0 -pu 0 -pd 100 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False --secure False 2022-03-01 21:09:32 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.007502794265747 seconds 2022-03-01 21:10:09 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-03-01 21:10:09 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://172.23.106.76:8091/nodes/self error [Errno 111] Connection refused 2022-03-01 21:10:12 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 90.79011583328247 sec 2022-03-01 21:10:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 21:10:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:10:12 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on dst_bucket.scope_1.coll_0 2022-03-01 21:10:12 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_0 2022-03-01 21:10:16 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.447648963s 2022-03-01 21:10:16 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:10:16 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2022-03-01 21:10:16 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 0 Expected : 0 2022-03-01 21:10:16 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 21:10:16 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:10:16 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on dst_bucket.scope_1.coll_1 2022-03-01 21:10:16 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_1 2022-03-01 21:10:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.466273971s 2022-03-01 21:10:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:10:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 21:10:19 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 40312 Expected : 0 ... 2022-03-01 21:10:49 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:10:49 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 21:10:49 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:10:49 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 21:10:49 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 31645 Expected : 0 ... 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 0 Expected : 0 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on dst_bucket.scope_1.coll_3 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 2022-03-01 21:11:19 | ERROR | MainProcess | test_thread | [rest_client._http_request] POST http://172.23.106.75:8093/query?statement=create+primary+index+on+dst_bucket.scope_1.coll_3 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 409 reason: unknown b'{\n"requestID": "112c4f40-6b99-49ca-ba3a-37b55fb989df",\n"signature": null,\n"results": [\n],\n"errors": [{"code":4300,"msg":"The index #primary already exists.","reason":{"name":"#primary"}}],\n"status": "errors",\n"metrics": {"elapsedTime": "1.310982ms","executionTime": "1.250828ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 0 Expected : 0 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on dst_bucket.scope_1.coll_4 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 2022-03-01 21:11:19 | ERROR | MainProcess | test_thread | [rest_client._http_request] POST http://172.23.106.75:8093/query?statement=create+primary+index+on+dst_bucket.scope_1.coll_4 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 409 reason: unknown b'{\n"requestID": "25be6c0a-ceb4-4225-ba91-87c50ec04609",\n"signature": null,\n"results": [\n],\n"errors": [{"code":4300,"msg":"The index #primary already exists.","reason":{"name":"#primary"}}],\n"status": "errors",\n"metrics": {"elapsedTime": "955.094\xc2\xb5s","executionTime": "913.905\xc2\xb5s","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 0 Expected : 0 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on src_bucket.scope_1.coll_1 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+src_bucket.scope_1.coll_1 2022-03-01 21:11:19 | ERROR | MainProcess | test_thread | [rest_client._http_request] POST http://172.23.106.75:8093/query?statement=create+primary+index+on+src_bucket.scope_1.coll_1 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 409 reason: unknown b'{\n"requestID": "b82765f1-2adf-4491-adf2-90ecb9baebd4",\n"signature": null,\n"results": [\n],\n"errors": [{"code":4300,"msg":"The index #primary already exists.","reason":{"name":"#primary"}}],\n"status": "errors",\n"metrics": {"elapsedTime": "1.164832ms","executionTime": "1.125699ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password 2022-03-01 21:11:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:11:20 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 2022-03-01 21:11:20 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 40320 Expected : 40320 2022-03-01 21:11:20 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 21:11:20 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to undeployed... ... 2022-03-01 21:11:25 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to undeployed... ... 2022-03-01 21:11:30 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to undeployed... ... 2022-03-01 21:11:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to undeployed... ... 2022-03-01 21:11:40 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to undeployed... ... 2022-03-01 21:11:45 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to undeployed... ... 2022-03-01 21:11:50 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to undeployed... ... 2022-03-01 21:11:55 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to undeployed... ... 2022-03-01 21:12:00 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to undeployed... ... 2022-03-01 21:12:05 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to undeployed... ... 2022-03-01 21:12:10 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to undeployed... ... 2022-03-01 21:12:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to undeployed... ... 2022-03-01 21:12:20 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to undeployed... ... 2022-03-01 21:12:25 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to undeployed... ... 2022-03-01 21:12:30 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to undeployed... ... 2022-03-01 21:12:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to undeployed... ... 2022-03-01 21:12:40 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 21:12:40 | INFO | MainProcess | test_thread | [eventing_base.print_eventing_stats_from_all_eventing_nodes] Stats for Node 172.23.106.77 is [] 2022-03-01 21:12:40 | INFO | MainProcess | test_thread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has started ============== 2022-03-01 21:12:40 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 21:12:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 21:12:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 21:12:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.77:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+metadata 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [eventing_base.tearDown] number of documents in metadata bucket 0 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.64:8091 => {'services': ['kv'], 'cpu_utilization': 6.268958543983822, 'mem_free': 2866589696, 'mem_total': 3945984000, 'swap_mem_used': 61775872, 'swap_mem_total': 2147479552} 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.74:8091 => {'services': ['index'], 'cpu_utilization': 0.9093205354887598, 'mem_free': 3058794496, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.75:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.8316532258064516, 'mem_free': 3180748800, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.77:8091 => {'services': ['eventing'], 'cpu_utilization': 6.556962025316456, 'mem_free': 3190579200, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 21:12:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:12:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:12:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 21:12:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 21:12:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:12:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:12:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 21:12:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 21:12:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:12:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:12:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 21:12:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 21:12:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:12:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:12:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 21:12:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 21:12:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:12:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:12:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 21:12:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 21:12:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:12:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:12:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 21:12:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 21:12:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:12:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:12:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 21:12:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 21:12:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 21:12:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 21:12:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_eventing_rebalance_swap_delete_recreate_collections ============== 2022-03-01 21:12:55 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.106.64 2022-03-01 21:12:56 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-03-01 21:12:56 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.64 existing buckets : ['metadata', 'src_bucket'] 2022-03-01 21:12:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : dst_bucket from 172.23.106.64 2022-03-01 21:12:56 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-03-01 21:12:56 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.64 existing buckets : ['src_bucket'] 2022-03-01 21:12:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : metadata from 172.23.106.64 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.64 existing buckets : [] 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : src_bucket from 172.23.106.64 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.67, nothing to delete 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.73, nothing to delete 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.74, nothing to delete 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.75, nothing to delete 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.76, nothing to delete 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.77, nothing to delete 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.78, nothing to delete 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-03-01 21:12:57 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.106.64:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.64,ns_1@172.23.106.74,ns_1@172.23.106.75,ns_1@172.23.106.77', 'ejectedNodes': 'ns_1@172.23.106.74,ns_1@172.23.106.75,ns_1@172.23.106.77', 'user': 'Administrator', 'password': 'password'} 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2022-03-01 21:12:57 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-03-01 21:13:07 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 75.00 % 2022-03-01 21:13:17 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 75.00 % 2022-03-01 21:13:37 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 40.05 seconds 2022-03-01 21:13:37 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.106.64 port:8091 ssh_username:root ? [('ns_1@172.23.106.74', 8091), ('ns_1@172.23.106.75', 8091), ('ns_1@172.23.106.77', 8091)] 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.64:8091 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.64:8091 is running 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.67:8091 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.67:8091 is running 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.73:8091 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.73:8091 is running 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.74:8091 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.74:8091 is running 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.75:8091 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.75:8091 is running 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.76:8091 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.76:8091 is running 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.77:8091 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.77:8091 is running 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.78:8091 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.78:8091 is running 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_eventing_rebalance_swap_delete_recreate_collections ============== 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-03-01 21:13:47 | INFO | MainProcess | test_thread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has completed ============== ok ---------------------------------------------------------------------- Ran 1 test in 889.612s OK