Starting Test: test_eventing_rebalance_swap_delete_recreate_collections 2022-03-01 22:44:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 22:44:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 22:44:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2383-enterprise 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_eventing_rebalance_swap_delete_recreate_collections============== 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has started ============== 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.64:8091 => {'services': ['kv'], 'cpu_utilization': 0.5259203606311045, 'mem_free': 3274723328, 'mem_total': 3945984000, 'swap_mem_used': 184197120, 'swap_mem_total': 2147479552} 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 22:44:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 22:44:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 22:44:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 22:44:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 22:44:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 22:44:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 22:44:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 22:44:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 22:44:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 22:44:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 22:44:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 22:44:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 22:44:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 22:44:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_eventing_rebalance_swap_delete_recreate_collections ============== 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.64, nothing to delete 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.67, nothing to delete 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.73, nothing to delete 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.74, nothing to delete 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.75, nothing to delete 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.76, nothing to delete 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.77, nothing to delete 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.78, nothing to delete 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-03-01 22:44:50 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.106.64:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.64:8091 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.64:8091 is running 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.67:8091 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.67:8091 is running 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.73:8091 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.73:8091 is running 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.74:8091 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.74:8091 is running 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.75:8091 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.75:8091 is running 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.76:8091 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.76:8091 is running 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.77:8091 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.77:8091 is running 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.78:8091 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.78:8091 is running 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_eventing_rebalance_swap_delete_recreate_collections ============== 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has completed ============== 2022-03-01 22:44:50 | INFO | MainProcess | test_thread | [basetestcase.setUp] Building docker image with java sdk client OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 2022-03-01 22:44:54 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2022-03-01 22:44:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 22:44:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 22:44:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 22:44:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 22:44:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:56 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.64 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:44:56 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 22:44:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: systemctl stop couchbase-server.service 2022-03-01 22:44:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:44:58 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:44:58 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 22:44:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 22:44:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:44:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 22:44:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:44:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 22:44:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 22:44:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:44:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:44:59 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.64 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:44:59 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:44:59 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 22:44:59 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 22:44:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: systemctl start couchbase-server.service 2022-03-01 22:44:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:44:59 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.64:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 22:45:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 22:45:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:04 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 46833 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 22:45:04 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:04 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.64: with pid 47013 2022-03-01 22:45:04 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 22:45:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 22:45:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 22:45:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 22:45:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 22:45:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:06 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.67 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:06 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 22:45:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: systemctl stop couchbase-server.service 2022-03-01 22:45:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:08 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:08 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 22:45:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 22:45:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 22:45:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 22:45:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 22:45:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:09 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.67 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:09 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:09 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 22:45:09 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 22:45:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: systemctl start couchbase-server.service 2022-03-01 22:45:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:09 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.67:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 22:45:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 22:45:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:14 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 91371 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 22:45:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.67: with pid 91514 2022-03-01 22:45:14 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 22:45:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 22:45:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 22:45:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 22:45:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 22:45:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:16 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.73 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:16 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 22:45:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: systemctl stop couchbase-server.service 2022-03-01 22:45:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:18 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:18 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 22:45:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 22:45:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 22:45:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 22:45:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 22:45:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:19 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.73 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:19 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:19 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 22:45:19 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 22:45:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: systemctl start couchbase-server.service 2022-03-01 22:45:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:19 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.73:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 22:45:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 22:45:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:24 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 85803 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 22:45:24 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:24 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.73: with pid 85983 2022-03-01 22:45:24 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 22:45:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 22:45:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 22:45:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 22:45:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 22:45:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:25 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.74 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:25 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 22:45:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: systemctl stop couchbase-server.service 2022-03-01 22:45:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:27 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:27 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 22:45:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 22:45:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 22:45:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 22:45:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 22:45:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:28 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.74 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:28 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:28 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 22:45:28 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 22:45:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: systemctl start couchbase-server.service 2022-03-01 22:45:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:28 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.74:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 22:45:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 22:45:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:33 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 89023 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 22:45:33 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:33 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.74: with pid 89166 2022-03-01 22:45:33 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 22:45:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 22:45:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 22:45:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 22:45:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 22:45:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:35 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.75 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:35 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 22:45:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: systemctl stop couchbase-server.service 2022-03-01 22:45:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:37 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:37 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 22:45:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 22:45:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 22:45:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 22:45:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 22:45:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:38 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.75 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:38 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:38 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 22:45:38 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 22:45:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: systemctl start couchbase-server.service 2022-03-01 22:45:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:38 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.75:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 22:45:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 22:45:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:43 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 76632 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 22:45:43 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:43 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.75: with pid 76774 2022-03-01 22:45:43 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 22:45:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 22:45:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 22:45:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 22:45:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 22:45:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:44 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.76 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:44 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 22:45:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: systemctl stop couchbase-server.service 2022-03-01 22:45:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:48 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:48 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 22:45:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 22:45:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 22:45:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 22:45:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 22:45:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:49 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.76 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:49 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:49 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 22:45:49 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 22:45:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: systemctl start couchbase-server.service 2022-03-01 22:45:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:49 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.76:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 22:45:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 22:45:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:54 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 36886 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 22:45:54 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:54 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.76: with pid 37037 2022-03-01 22:45:54 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 22:45:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 22:45:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 22:45:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 22:45:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 22:45:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:56 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.77 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:56 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 22:45:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: systemctl stop couchbase-server.service 2022-03-01 22:45:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:58 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:58 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 22:45:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 22:45:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 22:45:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 22:45:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 22:45:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:45:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:45:59 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.77 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:45:59 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:45:59 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 22:45:59 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 22:45:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: systemctl start couchbase-server.service 2022-03-01 22:45:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:45:59 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.77:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 22:46:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 22:46:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:04 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 99221 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 22:46:04 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:46:04 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.77: with pid 99364 2022-03-01 22:46:04 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 22:46:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 22:46:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 22:46:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:46:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:46:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 22:46:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 22:46:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:46:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:46:06 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.78 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:46:06 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2022-03-01 22:46:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: systemctl stop couchbase-server.service 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.78 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-01 22:46:08 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:46:09 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2022-03-01 22:46:09 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2022-03-01 22:46:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: systemctl start couchbase-server.service 2022-03-01 22:46:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:09 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.78:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-01 22:46:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-01 22:46:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:14 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 122050 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-01 22:46:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-03-01 22:46:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.78: with pid 122192 2022-03-01 22:46:14 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2022-03-01 22:46:14 | ERROR | MainProcess | test_thread | [rest_client._http_request] socket error while connecting to http://172.23.106.78:8091/nodes/self error [Errno 111] Connection refused 2022-03-01 22:46:17 | ERROR | MainProcess | test_thread | [rest_client._http_request] socket error while connecting to http://172.23.106.78:8091/nodes/self error [Errno 111] Connection refused 2022-03-01 22:46:23 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.64 port:8091 ssh_username:root, nodes/self 2022-03-01 22:46:23 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '81', 'memoryTotal': 3945984000, 'memoryFree': 3381776384, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.64:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2383-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.64', 'ip': '172.23.106.64', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 22:46:23 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.64:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 22:46:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 22:46:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.106.64,8091,['kv']) 2022-03-01 22:46:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.106.64: 8091:hostname=172.23.106.64&user=Administrator&password=password&services=kv 2022-03-01 22:46:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 22:46:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.64:8091:port=8091&username=Administrator&password=password 2022-03-01 22:46:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 22:46:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 22:46:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.64:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.64:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.67 port:8091 ssh_username:root, nodes/self 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '71', 'memoryTotal': 3945984000, 'memoryFree': 3359154176, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.67:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2383-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.67', 'ip': '172.23.106.67', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 22:46:24 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.67:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.67:8091:port=8091&username=Administrator&password=password 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 22:46:24 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.67:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.67:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.73 port:8091 ssh_username:root, nodes/self 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '61', 'memoryTotal': 3945984000, 'memoryFree': 3324239872, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.73:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2383-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.73', 'ip': '172.23.106.73', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 22:46:25 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.73:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.73:8091:port=8091&username=Administrator&password=password 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 22:46:25 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.73:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.73:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.74 port:8091 ssh_username:root, nodes/self 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '51', 'memoryTotal': 3945984000, 'memoryFree': 3360210944, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.74:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2383-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.74', 'ip': '172.23.106.74', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 22:46:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.74:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.74:8091:port=8091&username=Administrator&password=password 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.74:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.74:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.75 port:8091 ssh_username:root, nodes/self 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '46', 'memoryTotal': 3945984000, 'memoryFree': 3318149120, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.75:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2383-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.75', 'ip': '172.23.106.75', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 22:46:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.75:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 22:46:26 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.75:8091:port=8091&username=Administrator&password=password 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.75:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.75:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.76 port:8091 ssh_username:root, nodes/self 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '36', 'memoryTotal': 3945984000, 'memoryFree': 3362222080, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.76:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2383-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.76', 'ip': '172.23.106.76', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 22:46:27 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.76:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.76:8091:port=8091&username=Administrator&password=password 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 22:46:27 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.76:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.76:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.77 port:8091 ssh_username:root, nodes/self 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '26', 'memoryTotal': 3945984000, 'memoryFree': 3316981760, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.77:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2383-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.77', 'ip': '172.23.106.77', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 22:46:28 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.77:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.77:8091:port=8091&username=Administrator&password=password 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 22:46:28 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.77:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.77:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.78 port:8091 ssh_username:root, nodes/self 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '16', 'memoryTotal': 3945984000, 'memoryFree': 3250245632, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.78:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2383-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.78', 'ip': '172.23.106.78', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3763, 'curr_items': 0} 2022-03-01 22:46:29 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.78:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.78:8091:port=8091&username=Administrator&password=password 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 22:46:29 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:46:30 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:46:30 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 22:46:30 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:46:30 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 22:46:30 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.78:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:30 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.78:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 22:46:30 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 22:46:30 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.106.64 **** 2022-03-01 22:46:30 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.106.64:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-01 22:46:30 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-03-01 22:46:30 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-03-01 22:46:35 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-03-01 22:46:35 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2022-03-01 22:46:36 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.67:8091 to cluster 2022-03-01 22:46:36 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.67:18091 to this cluster @172.23.106.64:8091 2022-03-01 22:46:46 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 22:46:46 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 22:46:59 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.73:8091 to cluster 2022-03-01 22:46:59 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.73:18091 to this cluster @172.23.106.64:8091 2022-03-01 22:47:09 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 22:47:09 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 22:47:22 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.74:8091 to cluster 2022-03-01 22:47:22 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.74:18091 to this cluster @172.23.106.64:8091 2022-03-01 22:47:32 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2022-03-01 22:47:32 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 22:47:46 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.75:8091 to cluster 2022-03-01 22:47:46 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.75:18091 to this cluster @172.23.106.64:8091 2022-03-01 22:47:56 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 22:47:56 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 22:48:09 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.67 not part of cluster inactiveAdded 2022-03-01 22:48:09 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.73 not part of cluster inactiveAdded 2022-03-01 22:48:09 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.74 not part of cluster inactiveAdded 2022-03-01 22:48:09 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.75 not part of cluster inactiveAdded 2022-03-01 22:48:09 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.64,ns_1@172.23.106.67,ns_1@172.23.106.73,ns_1@172.23.106.74,ns_1@172.23.106.75', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-03-01 22:48:09 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-03-01 22:48:19 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-03-01 22:48:19 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.038691997528076 sec 2022-03-01 22:48:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 22:48:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 22:48:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:48:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:48:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 22:48:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 22:48:20 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 22:48:20 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.106.64 2022-03-01 22:48:20 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_eventing_rebalance_swap_delete_recreate_collections ============== 2022-03-01 22:48:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 22:48:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 22:48:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:48:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:48:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 22:48:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 22:48:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:48:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:48:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 22:48:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 22:48:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:48:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:48:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 22:48:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 22:48:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:48:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:48:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 22:48:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 22:48:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:48:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:48:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 22:48:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 22:48:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:48:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:48:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 22:48:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 22:48:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:48:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:48:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 22:48:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 22:48:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:48:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:48:32 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 22:48:32 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.64:8091 => {'services': ['kv'], 'cpu_utilization': 6.410256410256411, 'mem_free': 3334127616, 'mem_total': 3945984000, 'swap_mem_used': 184197120, 'swap_mem_total': 2147479552} 2022-03-01 22:48:32 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.67:8091 => {'services': ['eventing'], 'cpu_utilization': 5.385002516356316, 'mem_free': 3314425856, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 22:48:32 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.73:8091 => {'services': ['eventing'], 'cpu_utilization': 1.951951951951952, 'mem_free': 3285811200, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 22:48:32 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.74:8091 => {'services': ['index'], 'cpu_utilization': 5.378235737622518, 'mem_free': 3319070720, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 22:48:32 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.75:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.4518072289156627, 'mem_free': 3296112640, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 22:48:32 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 22:48:32 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 234.22250413894653 2022-03-01 22:48:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 22:48:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 22:48:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 22:48:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 22:48:36 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:48:36 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 22:48:36 | INFO | MainProcess | test_thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 22:48:36 | INFO | MainProcess | test_thread | [eventing_base.setUp] Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. 2022-03-01 22:48:36 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=330 2022-03-01 22:48:36 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2022-03-01 22:48:37 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : eventingMemoryQuota=512 2022-03-01 22:48:37 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:48:37 | INFO | MainProcess | test_thread | [eventing_base.setUp] ============== EventingBaseTest setup has completed ============== 2022-03-01 22:48:37 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=700 2022-03-01 22:48:37 | INFO | MainProcess | test_thread | [eventing_rebalance_collection.setUp] 200 2022-03-01 22:48:37 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.64:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-03-01 22:48:37 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket src_bucket 2022-03-01 22:48:37 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.106.64 to accept set ops 2022-03-01 22:48:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2022-03-01 22:48:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2022-03-01 22:48:39 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'src_bucket' was created with per node RAM quota: 200 2022-03-01 22:48:40 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.64:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-03-01 22:48:40 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket dst_bucket 2022-03-01 22:48:40 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.106.64 to accept set ops 2022-03-01 22:48:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2022-03-01 22:48:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2022-03-01 22:48:42 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'dst_bucket' was created with per node RAM quota: 200 2022-03-01 22:48:43 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.64:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-03-01 22:48:43 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket metadata 2022-03-01 22:48:43 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.106.64 to accept set ops 2022-03-01 22:48:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2022-03-01 22:48:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2022-03-01 22:48:45 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'metadata' was created with per node RAM quota: 200 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_0 {'uid': '1'} 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_1 {'uid': '2'} 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_2 {'uid': '3'} 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_3 {'uid': '4'} 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_4 {'uid': '5'} 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 2022-03-01 22:48:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_0 {'uid': '1'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_1 {'uid': '2'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_2 {'uid': '3'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_3 {'uid': '4'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_4 {'uid': '5'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.67:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 2022-03-01 22:48:46 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} 2022-03-01 22:48:47 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function bucket_op 2022-03-01 22:48:48 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function timers 2022-03-01 22:48:50 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function sbm 2022-03-01 22:48:51 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function curl 2022-03-01 22:48:52 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function n1ql 2022-03-01 22:48:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 22:48:53 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2022-03-01 22:48:58 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2022-03-01 22:49:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2022-03-01 22:49:08 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2022-03-01 22:49:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 22:49:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2022-03-01 22:49:19 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2022-03-01 22:49:24 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2022-03-01 22:49:29 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2022-03-01 22:49:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 22:49:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2022-03-01 22:49:40 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2022-03-01 22:49:45 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2022-03-01 22:49:50 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2022-03-01 22:49:55 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 22:49:56 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2022-03-01 22:50:01 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2022-03-01 22:50:07 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2022-03-01 22:50:12 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2022-03-01 22:50:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 22:50:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2022-03-01 22:50:23 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2022-03-01 22:50:28 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2022-03-01 22:50:33 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2022-03-01 22:50:38 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.73 port:8091 ssh_username:root] 2022-03-01 22:50:38 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.106.64 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False --secure False 2022-03-01 22:50:41 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.0054268836975098 seconds 2022-03-01 22:51:22 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.106.64 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False --secure False 2022-03-01 22:52:06 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.76:8091 to cluster 2022-03-01 22:52:06 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.76:18091 to this cluster @172.23.106.64:8091 2022-03-01 22:52:16 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 22:52:16 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 22:52:29 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.76 not part of cluster inactiveAdded 2022-03-01 22:52:29 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.64,ns_1@172.23.106.67,ns_1@172.23.106.73,ns_1@172.23.106.74,ns_1@172.23.106.75,ns_1@172.23.106.76', 'ejectedNodes': 'ns_1@172.23.106.67,ns_1@172.23.106.73', 'user': 'Administrator', 'password': 'password'} 2022-03-01 22:52:29 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-03-01 22:52:39 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 33.00 % 2022-03-01 22:52:39 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 33.00% 2022-03-01 22:52:59 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 82.00 % 2022-03-01 22:52:59 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 82.00% 2022-03-01 22:53:19 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 83.00 % 2022-03-01 22:53:19 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 83.00% 2022-03-01 22:53:39 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 83.00 % 2022-03-01 22:53:39 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 83.00% 2022-03-01 22:54:00 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 83.00 % 2022-03-01 22:54:00 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 83.00% 2022-03-01 22:54:20 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-03-01 22:54:20 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 110.20515155792236 sec 2022-03-01 22:54:20 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.76 port:8091 ssh_username:root] 2022-03-01 22:54:20 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:54:20 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on dst_bucket.scope_1.coll_3 2022-03-01 22:54:20 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 2022-03-01 22:54:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.087605896s 2022-03-01 22:54:22 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:54:22 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 2022-03-01 22:54:22 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 40320 Expected : 40320 2022-03-01 22:54:22 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.76 port:8091 ssh_username:root] 2022-03-01 22:54:22 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:54:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on dst_bucket.scope_1.coll_4 2022-03-01 22:54:22 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 2022-03-01 22:54:25 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.004404458s 2022-03-01 22:54:25 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:54:25 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 2022-03-01 22:54:25 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 40320 Expected : 40320 2022-03-01 22:54:25 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.76 port:8091 ssh_username:root] 2022-03-01 22:54:25 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:54:25 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on src_bucket.scope_1.coll_1 2022-03-01 22:54:25 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+src_bucket.scope_1.coll_1 2022-03-01 22:54:27 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.182090594s 2022-03-01 22:54:27 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:54:27 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 2022-03-01 22:54:27 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 80640 Expected : 80640 2022-03-01 22:54:27 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.76 port:8091 ssh_username:root] 2022-03-01 22:54:27 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.64:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 2022-03-01 22:54:27 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': 'e'} 2022-03-01 22:54:27 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.64:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 2022-03-01 22:54:27 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': 'f'} 2022-03-01 22:54:27 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://172.23.106.64:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 2022-03-01 22:54:27 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '10'} 2022-03-01 22:54:28 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2383-enterprise', '7.1.0-2383-enterprise', '7.1.0-2383-enterprise', '7.1.0-2383-enterprise'] 2022-03-01 22:54:28 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.77:8091 to cluster 2022-03-01 22:54:28 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.77:18091 to this cluster @172.23.106.64:8091 2022-03-01 22:54:38 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 22:54:38 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 22:54:52 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.77 not part of cluster inactiveAdded 2022-03-01 22:54:52 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.64,ns_1@172.23.106.74,ns_1@172.23.106.75,ns_1@172.23.106.76,ns_1@172.23.106.77', 'ejectedNodes': 'ns_1@172.23.106.76', 'user': 'Administrator', 'password': 'password'} 2022-03-01 22:54:52 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-03-01 22:54:52 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.106.64 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 0 -pu 0 -pd 100 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False --secure False 2022-03-01 22:55:36 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 40.00 % 2022-03-01 22:55:36 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 40.00% 2022-03-01 22:55:36 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.106.64 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 0 -pu 0 -pd 100 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False --secure False 2022-03-01 22:55:42 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.007028102874756 seconds 2022-03-01 22:56:19 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 79.00 % 2022-03-01 22:56:19 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 79.00% 2022-03-01 22:56:39 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-03-01 22:56:39 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 107.83002781867981 sec 2022-03-01 22:56:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 22:56:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:56:39 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on dst_bucket.scope_1.coll_0 2022-03-01 22:56:39 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_0 2022-03-01 22:56:43 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.988855935s 2022-03-01 22:56:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:56:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2022-03-01 22:56:43 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Final docs count... Current : 0 Expected : 0 2022-03-01 22:56:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 22:56:44 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:56:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on dst_bucket.scope_1.coll_1 2022-03-01 22:56:44 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_1 2022-03-01 22:56:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.298959004s 2022-03-01 22:56:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:56:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:56:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 40003 Expected : 0 ... 2022-03-01 22:57:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:57:17 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:57:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:57:17 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:57:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 18295 Expected : 0 ... 2022-03-01 22:57:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:57:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:57:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:57:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:57:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 22:58:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:58:17 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:58:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:58:17 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:58:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 22:58:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:58:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:58:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:58:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:58:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 22:59:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:59:17 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:59:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:59:17 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:59:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 22:59:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:59:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:59:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 22:59:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 22:59:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:00:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:00:17 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:00:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:00:17 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:00:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:00:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:00:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:00:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:00:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:00:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:01:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:01:17 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:01:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:01:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:01:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:01:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:01:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:01:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:01:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:01:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:02:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:02:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:02:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:02:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:02:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:02:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:02:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:02:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:02:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:02:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:03:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:03:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:03:18 | INFO | MainProcess | test_thread | [eventing_base.bucket_compaction] Compacting bucket : dst_bucket 2022-03-01 23:03:18 | INFO | MainProcess | test_thread | [rest_client.compact_bucket] bucket compaction successful 2022-03-01 23:03:18 | INFO | MainProcess | test_thread | [eventing_base.bucket_compaction] Compacting bucket : metadata 2022-03-01 23:03:18 | INFO | MainProcess | test_thread | [rest_client.compact_bucket] bucket compaction successful 2022-03-01 23:03:18 | INFO | MainProcess | test_thread | [eventing_base.bucket_compaction] Compacting bucket : src_bucket 2022-03-01 23:03:18 | INFO | MainProcess | test_thread | [rest_client.compact_bucket] bucket compaction successful 2022-03-01 23:03:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:03:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:03:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:03:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:03:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:03:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:03:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:03:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:04:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:04:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:04:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:04:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:04:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:04:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:04:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:04:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:04:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:04:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:05:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:05:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:05:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:05:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:05:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:05:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:05:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:05:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:05:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:05:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:06:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:06:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:06:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:06:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:06:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:06:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:06:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:06:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:06:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:06:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:07:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:07:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:07:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:07:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:07:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 311 Expected : 0 ... 2022-03-01 23:07:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:07:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:07:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.75 port:8091 ssh_username:root] 2022-03-01 23:07:48 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 2022-03-01 23:07:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 23:07:48 | INFO | MainProcess | test_thread | [eventing_base.print_eventing_stats_from_all_eventing_nodes] Stats for Node 172.23.106.77 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 41978, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 37197, "dcp_deletion_sent_to_worker": 37197, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 749, "failure_stats": 749, "latency_stats": 749, "lcb_exception_stats": 749, "log_level": 1, "num_processed_events": 37197, "processed_events_size": 9670980, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 40320 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 37197, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 37197, "dcp_delete_msg_counter": 37197, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 37197, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 41974, "no_op_counter": 0, "num_processed_events": 37197, "on_delete_failure": 0, "on_delete_success": 37197, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 9670980, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "101600": "2022-03-02T07:07:47Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "101600": "2022-03-02T07:07:47Z" }, "v8worker_events_lost": 0 }, "function_id": 2474913908, "function_name": "curl", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 35500, "50": 1200, "80": 1400, "90": 1700, "95": 2100, "99": 4900 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/curl_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/curl_0": 101600 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 43494, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 38789, "dcp_deletion_sent_to_worker": 38789, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 734, "failure_stats": 734, "latency_stats": 734, "lcb_exception_stats": 734, "log_level": 1, "num_processed_events": 38789, "processed_events_size": 10084860, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 40320 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 38789, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 38789, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 43490, "no_op_counter": 0, "num_processed_events": 38789, "on_delete_failure": 0, "on_delete_success": 38789, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 10084860, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "101628": "2022-03-02T07:07:48Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "101628": "2022-03-02T07:07:48Z" }, "v8worker_events_lost": 0 }, "function_id": 3969969105, "function_name": "n1ql", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 94300, "50": 2500, "80": 2900, "90": 3400, "95": 4300, "99": 8100 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/n1ql_0": 101628 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 44944, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 40320, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 3, "dcp_mutation_suppressed_counter": 3, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 718, "failure_stats": 718, "latency_stats": 718, "lcb_exception_stats": 718, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 10478172, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 44940, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 10478172, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "110939": "2022-03-02T07:07:47Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "110939": "2022-03-02T07:07:47Z" }, "v8worker_events_lost": 0 }, "function_id": 1628018449, "function_name": "sbm", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 900, "50": 200, "80": 300, "90": 400, "95": 700, "99": 700 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/sbm_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/sbm_0": 110939 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 45867, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 40320, "dcp_deletion_sent_to_worker": 40320, "dcp_seqno_advanced": 1024, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 698, "failure_stats": 698, "latency_stats": 698, "lcb_exception_stats": 698, "log_level": 1, "num_processed_events": 41344, "processed_events_size": 10576476, "thr_count": 1, "thr_map": 1, "timer_events": 50118, "timer_responses_received": 40320, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 45856, "no_op_counter": 1024, "num_processed_events": 41344, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 10576476, "timer_callback_failure": 0, "timer_callback_success": 50118, "timer_cancel_counter": 0, "timer_create_counter": 40320, "timer_create_failure": 0, "timer_msg_counter": 50118, "timer_responses_sent": 0, "timestamp": { "126872": "2022-03-02T07:07:48Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "126872": "2022-03-02T07:07:48Z" }, "v8worker_events_lost": 0 }, "function_id": 3391603128, "function_name": "timers", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 20400, "50": 800, "80": 1100, "90": 1300, "95": 1700, "99": 3600 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/timers_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/timers_0": 126872 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 37214, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 32352, "dcp_deletion_sent_to_worker": 32352, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 765, "failure_stats": 765, "latency_stats": 765, "lcb_exception_stats": 765, "log_level": 1, "num_processed_events": 32352, "processed_events_size": 8411476, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 40320 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 32352, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32352, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 37210, "no_op_counter": 0, "num_processed_events": 32352, "on_delete_failure": 0, "on_delete_success": 32352, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 8411476, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "101572": "2022-03-02T07:07:48Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "101572": "2022-03-02T07:07:48Z" }, "v8worker_events_lost": 0 }, "function_id": 451640982, "function_name": "bucket_op", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 13600, "50": 300, "80": 400, "90": 500, "95": 800, "99": 2000 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/bucket_op_0": 101572 } } ] 2022-03-01 23:07:52 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Stats for Node 172.23.106.77 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 37214, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 32352, "dcp_deletion_sent_to_worker": 32352, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 765, "failure_stats": 765, "latency_stats": 765, "lcb_exception_stats": 765, "log_level": 1, "num_processed_events": 32352, "processed_events_size": 8411476, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 40320 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 32352, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32352, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 37210, "no_op_counter": 0, "num_processed_events": 32352, "on_delete_failure": 0, "on_delete_success": 32352, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 8411476, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "101572": "2022-03-02T07:07:48Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "101572": "2022-03-02T07:07:48Z" }, "v8worker_events_lost": 0 }, "function_id": 451640982, "function_name": "bucket_op", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 13600, "50": 300, "80": 400, "90": 500, "95": 800, "99": 2000 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/bucket_op_0": 101572 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 41983, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 37197, "dcp_deletion_sent_to_worker": 37197, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 750, "failure_stats": 750, "latency_stats": 750, "lcb_exception_stats": 750, "log_level": 1, "num_processed_events": 37197, "processed_events_size": 9670980, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 40320 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 37197, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 37197, "dcp_delete_msg_counter": 37197, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 37197, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 41979, "no_op_counter": 0, "num_processed_events": 37197, "on_delete_failure": 0, "on_delete_success": 37197, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 9670980, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "101600": "2022-03-02T07:07:48Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "101600": "2022-03-02T07:07:48Z" }, "v8worker_events_lost": 0 }, "function_id": 2474913908, "function_name": "curl", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 35500, "50": 1200, "80": 1400, "90": 1700, "95": 2100, "99": 4900 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/curl_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/curl_0": 101600 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 43494, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 38789, "dcp_deletion_sent_to_worker": 38789, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 734, "failure_stats": 734, "latency_stats": 734, "lcb_exception_stats": 734, "log_level": 1, "num_processed_events": 38789, "processed_events_size": 10084860, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 40320 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 38789, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 38789, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 43490, "no_op_counter": 0, "num_processed_events": 38789, "on_delete_failure": 0, "on_delete_success": 38789, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 10084860, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "101628": "2022-03-02T07:07:48Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "101628": "2022-03-02T07:07:48Z" }, "v8worker_events_lost": 0 }, "function_id": 3969969105, "function_name": "n1ql", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 94300, "50": 2500, "80": 2900, "90": 3400, "95": 4300, "99": 8100 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/n1ql_0": 101628 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 44949, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 40320, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 3, "dcp_mutation_suppressed_counter": 3, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 719, "failure_stats": 719, "latency_stats": 719, "lcb_exception_stats": 719, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 10478172, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 44945, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 10478172, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "110939": "2022-03-02T07:07:48Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "110939": "2022-03-02T07:07:48Z" }, "v8worker_events_lost": 0 }, "function_id": 1628018449, "function_name": "sbm", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 900, "50": 200, "80": 300, "90": 400, "95": 700, "99": 700 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/sbm_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/sbm_0": 110939 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 45867, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 40320, "dcp_deletion_sent_to_worker": 40320, "dcp_seqno_advanced": 1024, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 698, "failure_stats": 698, "latency_stats": 698, "lcb_exception_stats": 698, "log_level": 1, "num_processed_events": 41344, "processed_events_size": 10576476, "thr_count": 1, "thr_map": 1, "timer_events": 50118, "timer_responses_received": 40320, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 45856, "no_op_counter": 1024, "num_processed_events": 41344, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 10576476, "timer_callback_failure": 0, "timer_callback_success": 50118, "timer_cancel_counter": 0, "timer_create_counter": 40320, "timer_create_failure": 0, "timer_msg_counter": 50118, "timer_responses_sent": 0, "timestamp": { "126872": "2022-03-02T07:07:48Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "126872": "2022-03-02T07:07:48Z" }, "v8worker_events_lost": 0 }, "function_id": 3391603128, "function_name": "timers", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 20400, "50": 800, "80": 1100, "90": 1300, "95": 1700, "99": 3600 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/timers_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/timers_0": 126872 } } ] 2022-03-01 23:07:53 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Execution stats update_success: 0 update_failure: 0 delete_success: 32352 delete_failure: 0 2022-03-01 23:07:53 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 23:07:53 | INFO | MainProcess | test_thread | [eventing_base.print_eventing_stats_from_all_eventing_nodes] Stats for Node 172.23.106.77 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 43514, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 38789, "dcp_deletion_sent_to_worker": 38789, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 738, "failure_stats": 738, "latency_stats": 738, "lcb_exception_stats": 738, "log_level": 1, "num_processed_events": 38789, "processed_events_size": 10084860, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 40320 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 38789, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 38789, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 43510, "no_op_counter": 0, "num_processed_events": 38789, "on_delete_failure": 0, "on_delete_success": 38789, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 10084860, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "101628": "2022-03-02T07:07:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "101628": "2022-03-02T07:07:52Z" }, "v8worker_events_lost": 0 }, "function_id": 3969969105, "function_name": "n1ql", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 94300, "50": 2500, "80": 2900, "90": 3400, "95": 4300, "99": 8100 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/n1ql_0": 101628 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 44969, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 40320, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 3, "dcp_mutation_suppressed_counter": 3, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 723, "failure_stats": 723, "latency_stats": 723, "lcb_exception_stats": 723, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 10478172, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 44965, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 10478172, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "110939": "2022-03-02T07:07:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "110939": "2022-03-02T07:07:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1628018449, "function_name": "sbm", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 900, "50": 200, "80": 300, "90": 400, "95": 700, "99": 700 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/sbm_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/sbm_0": 110939 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 45887, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 40320, "dcp_deletion_sent_to_worker": 40320, "dcp_seqno_advanced": 1024, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 702, "failure_stats": 702, "latency_stats": 702, "lcb_exception_stats": 702, "log_level": 1, "num_processed_events": 41344, "processed_events_size": 10576476, "thr_count": 1, "thr_map": 1, "timer_events": 50118, "timer_responses_received": 40320, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 45876, "no_op_counter": 1024, "num_processed_events": 41344, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 10576476, "timer_callback_failure": 0, "timer_callback_success": 50118, "timer_cancel_counter": 0, "timer_create_counter": 40320, "timer_create_failure": 0, "timer_msg_counter": 50118, "timer_responses_sent": 0, "timestamp": { "126872": "2022-03-02T07:07:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "126872": "2022-03-02T07:07:52Z" }, "v8worker_events_lost": 0 }, "function_id": 3391603128, "function_name": "timers", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 20400, "50": 800, "80": 1100, "90": 1300, "95": 1700, "99": 3600 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/timers_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/timers_0": 126872 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 37234, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 32352, "dcp_deletion_sent_to_worker": 32352, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 769, "failure_stats": 769, "latency_stats": 769, "lcb_exception_stats": 769, "log_level": 1, "num_processed_events": 32352, "processed_events_size": 8411476, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 40320 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 32352, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32352, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 37230, "no_op_counter": 0, "num_processed_events": 32352, "on_delete_failure": 0, "on_delete_success": 32352, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 8411476, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "101572": "2022-03-02T07:07:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "101572": "2022-03-02T07:07:52Z" }, "v8worker_events_lost": 0 }, "function_id": 451640982, "function_name": "bucket_op", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 13600, "50": 300, "80": 400, "90": 500, "95": 800, "99": 2000 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/bucket_op_0": 101572 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 42003, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 37197, "dcp_deletion_sent_to_worker": 37197, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 754, "failure_stats": 754, "latency_stats": 754, "lcb_exception_stats": 754, "log_level": 1, "num_processed_events": 37197, "processed_events_size": 9670980, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 40320 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 37197, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 37197, "dcp_delete_msg_counter": 37197, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 37197, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 41999, "no_op_counter": 0, "num_processed_events": 37197, "on_delete_failure": 0, "on_delete_success": 37197, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 9670980, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "101600": "2022-03-02T07:07:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "101600": "2022-03-02T07:07:52Z" }, "v8worker_events_lost": 0 }, "function_id": 2474913908, "function_name": "curl", "function_scope": { "bucket": "src_bucket", "scope": "_default" }, "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_src_bucket/_default/curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 35500, "50": 1200, "80": 1400, "90": 1700, "95": 2100, "99": 4900 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.77:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.106.77:8096": { "worker_src_bucket/_default/curl_0": "[0-1023]" } }, "worker_pids": { "worker_src_bucket/_default/curl_0": 101600 } } ] 2022-03-01 23:07:53 | INFO | MainProcess | test_thread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has started ============== 2022-03-01 23:07:53 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.77 port:8091 ssh_username:root] 2022-03-01 23:07:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 23:07:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 23:07:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:07:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.77:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.64:8091 => {'services': ['kv'], 'cpu_utilization': 2.820448249811131, 'mem_free': 2909327360, 'mem_total': 3945984000, 'swap_mem_used': 184197120, 'swap_mem_total': 2147479552} 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.74:8091 => {'services': ['index'], 'cpu_utilization': 1.433601609657948, 'mem_free': 3170168832, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.75:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.628140703517588, 'mem_free': 3190738944, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.77:8091 => {'services': ['eventing'], 'cpu_utilization': 2.149721800708144, 'mem_free': 2945150976, 'mem_total': 3945984000, 'swap_mem_used': 0, 'swap_mem_total': 2147479552} 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 23:07:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 23:07:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:07:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 23:07:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 23:07:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 23:07:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:07:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 23:07:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 23:07:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 23:07:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:07:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 23:07:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 23:07:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 23:07:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:07:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 23:07:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 23:07:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 23:07:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:07:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 23:07:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 23:07:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 23:07:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:07:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 23:07:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 23:07:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 23:07:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:07:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 23:08:07 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_eventing_rebalance_swap_delete_recreate_collections ============== 2022-03-01 23:08:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.106.64 2022-03-01 23:08:08 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-03-01 23:08:08 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.64 existing buckets : ['metadata', 'src_bucket'] 2022-03-01 23:08:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : dst_bucket from 172.23.106.64 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.64 existing buckets : ['src_bucket'] 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : metadata from 172.23.106.64 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.64 existing buckets : [] 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : src_bucket from 172.23.106.64 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.67, nothing to delete 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.73, nothing to delete 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.74, nothing to delete 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.75, nothing to delete 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.76, nothing to delete 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.77, nothing to delete 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.78, nothing to delete 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-03-01 23:08:09 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.106.64:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.106.64,ns_1@172.23.106.74,ns_1@172.23.106.75,ns_1@172.23.106.77', 'ejectedNodes': 'ns_1@172.23.106.74,ns_1@172.23.106.75,ns_1@172.23.106.77', 'user': 'Administrator', 'password': 'password'} 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2022-03-01 23:08:09 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-03-01 23:08:19 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % 2022-03-01 23:08:29 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % 2022-03-01 23:08:49 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 40.05 seconds 2022-03-01 23:08:49 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 23:08:59 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.106.64 port:8091 ssh_username:root ? [('ns_1@172.23.106.74', 8091), ('ns_1@172.23.106.75', 8091), ('ns_1@172.23.106.77', 8091)] 2022-03-01 23:08:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.64:8091 2022-03-01 23:08:59 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 23:08:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.64:8091 is running 2022-03-01 23:08:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.67:8091 2022-03-01 23:08:59 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 23:08:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.67:8091 is running 2022-03-01 23:08:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.73:8091 2022-03-01 23:08:59 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 23:08:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.73:8091 is running 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.74:8091 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.74:8091 is running 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.75:8091 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.75:8091 is running 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.76:8091 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.76:8091 is running 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.77:8091 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.77:8091 is running 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.78:8091 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.78:8091 is running 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_eventing_rebalance_swap_delete_recreate_collections ============== 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-03-01 23:09:00 | INFO | MainProcess | test_thread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has completed ============== 2022-03-01 23:09:00 | INFO | MainProcess | Thread-368 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.64 with username:root, attempt#1 of 5 2022-03-01 23:09:00 | INFO | MainProcess | Thread-369 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.67 with username:root, attempt#1 of 5 2022-03-01 23:09:00 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.74 with username:root, attempt#1 of 5 2022-03-01 23:09:00 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.73 with username:root, attempt#1 of 5 2022-03-01 23:09:00 | INFO | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.76 with username:root, attempt#1 of 5 2022-03-01 23:09:00 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.75 with username:root, attempt#1 of 5 2022-03-01 23:09:00 | INFO | MainProcess | Thread-377 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.77 with username:root, attempt#1 of 5 2022-03-01 23:09:00 | INFO | MainProcess | Thread-380 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.78 with username:root, attempt#1 of 5 2022-03-01 23:09:00 | INFO | MainProcess | Thread-368 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.64 as root 2022-03-01 23:09:00 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.74 as root 2022-03-01 23:09:00 | INFO | MainProcess | Thread-369 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.67 as root 2022-03-01 23:09:00 | INFO | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.76 as root 2022-03-01 23:09:00 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.73 as root 2022-03-01 23:09:00 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.75 as root 2022-03-01 23:09:00 | INFO | MainProcess | Thread-377 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.77 as root 2022-03-01 23:09:00 | INFO | MainProcess | Thread-380 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.78 as root 2022-03-01 23:09:00 | INFO | MainProcess | Thread-372 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:09:00 | INFO | MainProcess | Thread-368 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:09:00 | INFO | MainProcess | Thread-374 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:09:00 | INFO | MainProcess | Thread-377 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:09:00 | INFO | MainProcess | Thread-380 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:09:00 | INFO | MainProcess | Thread-369 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:09:00 | INFO | MainProcess | Thread-373 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:09:00 | INFO | MainProcess | Thread-370 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 23:09:00 | INFO | MainProcess | Thread-372 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.106.74 2022-03-01 23:09:00 | INFO | MainProcess | Thread-372 | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: /opt/couchbase/bin/cbcollect_info 172.23.106.74-20220301-2309-diag.zip 2022-03-01 23:09:00 | INFO | MainProcess | Thread-369 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.106.67 2022-03-01 23:09:00 | INFO | MainProcess | Thread-369 | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: /opt/couchbase/bin/cbcollect_info 172.23.106.67-20220301-2309-diag.zip 2022-03-01 23:09:00 | INFO | MainProcess | Thread-373 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.106.75 2022-03-01 23:09:00 | INFO | MainProcess | Thread-373 | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: /opt/couchbase/bin/cbcollect_info 172.23.106.75-20220301-2309-diag.zip 2022-03-01 23:09:00 | INFO | MainProcess | Thread-380 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.106.78 2022-03-01 23:09:00 | INFO | MainProcess | Thread-380 | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: /opt/couchbase/bin/cbcollect_info 172.23.106.78-20220301-2309-diag.zip 2022-03-01 23:09:00 | INFO | MainProcess | Thread-374 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.106.76 2022-03-01 23:09:00 | INFO | MainProcess | Thread-374 | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: /opt/couchbase/bin/cbcollect_info 172.23.106.76-20220301-2309-diag.zip 2022-03-01 23:09:00 | INFO | MainProcess | Thread-368 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.106.64 2022-03-01 23:09:00 | INFO | MainProcess | Thread-368 | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: /opt/couchbase/bin/cbcollect_info 172.23.106.64-20220301-2309-diag.zip 2022-03-01 23:09:00 | INFO | MainProcess | Thread-370 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.106.73 2022-03-01 23:09:00 | INFO | MainProcess | Thread-370 | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: /opt/couchbase/bin/cbcollect_info 172.23.106.73-20220301-2309-diag.zip 2022-03-01 23:09:00 | INFO | MainProcess | Thread-377 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.106.77 2022-03-01 23:09:00 | INFO | MainProcess | Thread-377 | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: /opt/couchbase/bin/cbcollect_info 172.23.106.77-20220301-2309-diag.zip 2022-03-01 23:10:52 | INFO | MainProcess | Thread-369 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.106.67 2022-03-01 23:10:52 | INFO | MainProcess | Thread-369 | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: rm -f /root/172.23.106.67-20220301-2309-diag.zip 2022-03-01 23:10:52 | INFO | MainProcess | Thread-369 | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 23:10:52 | INFO | MainProcess | Thread-373 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.106.75 2022-03-01 23:10:53 | INFO | MainProcess | Thread-373 | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: rm -f /root/172.23.106.75-20220301-2309-diag.zip 2022-03-01 23:10:53 | INFO | MainProcess | Thread-373 | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 23:10:53 | INFO | MainProcess | Thread-370 | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 23:10:53 | INFO | MainProcess | Thread-377 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.106.73 2022-03-01 23:10:53 | INFO | MainProcess | Thread-370 | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: rm -f /root/172.23.106.73-20220301-2309-diag.zip 2022-03-01 23:10:53 | INFO | MainProcess | Thread-370 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.106.77 2022-03-01 23:10:53 | INFO | MainProcess | Thread-377 | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: rm -f /root/172.23.106.77-20220301-2309-diag.zip 2022-03-01 23:10:53 | INFO | MainProcess | Thread-377 | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 23:10:54 | INFO | MainProcess | Thread-372 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.106.74 2022-03-01 23:10:54 | INFO | MainProcess | Thread-372 | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: rm -f /root/172.23.106.74-20220301-2309-diag.zip 2022-03-01 23:10:54 | INFO | MainProcess | Thread-372 | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 23:11:16 | INFO | MainProcess | Thread-374 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.106.76 2022-03-01 23:11:16 | INFO | MainProcess | Thread-374 | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: rm -f /root/172.23.106.76-20220301-2309-diag.zip 2022-03-01 23:11:16 | INFO | MainProcess | Thread-374 | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 23:11:22 | INFO | MainProcess | Thread-380 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.106.78 2022-03-01 23:11:22 | INFO | MainProcess | Thread-380 | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: rm -f /root/172.23.106.78-20220301-2309-diag.zip 2022-03-01 23:11:22 | INFO | MainProcess | Thread-380 | [remote_util.execute_command_raw] command executed successfully with root 2022-03-01 23:11:29 | INFO | MainProcess | Thread-368 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.106.64 2022-03-01 23:11:29 | INFO | MainProcess | Thread-368 | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: rm -f /root/172.23.106.64-20220301-2309-diag.zip 2022-03-01 23:11:30 | INFO | MainProcess | Thread-368 | [remote_util.execute_command_raw] command executed successfully with root ERROR ====================================================================== ERROR: test_eventing_rebalance_swap_delete_recreate_collections (eventing.eventing_rebalance_collection.EventingRebalanceCollection) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/eventing/eventing_rebalance_collection.py", line 583, in test_eventing_rebalance_swap_delete_recreate_collections self.verify_all_handler(0) File "pytests/eventing/eventing_rebalance_collection.py", line 80, in verify_all_handler self.verify_doc_count_collections("dst_bucket.scope_1.coll_1", number_of_docs) File "./pytests/eventing/eventing_base.py", line 1068, in verify_doc_count_collections actual_count, expected_count, total_dcp_backlog, timers_in_past, lcb)) Exception: duplicated data in destination bucket which is not expected. Current : 311 Expected : 0 dcp_backlog : 40320 TIMERS_IN_PAST : 0 lcb_exceptions : {'172.23.106.77': {}} ---------------------------------------------------------------------- Ran 1 test in 1461.660s FAILED (errors=1)