Starting Test: test_eventing_rebalance_in_kill_eventing_producer 2021-07-15 23:14:08 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.64 with username:root 2021-07-15 23:14:08 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.64 2021-07-15 23:14:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:14:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:14:08 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 6.6.2-9555-enterprise 2021-07-15 23:14:08 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'6.6.2-9555-enterprise', u'6.6.2-9555-enterprise', u'6.6.2-9555-enterprise'] 2021-07-15 23:14:08 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_eventing_rebalance_in_kill_eventing_producer============== 2021-07-15 23:14:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'dst_bucket', u'metadata', u'src_bucket'] on 172.23.106.64 2021-07-15 23:14:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket dst_bucket ... 2021-07-15 23:14:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : dst_bucket from 172.23.106.64 2021-07-15 23:14:10 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-07-15 23:14:10 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.64 existing buckets : [u'metadata', u'src_bucket'] 2021-07-15 23:14:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket metadata ... 2021-07-15 23:14:11 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : metadata from 172.23.106.64 2021-07-15 23:14:11 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-07-15 23:14:11 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.64 existing buckets : [u'src_bucket'] 2021-07-15 23:14:11 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket src_bucket ... 2021-07-15 23:14:12 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : src_bucket from 172.23.106.64 2021-07-15 23:14:12 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-07-15 23:14:12 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.64 existing buckets : [] 2021-07-15 23:14:12 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] sleep 2 seconds to make sure all buckets ([u'dst_bucket', u'metadata', u'src_bucket']) were deleted completely. 2021-07-15 23:14:14 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-07-15 23:14:14 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.106.73,ns_1@172.23.106.67', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.106.73,ns_1@172.23.106.67,ns_1@172.23.106.64'} 2021-07-15 23:14:14 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2021-07-15 23:14:14 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-07-15 23:14:24 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % 2021-07-15 23:14:34 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % 2021-07-15 23:14:54 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 40.07 seconds 2021-07-15 23:14:54 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.106.64 port:8091 ssh_username:root ? [(u'ns_1@172.23.106.73', 8091), (u'ns_1@172.23.106.67', 8091)] 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.64:8091 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.64:8091 is running 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.67:8091 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.67:8091 is running 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.73:8091 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.73:8091 is running 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.74:8091 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.74:8091 is running 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.75:8091 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.75:8091 is running 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.76:8091 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.76:8091 is running 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.77:8091 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.77:8091 is running 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.78:8091 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.78:8091 is running 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.79:8091 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.79:8091 is running 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.80:8091 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.80:8091 is running 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.64:8091 => {'swap_mem_used': 55746560, 'cpu_utilization': 0.2512562814070352, 'mem_free': 3089473536, 'swap_mem_total': 2147479552, 'mem_total': 3945984000, 'services': [u'kv']} 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-07-15 23:15:04 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED Cluster instance shutdown with force 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.64 with username:root 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.64 2021-07-15 23:15:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:05 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.64 with username:root 2021-07-15 23:15:05 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.64 2021-07-15 23:15:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:06 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.64 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:06 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-07-15 23:15:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: systemctl stop couchbase-server.service 2021-07-15 23:15:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:07 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-07-15 23:15:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-07-15 23:15:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-07-15 23:15:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:07 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.64 with username:root 2021-07-15 23:15:08 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.64 2021-07-15 23:15:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:08 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.64 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:08 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:15:08 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-07-15 23:15:09 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-07-15 23:15:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: systemctl start couchbase-server.service 2021-07-15 23:15:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:09 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.64:sleep for 5 secs. waiting for couchbase server to come up ... 2021-07-15 23:15:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-07-15 23:15:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:14 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 92748 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-07-15 23:15:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:15:14 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-07-15 23:15:14 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.67 with username:root 2021-07-15 23:15:14 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.67 2021-07-15 23:15:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:15 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.67 with username:root 2021-07-15 23:15:15 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.67 2021-07-15 23:15:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:16 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.67 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:16 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-07-15 23:15:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: systemctl stop couchbase-server.service 2021-07-15 23:15:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:17 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-07-15 23:15:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-07-15 23:15:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-07-15 23:15:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:17 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.67 with username:root 2021-07-15 23:15:18 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.67 2021-07-15 23:15:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:18 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.67 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:18 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:15:18 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-07-15 23:15:19 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-07-15 23:15:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: systemctl start couchbase-server.service 2021-07-15 23:15:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:19 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.67:sleep for 5 secs. waiting for couchbase server to come up ... 2021-07-15 23:15:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-07-15 23:15:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:24 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 67610 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-07-15 23:15:24 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:15:24 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-07-15 23:15:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.73 with username:root 2021-07-15 23:15:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.73 2021-07-15 23:15:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:25 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.73 with username:root 2021-07-15 23:15:25 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.73 2021-07-15 23:15:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:26 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.73 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:26 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-07-15 23:15:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: systemctl stop couchbase-server.service 2021-07-15 23:15:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:27 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-07-15 23:15:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-07-15 23:15:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-07-15 23:15:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:27 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.73 with username:root 2021-07-15 23:15:28 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.73 2021-07-15 23:15:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:28 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.73 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:28 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:15:28 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-07-15 23:15:29 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-07-15 23:15:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: systemctl start couchbase-server.service 2021-07-15 23:15:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:29 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.73:sleep for 5 secs. waiting for couchbase server to come up ... 2021-07-15 23:15:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-07-15 23:15:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:34 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 96681 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-07-15 23:15:34 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:15:34 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-07-15 23:15:34 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.74 with username:root 2021-07-15 23:15:34 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.74 2021-07-15 23:15:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:35 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.74 with username:root 2021-07-15 23:15:35 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.74 2021-07-15 23:15:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:36 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.74 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:36 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-07-15 23:15:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: systemctl stop couchbase-server.service 2021-07-15 23:15:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:37 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-07-15 23:15:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-07-15 23:15:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-07-15 23:15:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:37 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.74 with username:root 2021-07-15 23:15:37 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.74 2021-07-15 23:15:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:38 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.74 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:38 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:15:38 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-07-15 23:15:38 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-07-15 23:15:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: systemctl start couchbase-server.service 2021-07-15 23:15:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:38 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.74:sleep for 5 secs. waiting for couchbase server to come up ... 2021-07-15 23:15:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-07-15 23:15:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:43 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 120582 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-07-15 23:15:43 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:15:44 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-07-15 23:15:44 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.75 with username:root 2021-07-15 23:15:44 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.75 2021-07-15 23:15:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:45 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.75 with username:root 2021-07-15 23:15:45 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.75 2021-07-15 23:15:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:45 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.75 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:45 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-07-15 23:15:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: systemctl stop couchbase-server.service 2021-07-15 23:15:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:47 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-07-15 23:15:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-07-15 23:15:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-07-15 23:15:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:47 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.75 with username:root 2021-07-15 23:15:47 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.75 2021-07-15 23:15:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:48 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.75 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:48 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:15:48 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-07-15 23:15:48 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-07-15 23:15:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: systemctl start couchbase-server.service 2021-07-15 23:15:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:48 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.75:sleep for 5 secs. waiting for couchbase server to come up ... 2021-07-15 23:15:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-07-15 23:15:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:53 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 26438 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-07-15 23:15:53 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:15:54 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-07-15 23:15:54 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.76 with username:root 2021-07-15 23:15:54 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.76 2021-07-15 23:15:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:54 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.76 with username:root 2021-07-15 23:15:55 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.76 2021-07-15 23:15:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:55 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.76 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:55 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-07-15 23:15:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: systemctl stop couchbase-server.service 2021-07-15 23:15:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:57 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-07-15 23:15:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-07-15 23:15:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-07-15 23:15:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:57 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.76 with username:root 2021-07-15 23:15:57 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.76 2021-07-15 23:15:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:15:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:15:58 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.76 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:15:58 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:15:58 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-07-15 23:15:58 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-07-15 23:15:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: systemctl start couchbase-server.service 2021-07-15 23:15:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:15:58 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.76:sleep for 5 secs. waiting for couchbase server to come up ... 2021-07-15 23:16:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-07-15 23:16:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:03 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 82186 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-07-15 23:16:03 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:16:03 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-07-15 23:16:04 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.77 with username:root 2021-07-15 23:16:04 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.77 2021-07-15 23:16:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:04 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.77 with username:root 2021-07-15 23:16:04 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.77 2021-07-15 23:16:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:05 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.77 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:16:05 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-07-15 23:16:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: systemctl stop couchbase-server.service 2021-07-15 23:16:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:07 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-07-15 23:16:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-07-15 23:16:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-07-15 23:16:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:07 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.77 with username:root 2021-07-15 23:16:07 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.77 2021-07-15 23:16:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:08 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.77 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:16:08 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:16:08 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-07-15 23:16:08 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-07-15 23:16:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: systemctl start couchbase-server.service 2021-07-15 23:16:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:08 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.77:sleep for 5 secs. waiting for couchbase server to come up ... 2021-07-15 23:16:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-07-15 23:16:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:13 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 121478 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-07-15 23:16:13 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:16:13 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-07-15 23:16:14 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.78 with username:root 2021-07-15 23:16:14 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.78 2021-07-15 23:16:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:14 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.78 with username:root 2021-07-15 23:16:14 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.78 2021-07-15 23:16:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:15 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.78 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:16:15 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-07-15 23:16:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: systemctl stop couchbase-server.service 2021-07-15 23:16:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:17 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-07-15 23:16:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-07-15 23:16:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-07-15 23:16:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:17 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.78 with username:root 2021-07-15 23:16:17 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.78 2021-07-15 23:16:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:18 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.78 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:16:18 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:16:18 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-07-15 23:16:18 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-07-15 23:16:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: systemctl start couchbase-server.service 2021-07-15 23:16:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:18 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.78:sleep for 5 secs. waiting for couchbase server to come up ... 2021-07-15 23:16:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-07-15 23:16:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:23 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 17986 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-07-15 23:16:23 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:16:23 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-07-15 23:16:23 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.79 with username:root 2021-07-15 23:16:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.79 2021-07-15 23:16:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.79 with username:root 2021-07-15 23:16:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.79 2021-07-15 23:16:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:25 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.79 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:16:25 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-07-15 23:16:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.79: systemctl stop couchbase-server.service 2021-07-15 23:16:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:27 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-07-15 23:16:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.79: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-07-15 23:16:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.79: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-07-15 23:16:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:27 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.79 with username:root 2021-07-15 23:16:27 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.79 2021-07-15 23:16:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:28 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.79 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:16:28 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:16:28 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-07-15 23:16:28 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-07-15 23:16:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.79: systemctl start couchbase-server.service 2021-07-15 23:16:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:28 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.79:sleep for 5 secs. waiting for couchbase server to come up ... 2021-07-15 23:16:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.79: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-07-15 23:16:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:33 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 30229 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-07-15 23:16:33 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:16:33 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-07-15 23:16:33 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.80 with username:root 2021-07-15 23:16:34 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.80 2021-07-15 23:16:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:34 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.80 with username:root 2021-07-15 23:16:34 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.80 2021-07-15 23:16:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:35 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.80 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:16:35 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-07-15 23:16:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.80: systemctl stop couchbase-server.service 2021-07-15 23:16:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:37 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-07-15 23:16:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.80: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-07-15 23:16:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.80: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-07-15 23:16:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:37 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.80 with username:root 2021-07-15 23:16:37 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.80 2021-07-15 23:16:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:38 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.80 **** The version file /opt/couchbase/ VERSION.txt exists 2021-07-15 23:16:38 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:16:38 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-07-15 23:16:38 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-07-15 23:16:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.80: systemctl start couchbase-server.service 2021-07-15 23:16:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:38 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.80:sleep for 5 secs. waiting for couchbase server to come up ... 2021-07-15 23:16:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.80: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-07-15 23:16:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:43 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 56498 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-07-15 23:16:43 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-07-15 23:16:43 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-07-15 23:16:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2021-07-15 23:16:54 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.64 port:8091 ssh_username:root, nodes/self 2021-07-15 23:16:54 | INFO | MainProcess | Cluster_Thread | [task.execute] {'ip': u'172.23.106.64', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.106.64', 'uptime': u'102', 'mcdMemoryReserved': 3010, 'storageTotalRam': 3763, 'hostname': u'172.23.106.64:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9555-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3488313344, 'memoryTotal': 3945984000, 'memoryQuota': 1592, 'mcdMemoryAllocated': 3010, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2021-07-15 23:16:54 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.64:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-07-15 23:16:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2021-07-15 23:16:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.106.64,8091,['kv']) 2021-07-15 23:16:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.106.64: 8091:services=kv&password=password&hostname=172.23.106.64&user=Administrator 2021-07-15 23:16:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.64:8091:username=Administrator&password=password&port=8091 2021-07-15 23:16:54 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.106.64 with username:root 2021-07-15 23:16:54 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.106.64 2021-07-15 23:16:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.64:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.64:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.67 port:8091 ssh_username:root, nodes/self 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [task.execute] {'ip': u'172.23.106.67', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.106.67', 'uptime': u'93', 'mcdMemoryReserved': 3010, 'storageTotalRam': 3763, 'hostname': u'172.23.106.67:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9555-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3488911360, 'memoryTotal': 3945984000, 'memoryQuota': 1592, 'mcdMemoryAllocated': 3010, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2021-07-15 23:16:55 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.67:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.67:8091:username=Administrator&password=password&port=8091 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.106.67 with username:root 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.106.67 2021-07-15 23:16:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.67:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.67:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.73 port:8091 ssh_username:root, nodes/self 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [task.execute] {'ip': u'172.23.106.73', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.106.73', 'uptime': u'82', 'mcdMemoryReserved': 3010, 'storageTotalRam': 3763, 'hostname': u'172.23.106.73:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9555-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3464380416, 'memoryTotal': 3945984000, 'memoryQuota': 1592, 'mcdMemoryAllocated': 3010, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2021-07-15 23:16:56 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.73:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.73:8091:username=Administrator&password=password&port=8091 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.106.73 with username:root 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.106.73 2021-07-15 23:16:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.73: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.73:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.73:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.74 port:8091 ssh_username:root, nodes/self 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [task.execute] {'ip': u'172.23.106.74', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'72', 'mcdMemoryReserved': 3010, 'storageTotalRam': 3763, 'hostname': u'172.23.106.74:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9555-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3464450048, 'memoryTotal': 3945984000, 'memoryQuota': 1592, 'mcdMemoryAllocated': 3010, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2021-07-15 23:16:57 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.74:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.74:8091:username=Administrator&password=password&port=8091 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.106.74 with username:root 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.106.74 2021-07-15 23:16:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.74:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.74:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.75 port:8091 ssh_username:root, nodes/self 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [task.execute] {'ip': u'172.23.106.75', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'63', 'mcdMemoryReserved': 3010, 'storageTotalRam': 3763, 'hostname': u'172.23.106.75:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9555-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3462922240, 'memoryTotal': 3945984000, 'memoryQuota': 1592, 'mcdMemoryAllocated': 3010, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2021-07-15 23:16:58 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.75:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.75:8091:username=Administrator&password=password&port=8091 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.106.75 with username:root 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.106.75 2021-07-15 23:16:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.75: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.75:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.75:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.76 port:8091 ssh_username:root, nodes/self 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [task.execute] {'ip': u'172.23.106.76', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'57', 'mcdMemoryReserved': 3010, 'storageTotalRam': 3763, 'hostname': u'172.23.106.76:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9555-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3462492160, 'memoryTotal': 3945984000, 'memoryQuota': 1592, 'mcdMemoryAllocated': 3010, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2021-07-15 23:16:59 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.76:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.76:8091:username=Administrator&password=password&port=8091 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.106.76 with username:root 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.106.76 2021-07-15 23:16:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.76: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.76:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.76:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.77 port:8091 ssh_username:root, nodes/self 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [task.execute] {'ip': u'172.23.106.77', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'52', 'mcdMemoryReserved': 3010, 'storageTotalRam': 3763, 'hostname': u'172.23.106.77:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9555-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3467108352, 'memoryTotal': 3945984000, 'memoryQuota': 1592, 'mcdMemoryAllocated': 3010, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2021-07-15 23:17:05 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.77:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.77:8091:username=Administrator&password=password&port=8091 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.106.77 with username:root 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.106.77 2021-07-15 23:17:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.77: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.77:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.77:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.78 port:8091 ssh_username:root, nodes/self 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [task.execute] {'ip': u'172.23.106.78', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'42', 'mcdMemoryReserved': 3010, 'storageTotalRam': 3763, 'hostname': u'172.23.106.78:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9555-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3464347648, 'memoryTotal': 3945984000, 'memoryQuota': 1592, 'mcdMemoryAllocated': 3010, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2021-07-15 23:17:06 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.78:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.78:8091:username=Administrator&password=password&port=8091 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.106.78 with username:root 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.106.78 2021-07-15 23:17:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.78: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.78:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.78:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.79 port:8091 ssh_username:root, nodes/self 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [task.execute] {'ip': u'172.23.106.79', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'33', 'mcdMemoryReserved': 3010, 'storageTotalRam': 3763, 'hostname': u'172.23.106.79:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9555-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3469615104, 'memoryTotal': 3945984000, 'memoryQuota': 1592, 'mcdMemoryAllocated': 3010, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2021-07-15 23:17:07 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.79:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.79:8091:username=Administrator&password=password&port=8091 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.106.79 with username:root 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.106.79 2021-07-15 23:17:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.79: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.79:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.79:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.80 port:8091 ssh_username:root, nodes/self 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [task.execute] {'ip': u'172.23.106.80', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'22', 'mcdMemoryReserved': 3010, 'storageTotalRam': 3763, 'hostname': u'172.23.106.80:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9555-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3442012160, 'memoryTotal': 3945988096, 'memoryQuota': 1592, 'mcdMemoryAllocated': 3010, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2021-07-15 23:17:08 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.80:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2016 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.80:8091:username=Administrator&password=password&port=8091 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.106.80 with username:root 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.106.80 2021-07-15 23:17:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:17:09 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:17:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.80: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-07-15 23:17:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:17:09 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.80:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:17:09 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.80:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2021-07-15 23:17:09 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-07-15 23:17:09 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.106.64 **** 2021-07-15 23:17:09 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.106.64:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password 2021-07-15 23:17:09 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -"User was not found." 2021-07-15 23:17:09 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-07-15 23:17:14 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-07-15 23:17:14 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2021-07-15 23:17:15 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.67:8091 to cluster 2021-07-15 23:17:15 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.67:8091 to this cluster @172.23.106.64:8091 2021-07-15 23:17:17 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.73:8091 to cluster 2021-07-15 23:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.73:8091 to this cluster @172.23.106.64:8091 2021-07-15 23:17:18 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.106.73,ns_1@172.23.106.67,ns_1@172.23.106.64'} 2021-07-15 23:17:18 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-07-15 23:17:18 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-07-15 23:17:18 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-07-15 23:17:28 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-07-15 23:17:28 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.0299828053 sec 2021-07-15 23:17:28 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.64 with username:root 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.64 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.64: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.106.64 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_eventing_rebalance_in_kill_eventing_producer ============== 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.73:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.5, 'mem_free': 3392876544, 'swap_mem_total': 2147479552, 'mem_total': 3945984000, 'services': [u'index', u'n1ql']} 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.67:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.75, 'mem_free': 3443642368, 'swap_mem_total': 2147479552, 'mem_total': 3945984000, 'services': [u'eventing']} 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.64:8091 => {'swap_mem_used': 55746560, 'cpu_utilization': 0.2506265664160401, 'mem_free': 3453456384, 'swap_mem_total': 2147479552, 'mem_total': 3945984000, 'services': [u'kv']} 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.64 with username:root 2021-07-15 23:17:29 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.64 2021-07-15 23:17:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:17:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:17:32 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.73 port:8091 ssh_username:root] 2021-07-15 23:17:32 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root] 2021-07-15 23:17:32 | INFO | MainProcess | test_thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-07-15 23:17:32 | INFO | MainProcess | test_thread | [eventing_base.setUp] Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. 2021-07-15 23:17:32 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=330 2021-07-15 23:17:32 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-07-15 23:17:32 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : eventingMemoryQuota=512 2021-07-15 23:17:32 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.73 port:8091 ssh_username:root] 2021-07-15 23:17:32 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=700 2021-07-15 23:17:32 | INFO | MainProcess | test_thread | [eventing_recovery.setUp] 100 2021-07-15 23:17:32 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.64:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=100&authType=none&name=src_bucket&storageBackend=couchstore&flushEnabled=1&replicaNumber=1&compressionMode=passive&replicaIndex=1 2021-07-15 23:17:32 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket src_bucket 2021-07-15 23:17:32 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:33 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-07-15 23:17:33 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:34 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-07-15 23:17:34 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:34 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-07-15 23:17:34 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:34 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-07-15 23:17:34 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:34 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-07-15 23:17:34 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:17:35 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-07-15 23:17:36 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.64:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=100&authType=none&name=dst_bucket&storageBackend=couchstore&flushEnabled=1&replicaNumber=1&compressionMode=passive&replicaIndex=1 2021-07-15 23:17:36 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket dst_bucket 2021-07-15 23:17:36 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:37 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-07-15 23:17:37 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:37 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-07-15 23:17:37 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:37 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-07-15 23:17:37 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:38 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-07-15 23:17:38 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:38 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-07-15 23:17:38 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.106.64 to accept set ops 2021-07-15 23:17:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 dst_bucket 2021-07-15 23:17:38 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-07-15 23:17:39 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.64:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=400&authType=none&name=metadata&storageBackend=couchstore&flushEnabled=1&replicaNumber=1&compressionMode=passive&replicaIndex=1 2021-07-15 23:17:39 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket metadata 2021-07-15 23:17:39 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.106.64 to accept set ops 2021-07-15 23:17:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:40 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-07-15 23:17:40 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.106.64 to accept set ops 2021-07-15 23:17:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:40 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-07-15 23:17:40 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.106.64 to accept set ops 2021-07-15 23:17:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:41 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-07-15 23:17:41 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.106.64 to accept set ops 2021-07-15 23:17:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:41 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-07-15 23:17:41 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.106.64 to accept set ops 2021-07-15 23:17:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:41 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-07-15 23:17:41 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.106.64 to accept set ops 2021-07-15 23:17:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 metadata 2021-07-15 23:17:42 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-07-15 23:17:42 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.106.73 port:8091 ssh_username:root] 2021-07-15 23:17:42 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-07-15 23:17:42 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-07-15 23:17:42 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 213.283293ms 2021-07-15 23:17:42 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI 2021-07-15 23:17:42 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI 2021-07-15 23:17:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.692749267s 2021-07-15 23:17:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-07-15 23:17:44 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-07-15 23:17:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 25.013903ms 2021-07-15 23:17:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-07-15 23:17:44 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-07-15 23:17:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 19.778337ms 2021-07-15 23:17:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI 2021-07-15 23:17:44 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI 2021-07-15 23:17:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.577652328s 2021-07-15 23:17:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-07-15 23:17:45 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-07-15 23:17:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 33.468853ms 2021-07-15 23:17:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-07-15 23:17:45 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-07-15 23:17:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 19.65577ms 2021-07-15 23:17:45 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI 2021-07-15 23:17:45 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI 2021-07-15 23:17:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.513148556s 2021-07-15 23:17:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-07-15 23:17:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-07-15 23:17:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 30.151018ms 2021-07-15 23:17:47 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root] 2021-07-15 23:17:47 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] Deploying the following handler code : Function_296747078_test_eventing_rebalance_in_kill_eventing_producer with bindings: {'buckets': [{'access': 'rw', 'alias': 'dst_bucket', 'bucket_name': 'dst_bucket'}], 'source_bucket': 'src_bucket', 'curl': [], 'metadata_bucket': 'metadata'} and settings: {'cleanup_timers': False, 'worker_count': 3, 'skip_timer_threshold': 86400, 'dcp_stream_boundary': 'everything', 'use_memory_manager': True, 'sock_batch_size': 1, 'language_compatibility': '6.6.2', 'log_level': 'INFO', 'tick_duration': 5000, 'deadline_timeout': 21, 'timer_worker_pool_size': 3, 'deployment_status': True, 'description': 'Sample Description', 'dcp_gen_chan_size': 10000, 'timer_storage_chan_size': 10000, 'worker_queue_cap': 100000, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'processing_status': True, 'timer_processing_tick_interval': 500, 'data_chan_size': 10000, 'checkpoint_interval': 20000} 2021-07-15 23:17:47 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] function OnUpdate(doc, meta) { log("Doc created/updated", meta.id); var success = false; while(!success){ try{ var result= couchbase.insert(dst_bucket,meta,doc); log(result); success=result.success; if(result.error != undefined && result.error.name == "LCB_KEY_EEXISTS"){ break; } }catch(e){ log("error:",e); } } } function OnDelete(meta, options) { log("Doc deleted/expired", meta.id); var doc={"id":meta.id} var success = false; while(!success){ try{ var result = couchbase.delete(dst_bucket,doc); log(result); success=result.success; if(result.error != undefined && result.error.name == "LCB_KEY_ENOENT"){ break; } }catch(e){ log("error:",e); } } } 2021-07-15 23:17:48 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] deploy Application : { "code": 0, "info": { "status": "Stored function: 'Function_296747078_test_eventing_rebalance_in_kill_eventing_producer' in metakv", "warnings": null } } 2021-07-15 23:17:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to deployed... ... 2021-07-15 23:18:08 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to deployed... ... 2021-07-15 23:18:28 | INFO | MainProcess | test_thread | [basetestcase.load] create 20160 to src_bucket documents... 2021-07-15 23:18:29 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:18:44 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2021-07-15 23:18:44 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root] 2021-07-15 23:18:44 | INFO | MainProcess | test_thread | [eventing_base.pause_function] Pause Application : Function_296747078_test_eventing_rebalance_in_kill_eventing_producer 2021-07-15 23:18:44 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:19:04 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:19:24 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-07-15 23:19:25 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.74:8091 to cluster 2021-07-15 23:19:25 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.74:8091 to this cluster @172.23.106.64:8091 2021-07-15 23:19:27 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.106.73,ns_1@172.23.106.67,ns_1@172.23.106.74,ns_1@172.23.106.64'} 2021-07-15 23:19:27 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-07-15 23:19:27 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-07-15 23:19:27 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-07-15 23:19:29 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 37.50 % 2021-07-15 23:19:32 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >37.5% in 3.00740695 seconds 2021-07-15 23:19:32 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.67 with username:root 2021-07-15 23:19:32 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.67 2021-07-15 23:19:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:19:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:19:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: killall -9 eventing-producer 2021-07-15 23:19:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:19:33 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:19:37 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2021-07-15 23:19:37 | INFO | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.106.64: 2021-07-15 23:19:37 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.64', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {agent_died,<17531.3620.0>,\n {lost_connection,shutdown}}}.\nRebalance Operation Id = dca155c2abd515ac7ec2c2d992d7e438', u'shortText': u'message', u'serverTime': u'2021-07-15T23:19:33.307Z', u'module': u'ns_orchestrator', u'tstamp': 1626416373307, u'type': u'critical'} 2021-07-15 23:19:37 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.67', u'code': 0, u'text': u"Service 'eventing' exited with status 137. Restarting. Messages:\n2021-07-15T23:19:28.390-07:00 [Info] Producer::Serve [Function_296747078_test_eventing_rebalance_in_kill_eventing_producer:0] Got topology change msg: &{start-rebalance /eventing/rebalanceToken/5de6551860e9cd5dbf4f309eb54514ce} from super_supervisor\n2021-07-15T23:19:28.390-07:00 [Info] Producer::Serve [Function_296747078_test_eventing_rebalance_in_kill_eventing_producer:0] Planner status: true, before vbucket to node assignment as part of rebalance\n2021-07-15T23:19:28.392-07:00 [Info] SuperSupervisor::TopologyChangeNotifCallback [1] Apps in primary store: [Function_296747078_test_eventing_rebalance_in_kill_eventing_producer], running apps: map[Function_296747078_test_eventing_rebalance_in_kill_eventing_producer:Producer => function: Function_296747078_test_eventing_rebalance_in_kill_eventing_producer tcpPort: /tmp/127.0.0.1:8091_2_3592799128.sock]\n2021-07-15T23:19:28.393-07:00 [Info] SuperSupervisor::TopologyChangeNotifCallback [1] Function: Function_296747078_test_eventing_rebalance_in_kill_eventing_producer deployment_status: true processing_status: false runningProducer: Producer => function: Function_296747078_test_eventing_rebalance_in_kill_eventing_producer tcpPort: /tmp/127.0.0.1:8091_2_3592799128.sock\n", u'shortText': u'message', u'serverTime': u'2021-07-15T23:19:33.303Z', u'module': u'ns_log', u'tstamp': 1626416373303, u'type': u'info'} 2021-07-15 23:19:37 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.64', u'code': 0, u'text': u'Bucket "src_bucket" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2021-07-15T23:19:27.898Z', u'module': u'ns_vbucket_mover', u'tstamp': 1626416367898, u'type': u'info'} 2021-07-15 23:19:37 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.64', u'code': 0, u'text': u'Started rebalancing bucket src_bucket', u'shortText': u'message', u'serverTime': u'2021-07-15T23:19:27.798Z', u'module': u'ns_rebalancer', u'tstamp': 1626416367798, u'type': u'info'} 2021-07-15 23:19:37 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.64', u'code': 0, u'text': u'Bucket "dst_bucket" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2021-07-15T23:19:27.787Z', u'module': u'ns_vbucket_mover', u'tstamp': 1626416367787, u'type': u'info'} 2021-07-15 23:19:37 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.64', u'code': 0, u'text': u'Started rebalancing bucket dst_bucket', u'shortText': u'message', u'serverTime': u'2021-07-15T23:19:27.702Z', u'module': u'ns_rebalancer', u'tstamp': 1626416367702, u'type': u'info'} 2021-07-15 23:19:37 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.64', u'code': 0, u'text': u'Bucket "metadata" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2021-07-15T23:19:27.686Z', u'module': u'ns_vbucket_mover', u'tstamp': 1626416367686, u'type': u'info'} 2021-07-15 23:19:37 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.64', u'code': 0, u'text': u'Started rebalancing bucket metadata', u'shortText': u'message', u'serverTime': u'2021-07-15T23:19:27.587Z', u'module': u'ns_rebalancer', u'tstamp': 1626416367587, u'type': u'info'} 2021-07-15 23:19:37 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.64', u'code': 0, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.106.73','ns_1@172.23.106.67',\n 'ns_1@172.23.106.74','ns_1@172.23.106.64'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = dca155c2abd515ac7ec2c2d992d7e438", u'shortText': u'message', u'serverTime': u'2021-07-15T23:19:27.354Z', u'module': u'ns_orchestrator', u'tstamp': 1626416367354, u'type': u'info'} 2021-07-15 23:19:37 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.74', u'code': 0, u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', u'shortText': u'message', u'serverTime': u'2021-07-15T23:19:27.327Z', u'module': u'memcached_config_mgr', u'tstamp': 1626416367327, u'type': u'info'} [('/usr/lib64/python2.7/threading.py', 785, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 812, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 82, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 545, 'check', 'self.set_exception(ex)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')] Thu Jul 15 23:19:37 2021 2021-07-15 23:19:53 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:20:13 | INFO | MainProcess | test_thread | [eventing_base.resume_function] Settings after deleting dcp_stream_boundary : {'cleanup_timers': False, 'worker_count': 3, 'skip_timer_threshold': 86400, 'use_memory_manager': True, 'sock_batch_size': 1, 'language_compatibility': '6.6.2', 'log_level': 'INFO', 'tick_duration': 5000, 'deadline_timeout': 21, 'timer_worker_pool_size': 3, 'deployment_status': True, 'description': 'Sample Description', 'dcp_gen_chan_size': 10000, 'timer_storage_chan_size': 10000, 'worker_queue_cap': 100000, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'processing_status': True, 'timer_processing_tick_interval': 500, 'data_chan_size': 10000, 'checkpoint_interval': 20000} 2021-07-15 23:20:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.74 port:8091 ssh_username:root] 2021-07-15 23:20:14 | INFO | MainProcess | test_thread | [eventing_base.resume_function] Resume Application : Function_296747078_test_eventing_rebalance_in_kill_eventing_producer 2021-07-15 23:20:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to deployed... ... 2021-07-15 23:20:34 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to deployed... ... [('/usr/lib64/python2.7/threading.py', 785, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 812, '__bootstrap_inner', 'self.run()'), ('./testrunner.py', 543, 'run', '**self._Thread__kwargs)'), ('/usr/lib64/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib64/python2.7/unittest/case.py', 433, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.7/unittest/case.py', 369, 'run', 'testMethod()'), ('pytests/eventing/eventing_recovery.py', 886, 'test_eventing_rebalance_in_kill_eventing_producer', 'rebalance.result()'), ('lib/tasks/future.py', 153, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')] 2021-07-15 23:20:54 | INFO | MainProcess | test_thread | [eventing_recovery.test_eventing_rebalance_in_kill_eventing_producer] Rebalance failed as expected after eventing got killed: Rebalance Failed: {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2021-07-15 23:20:54 | INFO | MainProcess | test_thread | [eventing_base.resume_function] Settings after deleting dcp_stream_boundary : {'cleanup_timers': False, 'worker_count': 3, 'skip_timer_threshold': 86400, 'use_memory_manager': True, 'sock_batch_size': 1, 'language_compatibility': '6.6.2', 'log_level': 'INFO', 'tick_duration': 5000, 'deadline_timeout': 21, 'timer_worker_pool_size': 3, 'deployment_status': True, 'description': 'Sample Description', 'dcp_gen_chan_size': 10000, 'timer_storage_chan_size': 10000, 'worker_queue_cap': 100000, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'processing_status': True, 'timer_processing_tick_interval': 500, 'data_chan_size': 10000, 'checkpoint_interval': 20000} 2021-07-15 23:20:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.74 port:8091 ssh_username:root] 2021-07-15 23:20:54 | INFO | MainProcess | test_thread | [eventing_base.resume_function] Resume Application : Function_296747078_test_eventing_rebalance_in_kill_eventing_producer 2021-07-15 23:20:54 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to deployed... ... 2021-07-15 23:21:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to deployed... ... 2021-07-15 23:21:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.74 port:8091 ssh_username:root] 2021-07-15 23:21:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.74 port:8091 ssh_username:root] 2021-07-15 23:21:34 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Documents in source bucket : 20160 2021-07-15 23:21:34 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Final docs count... Current : 20160 Expected : 20160 2021-07-15 23:21:35 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Stats for Node 172.23.106.67 is [ { "dcp_feed_boundary": "from_prior", "event_processing_stats": { "agg_messages_sent_to_worker": 1583, "agg_queue_memory": 0, "agg_queue_memory_cap": 267386880, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_stream_req_counter": 512, "dcp_streamreq": 512, "execution_stats": 210, "failure_stats": 210, "latency_stats": 210, "lcb_exception_stats": 210, "log_level": 3, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 1571, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "68684": "2021-07-16T06:21:34Z", "68694": "2021-07-16T06:21:34Z", "68707": "2021-07-16T06:21:34Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "68684": "2021-07-16T06:21:34Z", "68694": "2021-07-16T06:21:34Z", "68707": "2021-07-16T06:21:34Z" }, "v8worker_events_lost": 0 }, "function_id": 3592799128, "function_name": "Function_296747078_test_eventing_rebalance_in_kill_eventing_producer", "gocb_creds_request_counter": 8, "internal_vb_distribution_stats": { "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_0": "[0-170]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_1": "[171-341]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_2": "[342-511]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 6, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.67:8096", "start_vb": 0, "vb_count": 512 }, { "host_name": "172.23.106.74:8096", "start_vb": 512, "vb_count": 512 } ], "vb_distribution_stats_from_metadata": { "172.23.106.67:8096": { "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_0": "[0-170]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_1": "[171-341]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_2": "[342-511]" }, "172.23.106.74:8096": { "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_0": "[512-682]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_1": "[683-853]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_2": "[854-1023]" } }, "worker_pids": { "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_0": 68707, "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_1": 68684, "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_2": 68694 } } ] 2021-07-15 23:21:36 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Stats for Node 172.23.106.74 is [ { "dcp_feed_boundary": "from_prior", "event_processing_stats": { "agg_messages_sent_to_worker": 1598, "agg_queue_memory": 0, "agg_queue_memory_cap": 267386880, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_stream_req_counter": 512, "dcp_streamreq": 512, "execution_stats": 213, "failure_stats": 213, "latency_stats": 213, "lcb_exception_stats": 213, "log_level": 3, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 1586, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "121512": "2021-07-16T06:21:35Z", "121524": "2021-07-16T06:21:35Z", "121535": "2021-07-16T06:21:35Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "121512": "2021-07-16T06:21:35Z", "121524": "2021-07-16T06:21:35Z", "121535": "2021-07-16T06:21:35Z" }, "v8worker_events_lost": 0 }, "function_id": 3592799128, "function_name": "Function_296747078_test_eventing_rebalance_in_kill_eventing_producer", "gocb_creds_request_counter": 8, "internal_vb_distribution_stats": { "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_0": "[512-682]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_1": "[683-853]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_2": "[854-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 6, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.106.67:8096", "start_vb": 0, "vb_count": 512 }, { "host_name": "172.23.106.74:8096", "start_vb": 512, "vb_count": 512 } ], "vb_distribution_stats_from_metadata": { "172.23.106.67:8096": { "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_0": "[0-170]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_1": "[171-341]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_2": "[342-511]" }, "172.23.106.74:8096": { "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_0": "[512-682]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_1": "[683-853]", "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_2": "[854-1023]" } }, "worker_pids": { "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_0": 121512, "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_1": 121524, "worker_Function_296747078_test_eventing_rebalance_in_kill_eventing_producer_2": 121535 } } ] 2021-07-15 23:21:37 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2021-07-15 23:21:37 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.106.73,ns_1@172.23.106.67,ns_1@172.23.106.74,ns_1@172.23.106.64'} 2021-07-15 23:21:37 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-07-15 23:21:37 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-07-15 23:21:37 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-07-15 23:21:47 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 37.50 % 2021-07-15 23:21:47 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 37.50% 2021-07-15 23:21:57 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-07-15 23:21:57 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 20.0650789738 sec 2021-07-15 23:22:10 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.00632405281 seconds 2021-07-15 23:22:10 | INFO | MainProcess | test_thread | [basetestcase.load] delete 20160 to src_bucket documents... 2021-07-15 23:22:10 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.64:11210 src_bucket 2021-07-15 23:22:23 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2021-07-15 23:22:23 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.74 port:8091 ssh_username:root] 2021-07-15 23:22:23 | INFO | MainProcess | test_thread | [eventing_base.pause_function] Pause Application : Function_296747078_test_eventing_rebalance_in_kill_eventing_producer 2021-07-15 23:22:23 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:22:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:23:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2021-07-15 23:23:33 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.67 with username:root 2021-07-15 23:23:33 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.67 2021-07-15 23:23:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:23:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:23:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: killall -9 eventing-producer 2021-07-15 23:23:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:23:34 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 120 secs. ... 2021-07-15 23:25:34 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:25:54 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:26:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:26:34 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:26:54 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:27:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:27:34 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:27:54 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:28:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:28:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:28:55 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:29:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:29:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:29:55 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:30:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:30:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:30:55 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:31:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:31:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:31:55 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... 2021-07-15 23:32:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_296747078_test_eventing_rebalance_in_kill_eventing_producer to paused... ... ERROR 2021-07-15 23:32:35 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.106.67 port:8091 ssh_username:root, ip:172.23.106.74 port:8091 ssh_username:root] 2021-07-15 23:32:35 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.67 with username:root 2021-07-15 23:32:35 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.67 2021-07-15 23:32:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:32:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:32:36 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.67:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-07-15 23:32:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2021-07-15 23:32:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:32:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.67: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2021-07-15 23:32:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:32:36 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.74 with username:root 2021-07-15 23:32:36 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.74 2021-07-15 23:32:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-07-15 23:32:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 2021-07-15 23:32:37 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.74:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-07-15 23:32:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2021-07-15 23:32:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:32:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.74: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2021-07-15 23:32:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2021-07-15 23:32:37 | INFO | MainProcess | test_thread | [eventing_base.tearDown] Bucket dst_bucket DGM is 100 2021-07-15 23:32:37 | INFO | MainProcess | test_thread | [eventing_base.tearDown] Bucket metadata DGM is 100 2021-07-15 23:32:37 | INFO | MainProcess | test_thread | [eventing_base.tearDown] Bucket src_bucket DGM is 100 2021-07-15 23:32:37 | INFO | MainProcess | test_thread | [eventing_base.tearDown] number of documents in metadata bucket 1024 ERROR