./testrunner -i b/resources/5-nodes-template.ini -p get-cbcollect-info=True,GROUP=bucket_op,skip_log_scan=False,bucket_storage=couchstore -t eventing.eventing_rebalance_collection.EventingRebalanceCollection.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=20,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,GROUP=bucket_op,java_sdk_client=True,get-cbcollect-info=True,skip_log_scan=False,bucket_storage=couchstore INFO:root:__main__ *** TestRunner *** INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: {'GROUP': 'bucket_op', 'bucket_storage': 'couchstore', 'cluster_name': '5-nodes-template', 'get-cbcollect-info': 'True', 'ini': 'b/resources/5-nodes-template.ini', 'num_nodes': 5, 'skip_log_scan': 'False'} INFO:__main__: Number of tests initially selected before GROUP filters: 1 Only cases in GROUPs 'bucket_op' will be executed INFO:__main__:--> Running test: eventing.eventing_rebalance_collection.EventingRebalanceCollection.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=20,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,GROUP=bucket_op,java_sdk_client=True,get-cbcollect-info=True,skip_log_scan=False,bucket_storage=couchstore INFO:__main__:Logs folder: /Users/chanabasappaghali/testrunner_mad-hatter/testrunner_latest/testrunner/logs/testrunner-21-May-05_10-41-08/test_1 Logs will be stored at /Users/chanabasappaghali/testrunner_mad-hatter/testrunner_latest/testrunner/logs/testrunner-21-May-05_10-41-08/test_1 ./testrunner -i b/resources/5-nodes-template.ini -p get-cbcollect-info=True,GROUP=bucket_op,skip_log_scan=False,bucket_storage=couchstore -t eventing.eventing_rebalance_collection.EventingRebalanceCollection.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=20,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,GROUP=bucket_op,java_sdk_client=True,get-cbcollect-info=True,skip_log_scan=False,bucket_storage=couchstore Test Input params: {'doc-per-day': '20', 'dataset': 'default', 'nodes_init': '5', 'services_init': 'kv-kv-kv-eventing-index:n1ql', 'groups': 'simple', 'reset_services': 'True', 'GROUP': 'bucket_op', 'java_sdk_client': 'True', 'get-cbcollect-info': 'True', 'skip_log_scan': 'False', 'bucket_storage': 'couchstore', 'ini': 'b/resources/5-nodes-template.ini', 'cluster_name': '5-nodes-template', 'num_nodes': 5, 'case_number': 1, 'logs_folder': '/Users/chanabasappaghali/testrunner_mad-hatter/testrunner_latest/testrunner/logs/testrunner-21-May-05_10-41-08/test_1'} Run before suite setup for eventing.eventing_rebalance_collection.EventingRebalanceCollection.test_eventing_rebalance_with_multiple_kv_nodes -->before_suite_name:eventing.eventing_rebalance_collection.EventingRebalanceCollection.suite_setUp,suite: ]> suite_setUp (eventing.eventing_rebalance_collection.EventingRebalanceCollection) ... 2021-05-05 10:41:08 | INFO | MainProcess | MainThread | [eventing_base.setUp] ============== EventingBaseTest setup has started ============== 2021-05-05 10:41:08 | INFO | MainProcess | MainThread | [eventing_base.setUp] Starting Test: suite_setUp 2021-05-05 10:41:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:41:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:41:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:41:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:41:08 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:41:08 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-5092-enterprise 2021-05-05 10:41:09 | INFO | MainProcess | MainThread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.0-5092-enterprise', '7.0.0-5092-enterprise', '7.0.0-5092-enterprise', '7.0.0-5092-enterprise', '7.0.0-5092-enterprise'] 2021-05-05 10:41:09 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2021-05-05 10:41:09 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 10.112.206.101 2021-05-05 10:41:10 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-05-05 10:41:10 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.206.101 existing buckets : ['metadata', 'src_bucket'] 2021-05-05 10:41:10 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : dst_bucket from 10.112.206.101 2021-05-05 10:41:10 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-05-05 10:41:10 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.206.101 existing buckets : ['src_bucket'] 2021-05-05 10:41:10 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : metadata from 10.112.206.101 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.206.101 existing buckets : [] 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : src_bucket from 10.112.206.101 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.102, nothing to delete 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.103, nothing to delete 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.104, nothing to delete 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.105, nothing to delete 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.206.101,ns_1@10.112.206.102,ns_1@10.112.206.103,ns_1@10.112.206.104,ns_1@10.112.206.105', 'ejectedNodes': 'ns_1@10.112.206.102,ns_1@10.112.206.103,ns_1@10.112.206.104,ns_1@10.112.206.105', 'user': 'Administrator', 'password': 'password'} 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2021-05-05 10:41:11 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-05-05 10:41:21 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 70.00 % 2021-05-05 10:41:31 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 90.00 % 2021-05-05 10:41:51 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 40.01 seconds 2021-05-05 10:41:51 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:10.112.206.101 port:8091 ssh_username:root ? [('ns_1@10.112.206.102', 8091), ('ns_1@10.112.206.103', 8091), ('ns_1@10.112.206.104', 8091), ('ns_1@10.112.206.105', 8091)] 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.101:8091 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.101:8091 is running 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.102:8091 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.102:8091 is running 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.103:8091 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.103:8091 is running 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.104:8091 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.104:8091 is running 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.105:8091 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.105:8091 is running 2021-05-05 10:42:01 | INFO | MainProcess | MainThread | [basetestcase.setUp] Building docker image with java sdk client POM file java_sdk_client/collections/pom.xml specified with the -f/--file command line argument does not exist 2021-05-05 10:42:02 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2021-05-05 10:42:02 | INFO | MainProcess | MainThread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has started ============== 2021-05-05 10:42:02 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2021-05-05 10:42:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-05-05 10:42:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.206.101:8091 => {'services': ['kv'], 'cpu_utilization': 0.9164969450101833, 'mem_free': 1402003456, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:42:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-05-05 10:42:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:42:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:42:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:42:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:42:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:42:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:42:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:42:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:42:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:42:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:42:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.101, nothing to delete 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.102, nothing to delete 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.103, nothing to delete 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.104, nothing to delete 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.105, nothing to delete 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.101:8091 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.101:8091 is running 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.102:8091 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.102:8091 is running 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.103:8091 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.103:8091 is running 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.104:8091 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.104:8091 is running 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.105:8091 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.105:8091 is running 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:42:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:09 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:42:09 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:42:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl stop couchbase-server.service 2021-05-05 10:42:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:10 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:42:10 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:42:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:42:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:42:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:42:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:42:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:16 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:42:16 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:42:16 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:42:16 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:42:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl start couchbase-server.service 2021-05-05 10:42:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:16 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.101:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:42:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:42:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:21 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 20964 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:42:21 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:42:21 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.101: with pid 21045 2021-05-05 10:42:21 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:42:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:42:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:42:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:42:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:42:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:22 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:42:22 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:42:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl stop couchbase-server.service 2021-05-05 10:42:24 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:24 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:42:24 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl start couchbase-server.service 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:29 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.102:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:42:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:42:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:34 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 21113 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:42:34 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:42:34 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.102: with pid 21194 2021-05-05 10:42:34 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:42:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:42:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:42:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:42:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:42:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:35 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:42:35 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:42:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl stop couchbase-server.service 2021-05-05 10:42:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:37 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:42:37 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:42:42 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:42:42 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:42 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:42:42 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:42:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:42:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:42 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:42:42 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:42:42 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:42:43 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:42:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl start couchbase-server.service 2021-05-05 10:42:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:43 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.103:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 21049 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.103: with pid 21130 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:42:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:49 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:42:49 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:42:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl stop couchbase-server.service 2021-05-05 10:42:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:50 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:42:50 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:42:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:42:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:42:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:42:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:42:55 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:42:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:42:56 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:42:56 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:42:56 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:42:56 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:42:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl start couchbase-server.service 2021-05-05 10:42:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:42:56 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.104:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 20810 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.104: with pid 20891 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:43:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:02 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:43:02 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:43:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl stop couchbase-server.service 2021-05-05 10:43:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:03 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:43:03 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:43:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:43:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:43:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:43:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:43:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:09 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:43:09 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:43:09 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:43:09 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:43:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl start couchbase-server.service 2021-05-05 10:43:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:09 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.105:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:43:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:43:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:14 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 20490 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:43:14 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:43:14 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.105: with pid 20571 2021-05-05 10:43:14 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:43:14 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... Cluster instance shutdown with force 2021-05-05 10:43:24 | INFO | MainProcess | MainThread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has completed ============== 2021-05-05 10:43:24 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2021-05-05 10:43:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:43:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:43:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:43:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:43:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:25 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:43:25 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:43:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl stop couchbase-server.service 2021-05-05 10:43:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:27 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:43:27 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl start couchbase-server.service 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:32 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.101:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:43:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:43:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:37 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 21345 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:43:37 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:43:37 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.101: with pid 21426 2021-05-05 10:43:37 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:43:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:43:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:43:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:43:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:43:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:38 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:43:38 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:43:38 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl stop couchbase-server.service 2021-05-05 10:43:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:40 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:43:40 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl start couchbase-server.service 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:45 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.102:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:43:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:43:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:50 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 21493 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:43:50 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:43:51 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.102: with pid 21574 2021-05-05 10:43:51 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:43:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:43:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:43:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:43:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:43:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:51 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:43:52 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:43:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl stop couchbase-server.service 2021-05-05 10:43:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:53 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:43:53 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:43:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:43:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:43:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:43:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:43:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:43:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:43:58 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:43:58 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:43:59 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:43:59 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:43:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl start couchbase-server.service 2021-05-05 10:43:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:43:59 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.103:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 21429 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.103: with pid 21510 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:44:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:44:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:44:05 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:44:05 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:44:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl stop couchbase-server.service 2021-05-05 10:44:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:06 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:44:06 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:44:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:44:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:44:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:44:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:44:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:44:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:44:12 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:44:12 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:44:12 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:44:12 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:44:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl start couchbase-server.service 2021-05-05 10:44:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:12 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.104:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 21190 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.104: with pid 21272 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:44:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:44:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:44:18 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:44:18 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:44:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl stop couchbase-server.service 2021-05-05 10:44:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:19 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:44:19 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:44:24 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:44:24 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:24 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:44:24 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:44:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:44:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:44:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:44:25 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:44:25 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:44:25 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:44:25 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:44:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl start couchbase-server.service 2021-05-05 10:44:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:25 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.105:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:44:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:44:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:30 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 20870 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:44:30 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:44:30 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.105: with pid 20951 2021-05-05 10:44:30 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:44:30 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... 2021-05-05 10:44:40 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.206.101 port:8091 ssh_username:root, nodes/self 2021-05-05 10:44:40 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '64', 'memoryTotal': 1930829824, 'memoryFree': 1535332352, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.206.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5092-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.206.101', 'ip': '10.112.206.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-05-05 10:44:40 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.206.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:44:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-05-05 10:44:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.206.101,8091,['kv']) 2021-05-05 10:44:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.206.101: 8091:hostname=10.112.206.101&user=Administrator&password=password&services=kv 2021-05-05 10:44:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-05-05 10:44:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.206.101:8091:port=8091&username=Administrator&password=password 2021-05-05 10:44:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-05-05 10:44:40 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.206.102 port:8091 ssh_username:root, nodes/self 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '53', 'memoryTotal': 1930829824, 'memoryFree': 1568546816, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.206.102:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5092-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.206.102', 'ip': '10.112.206.102', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-05-05 10:44:41 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.206.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.206.102:8091:port=8091&username=Administrator&password=password 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-05-05 10:44:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.206.103 port:8091 ssh_username:root, nodes/self 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '39', 'memoryTotal': 1930829824, 'memoryFree': 1566728192, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.206.103:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5092-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.206.103', 'ip': '10.112.206.103', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-05-05 10:44:42 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.206.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.206.103:8091:port=8091&username=Administrator&password=password 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-05-05 10:44:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.206.104 port:8091 ssh_username:root, nodes/self 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '28', 'memoryTotal': 1930829824, 'memoryFree': 1570058240, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.206.104:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5092-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.206.104', 'ip': '10.112.206.104', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-05-05 10:44:43 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.206.104:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.206.104:8091:port=8091&username=Administrator&password=password 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:44:43 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.206.105 port:8091 ssh_username:root, nodes/self 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '13', 'memoryTotal': 1930829824, 'memoryFree': 1563594752, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.206.105:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5092-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.206.105', 'ip': '10.112.206.105', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-05-05 10:44:44 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.206.105:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.206.105:8091:port=8091&username=Administrator&password=password 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:44:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:44:45 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:44:45 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:44:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:44:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:44:45 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:44:45 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.105:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:44:45 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.105:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:44:45 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:44:45 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.206.101 **** 2021-05-05 10:44:45 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://10.112.206.101:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-05-05 10:44:45 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-05-05 10:44:45 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:44:50 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-05-05 10:44:50 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2021-05-05 10:44:51 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.206.102:8091 to cluster 2021-05-05 10:44:51 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.206.102:8091 to this cluster @10.112.206.101:8091 2021-05-05 10:45:01 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-05-05 10:45:01 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-05-05 10:45:18 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.206.103:8091 to cluster 2021-05-05 10:45:18 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.206.103:8091 to this cluster @10.112.206.101:8091 2021-05-05 10:45:28 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-05-05 10:45:28 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-05-05 10:45:46 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.206.104:8091 to cluster 2021-05-05 10:45:46 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.206.104:8091 to this cluster @10.112.206.101:8091 2021-05-05 10:45:56 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.00 seconds 2021-05-05 10:45:56 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-05-05 10:46:13 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.206.105:8091 to cluster 2021-05-05 10:46:13 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.206.105:8091 to this cluster @10.112.206.101:8091 2021-05-05 10:46:23 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-05-05 10:46:23 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-05-05 10:46:41 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.206.101,ns_1@10.112.206.102,ns_1@10.112.206.103,ns_1@10.112.206.104,ns_1@10.112.206.105', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-05-05 10:46:41 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-05-05 10:46:51 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-05-05 10:46:51 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.00766396522522 sec 2021-05-05 10:46:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:46:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:46:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:46:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:46:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.206.101 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ============== 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:46:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:46:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:46:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:46:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:46:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:46:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:46:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:46:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:46:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:46:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:46:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:46:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:46:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-05-05 10:46:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.206.101:8091 => {'services': ['kv'], 'cpu_utilization': 5.741127348643007, 'mem_free': 1496342528, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:46:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.206.102:8091 => {'services': ['kv'], 'cpu_utilization': 7.322175732217573, 'mem_free': 1536675840, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:46:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.206.103:8091 => {'services': ['kv'], 'cpu_utilization': 2.561475409836066, 'mem_free': 1536143360, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:46:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.206.104:8091 => {'services': ['eventing'], 'cpu_utilization': 6.849315068493151, 'mem_free': 1536929792, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:46:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.206.105:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 1.323828920570265, 'mem_free': 1547792384, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:46:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-05-05 10:46:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:46:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:46:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:46:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:47:00 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:47:00 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.112.206.104 port:8091 ssh_username:root] 2021-05-05 10:47:00 | INFO | MainProcess | MainThread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:47:00 | INFO | MainProcess | MainThread | [eventing_base.setUp] Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. 2021-05-05 10:47:00 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=330 2021-05-05 10:47:00 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-05-05 10:47:00 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : eventingMemoryQuota=512 2021-05-05 10:47:00 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:47:00 | INFO | MainProcess | MainThread | [eventing_base.setUp] ============== EventingBaseTest setup has completed ============== 2021-05-05 10:47:00 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=700 2021-05-05 10:47:00 | INFO | MainProcess | MainThread | [eventing_rebalance_collection.setUp] 200 2021-05-05 10:47:00 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.206.101:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-05-05 10:47:00 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket src_bucket 2021-05-05 10:47:00 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:01 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:01 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:01 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:01 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:02 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:02 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:02 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-05-05 10:47:02 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:02 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:02 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:02 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:02 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:02 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:02 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:03 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-05-05 10:47:03 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:03 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-05-05 10:47:03 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:04 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-05-05 10:47:04 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:05 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-05-05 10:47:05 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:47:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:47:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:47:06 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-05-05 10:47:07 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.206.101:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-05-05 10:47:07 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket dst_bucket 2021-05-05 10:47:07 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:09 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-05-05 10:47:09 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:09 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:10 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-05-05 10:47:10 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:10 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-05-05 10:47:10 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:11 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-05-05 10:47:11 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:12 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-05-05 10:47:12 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:47:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:47:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:47:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:47:13 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-05-05 10:47:14 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.206.101:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-05-05 10:47:14 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket metadata 2021-05-05 10:47:14 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:47:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:16 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-05-05 10:47:16 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:47:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:17 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-05-05 10:47:17 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:47:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:17 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-05-05 10:47:17 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:47:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:18 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-05-05 10:47:18 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:47:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:19 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-05-05 10:47:19 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:47:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:47:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:47:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:47:20 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] Scope created dst_bucket->scope_0 {'uid': '1'} 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] Scope created dst_bucket->scope_1 {'uid': '2'} 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] Scope created dst_bucket->scope_2 {'uid': '3'} 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] Scope created dst_bucket->scope_3 {'uid': '4'} 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] Scope created dst_bucket->scope_4 {'uid': '5'} 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] Scope created src_bucket->scope_0 {'uid': '1'} 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] Scope created src_bucket->scope_1 {'uid': '2'} 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] Scope created src_bucket->scope_2 {'uid': '3'} 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 2021-05-05 10:47:20 | INFO | MainProcess | MainThread | [rest_client.create_scope] Scope created src_bucket->scope_3 {'uid': '4'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_scope] Scope created src_bucket->scope_4 {'uid': '5'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.112.206.104 port:8091 ssh_username:root] 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [eventing_base.print_eventing_stats_from_all_eventing_nodes] Stats for Node 10.112.206.104 is [] 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has started ============== 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.112.206.104 port:8091 ssh_username:root] 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.206.104:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-05-05 10:47:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+metadata 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [eventing_base.tearDown] number of documents in metadata bucket 0 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.206.101:8091 => {'services': ['kv'], 'cpu_utilization': 56.36147443519619, 'mem_free': 1401249792, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.206.102:8091 => {'services': ['kv'], 'cpu_utilization': 53.66146458583433, 'mem_free': 1440206848, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.206.103:8091 => {'services': ['kv'], 'cpu_utilization': 41.46919431279621, 'mem_free': 1460789248, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.206.104:8091 => {'services': ['eventing'], 'cpu_utilization': 5.433646812957158, 'mem_free': 1514573824, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.206.105:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 7.074973600844773, 'mem_free': 1434083328, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:47:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:47:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:47:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:47:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:47:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:47:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:47:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:47:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:47:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:47:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:47:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:47:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:47:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:47:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:47:28 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-05-05 10:47:28 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 10.112.206.101 2021-05-05 10:47:28 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.206.101 existing buckets : ['metadata', 'src_bucket'] 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : dst_bucket from 10.112.206.101 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.206.101 existing buckets : ['src_bucket'] 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : metadata from 10.112.206.101 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.206.101 existing buckets : [] 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : src_bucket from 10.112.206.101 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.102, nothing to delete 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.103, nothing to delete 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.104, nothing to delete 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.105, nothing to delete 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.206.101,ns_1@10.112.206.102,ns_1@10.112.206.103,ns_1@10.112.206.104,ns_1@10.112.206.105', 'ejectedNodes': 'ns_1@10.112.206.102,ns_1@10.112.206.103,ns_1@10.112.206.104,ns_1@10.112.206.105', 'user': 'Administrator', 'password': 'password'} 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2021-05-05 10:47:29 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-05-05 10:47:39 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 70.00 % 2021-05-05 10:47:49 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 70.00 % 2021-05-05 10:48:09 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 40.02 seconds 2021-05-05 10:48:09 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:10.112.206.101 port:8091 ssh_username:root ? [('ns_1@10.112.206.102', 8091), ('ns_1@10.112.206.103', 8091), ('ns_1@10.112.206.104', 8091), ('ns_1@10.112.206.105', 8091)] 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.101:8091 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.101:8091 is running 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.102:8091 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.102:8091 is running 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.103:8091 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.103:8091 is running 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.104:8091 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.104:8091 is running 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.105:8091 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.105:8091 is running 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections 2021-05-05 10:48:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:48:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:48:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:48:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:48:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:48:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:48:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:48:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:48:20 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:48:20 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:48:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl stop couchbase-server.service 2021-05-05 10:48:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:22 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:48:22 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:48:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:48:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:48:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:48:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:48:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:48:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:48:27 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:48:27 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:48:27 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:48:28 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:48:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl start couchbase-server.service 2021-05-05 10:48:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:28 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.101:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 24079 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.101: with pid 24160 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:48:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:48:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:48:34 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:48:34 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:48:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl stop couchbase-server.service 2021-05-05 10:48:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:35 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:48:35 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:48:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:48:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:48:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:48:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:48:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:48:41 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:48:41 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:48:41 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:48:41 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:48:41 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:48:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl start couchbase-server.service 2021-05-05 10:48:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:41 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.102:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 24359 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.102: with pid 24440 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:48:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:48:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:48:47 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:48:47 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:48:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl stop couchbase-server.service 2021-05-05 10:48:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:48 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:48:48 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:48:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:48:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:48:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:48:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:48:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:48:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:48:54 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:48:54 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:48:54 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:48:54 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:48:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl start couchbase-server.service 2021-05-05 10:48:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:54 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.103:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:48:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:48:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:48:59 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 24296 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:48:59 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:48:59 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.103: with pid 24377 2021-05-05 10:48:59 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:48:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:48:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:48:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:48:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:48:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:49:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:49:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:00 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:49:00 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:49:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl stop couchbase-server.service 2021-05-05 10:49:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:01 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:49:01 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:49:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:49:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl start couchbase-server.service 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:07 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.104:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:49:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:49:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:12 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 24170 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:49:12 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:49:12 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.104: with pid 24251 2021-05-05 10:49:12 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:49:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:49:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:49:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:49:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:49:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:13 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:49:13 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:49:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl stop couchbase-server.service 2021-05-05 10:49:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:15 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:49:15 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 10.112.206.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl start couchbase-server.service 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:20 | INFO | MainProcess | MainThread | [remote_util.sleep] 10.112.206.105:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:49:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:49:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:25 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 24035 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:49:25 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:49:25 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.105: with pid 24116 2021-05-05 10:49:25 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-05-05 10:49:25 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... Cluster instance shutdown with force 2021-05-05 10:49:35 | INFO | MainProcess | MainThread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has completed ============== ok ---------------------------------------------------------------------- Ran 1 test in 507.322s OK -->result: test_eventing_rebalance_with_multiple_kv_nodes (eventing.eventing_rebalance_collection.EventingRebalanceCollection) ... 2021-05-05 10:49:35 | INFO | MainProcess | test_thread | [eventing_base.setUp] ============== EventingBaseTest setup has started ============== 2021-05-05 10:49:35 | INFO | MainProcess | test_thread | [eventing_base.setUp] Starting Test: test_eventing_rebalance_with_multiple_kv_nodes 2021-05-05 10:49:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:49:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-5092-enterprise 2021-05-05 10:49:36 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://10.112.206.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [] 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_eventing_rebalance_with_multiple_kv_nodes============== 2021-05-05 10:49:36 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://10.112.206.101:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.101, nothing to delete 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.102, nothing to delete 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.103, nothing to delete 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.104, nothing to delete 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.105, nothing to delete 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.101:8091 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.101:8091 is running 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.102:8091 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.102:8091 is running 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.103:8091 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.103:8091 is running 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.104:8091 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.104:8091 is running 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.105:8091 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.105:8091 is running 2021-05-05 10:49:36 | INFO | MainProcess | test_thread | [basetestcase.setUp] Building docker image with java sdk client POM file java_sdk_client/collections/pom.xml specified with the -f/--file command line argument does not exist 2021-05-05 10:49:37 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://10.112.206.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:49:37 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2021-05-05 10:49:37 | INFO | MainProcess | test_thread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has started ============== 2021-05-05 10:49:37 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://10.112.206.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:49:37 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://10.112.206.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:49:37 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2021-05-05 10:49:37 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://10.112.206.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:49:37 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-05-05 10:49:37 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-05-05 10:49:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:49:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:49:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:49:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:49:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:49:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:49:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:49:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:49:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:49:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:49:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_eventing_rebalance_with_multiple_kv_nodes ============== 2021-05-05 10:49:43 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://10.112.206.101:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.101, nothing to delete 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.102, nothing to delete 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.103, nothing to delete 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.104, nothing to delete 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.206.105, nothing to delete 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.101:8091 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.101:8091 is running 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.102:8091 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.102:8091 is running 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.103:8091 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.103:8091 is running 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.104:8091 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.104:8091 is running 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.206.105:8091 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.206.105:8091 is running 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_eventing_rebalance_with_multiple_kv_nodes ============== 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:49:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:44 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:49:44 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:49:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl stop couchbase-server.service 2021-05-05 10:49:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:45 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:49:45 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:49:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:49:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:49:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:49:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:49:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:51 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:49:51 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:49:51 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:49:51 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:49:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl start couchbase-server.service 2021-05-05 10:49:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:51 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.206.101:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 25590 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.101: with pid 25671 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:49:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:49:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:49:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:49:57 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:49:57 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:49:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl stop couchbase-server.service 2021-05-05 10:49:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:49:58 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:49:58 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:50:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:50:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:50:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:50:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:50:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:04 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:50:04 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:50:04 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:50:04 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:50:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl start couchbase-server.service 2021-05-05 10:50:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:04 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.206.102:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:50:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:50:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:09 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 25841 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:50:09 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:50:09 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.102: with pid 25922 2021-05-05 10:50:09 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-05-05 10:50:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:50:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:50:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:50:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:50:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:10 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:50:10 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:50:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl stop couchbase-server.service 2021-05-05 10:50:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:12 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:50:12 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl start couchbase-server.service 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:17 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.206.103:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:50:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:50:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:22 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 25779 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:50:22 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:50:22 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.103: with pid 25860 2021-05-05 10:50:22 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-05-05 10:50:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:50:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:50:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:50:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:50:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:23 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:50:23 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:50:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl stop couchbase-server.service 2021-05-05 10:50:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:25 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:50:25 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl start couchbase-server.service 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:30 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.206.104:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:50:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:50:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:35 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 25701 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:50:35 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:50:35 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.104: with pid 25782 2021-05-05 10:50:35 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-05-05 10:50:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:50:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:50:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:50:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:50:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:36 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:50:36 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:50:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl stop couchbase-server.service 2021-05-05 10:50:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:38 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:50:38 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl start couchbase-server.service 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:43 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.206.105:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:50:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:50:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:50:48 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 25661 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:50:48 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:50:48 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.105: with pid 25742 2021-05-05 10:50:48 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-05-05 10:50:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... Cluster instance shutdown with force 2021-05-05 10:50:58 | INFO | MainProcess | test_thread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has completed ============== 2021-05-05 10:50:58 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2021-05-05 10:50:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:50:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:50:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:50:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:50:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:50:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:50:59 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:50:59 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:50:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl stop couchbase-server.service 2021-05-05 10:51:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:01 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:51:01 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:51:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:51:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:51:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:51:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:51:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:06 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:51:06 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:51:07 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:51:07 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:51:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl start couchbase-server.service 2021-05-05 10:51:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:07 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.206.101:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 25970 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.101: with pid 26051 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:51:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:13 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:51:13 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:51:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl stop couchbase-server.service 2021-05-05 10:51:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:14 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:51:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:51:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:51:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:51:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:51:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:51:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:20 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:51:20 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:51:20 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:51:20 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:51:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl start couchbase-server.service 2021-05-05 10:51:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:20 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.206.102:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 26223 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.102: with pid 26304 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:51:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:51:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:26 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:51:26 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:51:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl stop couchbase-server.service 2021-05-05 10:51:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:27 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:51:27 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:51:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:51:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:51:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:51:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:51:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:33 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:51:33 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:51:33 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:51:33 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:51:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl start couchbase-server.service 2021-05-05 10:51:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:33 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.206.103:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:51:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:51:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:38 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 26158 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:51:38 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:51:38 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.103: with pid 26239 2021-05-05 10:51:38 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-05-05 10:51:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:51:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:51:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:51:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:51:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:39 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:51:39 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:51:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl stop couchbase-server.service 2021-05-05 10:51:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:41 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:51:41 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl start couchbase-server.service 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:46 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.206.104:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:51:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:51:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:51 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 26081 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:51:51 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:51:51 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.104: with pid 26162 2021-05-05 10:51:51 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-05-05 10:51:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:51:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:51:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:51:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:51:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:52 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:51:52 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-05-05 10:51:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl stop couchbase-server.service 2021-05-05 10:51:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:54 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-05-05 10:51:54 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.206.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl start couchbase-server.service 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:51:59 | INFO | MainProcess | test_thread | [remote_util.sleep] 10.112.206.105:sleep for 5 secs. waiting for couchbase server to come up ... 2021-05-05 10:52:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-05-05 10:52:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:52:04 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 26041 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-05-05 10:52:04 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-05-05 10:52:04 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 10.112.206.105: with pid 26122 2021-05-05 10:52:04 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-05-05 10:52:04 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2021-05-05 10:52:15 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.206.101 port:8091 ssh_username:root, nodes/self 2021-05-05 10:52:15 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '64', 'memoryTotal': 1930829824, 'memoryFree': 1519304704, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.206.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5092-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.206.101', 'ip': '10.112.206.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-05-05 10:52:15 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.206.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:52:15 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-05-05 10:52:15 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.206.101,8091,['kv']) 2021-05-05 10:52:15 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.206.101: 8091:hostname=10.112.206.101&user=Administrator&password=password&services=kv 2021-05-05 10:52:15 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-05-05 10:52:15 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.206.101:8091:port=8091&username=Administrator&password=password 2021-05-05 10:52:15 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-05-05 10:52:15 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:52:15 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:52:15 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.206.102 port:8091 ssh_username:root, nodes/self 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '54', 'memoryTotal': 1930829824, 'memoryFree': 1569878016, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.206.102:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5092-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.206.102', 'ip': '10.112.206.102', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-05-05 10:52:16 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.206.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.206.102:8091:port=8091&username=Administrator&password=password 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:52:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.206.103 port:8091 ssh_username:root, nodes/self 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '39', 'memoryTotal': 1930829824, 'memoryFree': 1569767424, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.206.103:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5092-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.206.103', 'ip': '10.112.206.103', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-05-05 10:52:17 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.206.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.206.103:8091:port=8091&username=Administrator&password=password 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:52:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.206.104 port:8091 ssh_username:root, nodes/self 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '28', 'memoryTotal': 1930829824, 'memoryFree': 1565585408, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.206.104:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5092-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.206.104', 'ip': '10.112.206.104', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-05-05 10:52:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.206.104:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.206.104:8091:port=8091&username=Administrator&password=password 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:52:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.206.105 port:8091 ssh_username:root, nodes/self 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '13', 'memoryTotal': 1930829824, 'memoryFree': 1564119040, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.206.105:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5092-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.206.105', 'ip': '10.112.206.105', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2021-05-05 10:52:19 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.206.105:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=986 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.206.105:8091:port=8091&username=Administrator&password=password 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:52:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.105: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:52:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:52:20 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:52:20 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.105:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:52:20 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.105:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-05 10:52:20 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:52:20 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.206.101 **** 2021-05-05 10:52:20 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://10.112.206.101:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-05-05 10:52:20 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-05-05 10:52:20 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-05-05 10:52:25 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-05-05 10:52:25 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2021-05-05 10:52:26 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.206.102:8091 to cluster 2021-05-05 10:52:26 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.206.102:8091 to this cluster @10.112.206.101:8091 2021-05-05 10:52:36 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-05-05 10:52:36 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-05-05 10:52:48 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.206.103:8091 to cluster 2021-05-05 10:52:48 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.206.103:8091 to this cluster @10.112.206.101:8091 2021-05-05 10:52:58 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-05-05 10:52:58 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-05-05 10:53:16 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.206.104:8091 to cluster 2021-05-05 10:53:16 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.206.104:8091 to this cluster @10.112.206.101:8091 2021-05-05 10:53:26 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-05-05 10:53:26 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-05-05 10:53:38 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.206.105:8091 to cluster 2021-05-05 10:53:38 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.206.105:8091 to this cluster @10.112.206.101:8091 2021-05-05 10:53:48 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-05-05 10:53:48 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-05-05 10:54:06 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.206.101,ns_1@10.112.206.102,ns_1@10.112.206.103,ns_1@10.112.206.104,ns_1@10.112.206.105', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-05-05 10:54:06 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-05-05 10:54:16 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-05-05 10:54:16 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.008650064468384 sec 2021-05-05 10:54:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:54:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:54:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:54:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:54:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-05 10:54:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 10:54:16 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-05-05 10:54:16 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.206.101 2021-05-05 10:54:16 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_eventing_rebalance_with_multiple_kv_nodes ============== 2021-05-05 10:54:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:54:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:54:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:54:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:54:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 10:54:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 10:54:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:54:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:54:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 10:54:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 10:54:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:54:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:54:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 10:54:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 10:54:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:54:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:54:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 10:54:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.105 as root 2021-05-05 10:54:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:54:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:54:22 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-05-05 10:54:22 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.206.101:8091 => {'services': ['kv'], 'cpu_utilization': 6.010362694300518, 'mem_free': 1492131840, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:54:22 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.206.102:8091 => {'services': ['kv'], 'cpu_utilization': 1.844262295081967, 'mem_free': 1534828544, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:54:22 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.206.103:8091 => {'services': ['kv'], 'cpu_utilization': 5.650319829424307, 'mem_free': 1540009984, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:54:22 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.206.104:8091 => {'services': ['eventing'], 'cpu_utilization': 6.475583864118896, 'mem_free': 1536524288, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:54:22 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.206.105:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 1.218274111675127, 'mem_free': 1546358784, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 10:54:22 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-05-05 10:54:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 10:54:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 10:54:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 10:54:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 10:54:24 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:54:24 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.112.206.104 port:8091 ssh_username:root] 2021-05-05 10:54:24 | INFO | MainProcess | test_thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-05-05 10:54:24 | INFO | MainProcess | test_thread | [eventing_base.setUp] Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. 2021-05-05 10:54:24 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=330 2021-05-05 10:54:24 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-05-05 10:54:24 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : eventingMemoryQuota=512 2021-05-05 10:54:24 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:54:24 | INFO | MainProcess | test_thread | [eventing_base.setUp] ============== EventingBaseTest setup has completed ============== 2021-05-05 10:54:24 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=700 2021-05-05 10:54:24 | INFO | MainProcess | test_thread | [eventing_rebalance_collection.setUp] 200 2021-05-05 10:54:25 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.206.101:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-05-05 10:54:25 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket src_bucket 2021-05-05 10:54:25 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:27 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-05-05 10:54:27 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:27 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-05-05 10:54:27 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:28 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-05-05 10:54:28 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:29 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-05-05 10:54:29 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:30 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-05-05 10:54:30 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 src_bucket 2021-05-05 10:54:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 src_bucket 2021-05-05 10:54:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 src_bucket 2021-05-05 10:54:30 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-05-05 10:54:31 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.206.101:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-05-05 10:54:32 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket dst_bucket 2021-05-05 10:54:32 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:32 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:33 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-05-05 10:54:33 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:34 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-05-05 10:54:34 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:35 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-05-05 10:54:35 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:36 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-05-05 10:54:36 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:36 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-05-05 10:54:36 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.112.206.101 to accept set ops 2021-05-05 10:54:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 dst_bucket 2021-05-05 10:54:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 dst_bucket 2021-05-05 10:54:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 dst_bucket 2021-05-05 10:54:37 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-05-05 10:54:38 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.206.101:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-05-05 10:54:38 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket metadata 2021-05-05 10:54:38 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:54:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:40 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-05-05 10:54:40 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:54:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:41 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-05-05 10:54:41 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:54:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:42 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-05-05 10:54:42 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:54:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:43 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-05-05 10:54:43 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:54:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:44 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-05-05 10:54:44 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.112.206.101 to accept set ops 2021-05-05 10:54:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.101:11210 metadata 2021-05-05 10:54:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.102:11210 metadata 2021-05-05 10:54:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.206.103:11210 metadata 2021-05-05 10:54:44 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-05-05 10:54:44 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 2021-05-05 10:54:44 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_0 {'uid': '1'} 2021-05-05 10:54:44 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 2021-05-05 10:54:44 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_1 {'uid': '2'} 2021-05-05 10:54:44 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 2021-05-05 10:54:44 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_2 {'uid': '3'} 2021-05-05 10:54:44 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 2021-05-05 10:54:44 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_3 {'uid': '4'} 2021-05-05 10:54:44 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 2021-05-05 10:54:44 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created dst_bucket->scope_4 {'uid': '5'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_0 {'uid': '1'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_1 {'uid': '2'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_2 {'uid': '3'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_3 {'uid': '4'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_scope] Scope created src_bucket->scope_4 {'uid': '5'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client._create_single_collection] http://10.112.206.104:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 2021-05-05 10:54:45 | INFO | MainProcess | test_thread | [rest_client.create_collection] Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} 2021-05-05 10:54:46 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function bucket_op 2021-05-05 10:54:48 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function timers 2021-05-05 10:54:49 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function sbm 2021-05-05 10:54:50 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function curl 2021-05-05 10:54:52 | INFO | MainProcess | test_thread | [eventing_base.create_function_with_collection] saving function n1ql 2021-05-05 10:54:53 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2021-05-05 10:54:58 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2021-05-05 10:55:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2021-05-05 10:55:08 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for bucket_op to deployed... ... 2021-05-05 10:55:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2021-05-05 10:55:20 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2021-05-05 10:55:25 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2021-05-05 10:55:30 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for timers to deployed... ... 2021-05-05 10:55:36 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2021-05-05 10:55:41 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2021-05-05 10:55:46 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2021-05-05 10:55:51 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for sbm to deployed... ... 2021-05-05 10:55:58 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2021-05-05 10:56:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2021-05-05 10:56:08 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2021-05-05 10:56:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for curl to deployed... ... 2021-05-05 10:56:19 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2021-05-05 10:56:24 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2021-05-05 10:56:29 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2021-05-05 10:56:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for n1ql to deployed... ... 2021-05-05 10:56:40 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.206.101 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False Error: Unable to access jarfile java_sdk_client/collections/target/javaclient/javaclient.jar [, , , , , , ] Wed May 5 10:56:40 2021 2021-05-05 10:56:43 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.006394863128662 seconds 2021-05-05 10:57:10 | INFO | MainProcess | Cluster_Thread | [task.execute_for_collection] java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.206.101 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False Error: Unable to access jarfile java_sdk_client/collections/target/javaclient/javaclient.jar [, , , , , , ] Wed May 5 10:57:10 2021 2021-05-05 10:57:40 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.206.101,ns_1@10.112.206.102,ns_1@10.112.206.103,ns_1@10.112.206.104,ns_1@10.112.206.105', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-05-05 10:57:40 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-05-05 10:57:50 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 70.00 % 2021-05-05 10:57:50 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 70.00% 2021-05-05 10:58:10 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-05-05 10:58:10 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 30.064152002334595 sec 2021-05-05 10:58:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.112.206.104 port:8091 ssh_username:root] 2021-05-05 10:58:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:58:10 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY create primary index on dst_bucket.scope_1.coll_0 2021-05-05 10:58:10 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_0 2021-05-05 10:58:12 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.336045521s 2021-05-05 10:58:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:58:12 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 10:58:12 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 10:58:42 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:58:42 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 10:58:42 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:58:42 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 10:58:42 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 10:59:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:59:12 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 10:59:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:59:12 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 10:59:12 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 10:59:42 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:59:42 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 10:59:42 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 10:59:42 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 10:59:42 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:00:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:00:12 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:00:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:00:12 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:00:12 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:00:42 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:00:42 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:00:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:00:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:00:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:01:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:01:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:01:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:01:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:01:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:01:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:01:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:01:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:01:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:01:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:02:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:02:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:02:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:02:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:02:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:02:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:02:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:02:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:02:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:02:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:03:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:03:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:03:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:03:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:03:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:03:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:03:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:03:43 | INFO | MainProcess | test_thread | [eventing_base.bucket_compaction] Compacting bucket : dst_bucket 2021-05-05 11:03:43 | INFO | MainProcess | test_thread | [rest_client.compact_bucket] bucket compaction successful 2021-05-05 11:03:43 | INFO | MainProcess | test_thread | [eventing_base.bucket_compaction] Compacting bucket : metadata 2021-05-05 11:03:43 | INFO | MainProcess | test_thread | [rest_client.compact_bucket] bucket compaction successful 2021-05-05 11:03:43 | INFO | MainProcess | test_thread | [eventing_base.bucket_compaction] Compacting bucket : src_bucket 2021-05-05 11:03:43 | INFO | MainProcess | test_thread | [rest_client.compact_bucket] bucket compaction successful 2021-05-05 11:03:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:03:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:03:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:04:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:04:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:04:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:04:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:04:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:04:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:04:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:04:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:04:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:04:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:05:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:05:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:05:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:05:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:05:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:05:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:05:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:05:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:05:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:05:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:06:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:06:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:06:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:06:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:06:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:06:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:06:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:06:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:06:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:06:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:07:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:07:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:07:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:07:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:07:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:07:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:07:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:07:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:07:43 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:07:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_0 to complete bucket operations... Current : 0 Expected : 40320 ... 2021-05-05 11:08:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:08:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:08:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.206.105 port:8091 ssh_username:root] 2021-05-05 11:08:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 2021-05-05 11:08:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.112.206.104 port:8091 ssh_username:root] 2021-05-05 11:08:13 | INFO | MainProcess | test_thread | [eventing_base.print_eventing_stats_from_all_eventing_nodes] Stats for Node 10.112.206.104 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4769, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 747, "failure_stats": 747, "latency_stats": 747, "lcb_exception_stats": 747, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4765, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27773": "2021-05-05T05:38:13Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27773": "2021-05-05T05:38:13Z" }, "v8worker_events_lost": 0 }, "function_id": 391676807, "function_name": "sbm", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 27773 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4658, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 725, "failure_stats": 725, "latency_stats": 725, "lcb_exception_stats": 725, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4654, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27793": "2021-05-05T05:38:12Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27793": "2021-05-05T05:38:12Z" }, "v8worker_events_lost": 0 }, "function_id": 4220563968, "function_name": "curl", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 27793 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4547, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 703, "failure_stats": 703, "latency_stats": 703, "lcb_exception_stats": 703, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4543, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27813": "2021-05-05T05:38:12Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27813": "2021-05-05T05:38:12Z" }, "v8worker_events_lost": 0 }, "function_id": 3965066002, "function_name": "n1ql", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 27813 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4981, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 789, "failure_stats": 789, "latency_stats": 789, "lcb_exception_stats": 789, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4977, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27734": "2021-05-05T05:38:12Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27734": "2021-05-05T05:38:12Z" }, "v8worker_events_lost": 0 }, "function_id": 2585655416, "function_name": "bucket_op", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 27734 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4870, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 767, "failure_stats": 767, "latency_stats": 767, "lcb_exception_stats": 767, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4866, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27753": "2021-05-05T05:38:13Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27753": "2021-05-05T05:38:13Z" }, "v8worker_events_lost": 0 }, "function_id": 1916364906, "function_name": "timers", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 27753 } } ] 2021-05-05 11:08:18 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Stats for Node 10.112.206.104 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4658, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 725, "failure_stats": 725, "latency_stats": 725, "lcb_exception_stats": 725, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4654, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27793": "2021-05-05T05:38:12Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27793": "2021-05-05T05:38:12Z" }, "v8worker_events_lost": 0 }, "function_id": 4220563968, "function_name": "curl", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 27793 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4547, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 703, "failure_stats": 703, "latency_stats": 703, "lcb_exception_stats": 703, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4543, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27813": "2021-05-05T05:38:12Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27813": "2021-05-05T05:38:12Z" }, "v8worker_events_lost": 0 }, "function_id": 3965066002, "function_name": "n1ql", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 27813 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4981, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 789, "failure_stats": 789, "latency_stats": 789, "lcb_exception_stats": 789, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4977, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27734": "2021-05-05T05:38:12Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27734": "2021-05-05T05:38:12Z" }, "v8worker_events_lost": 0 }, "function_id": 2585655416, "function_name": "bucket_op", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 27734 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4870, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 767, "failure_stats": 767, "latency_stats": 767, "lcb_exception_stats": 767, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4866, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27753": "2021-05-05T05:38:13Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27753": "2021-05-05T05:38:13Z" }, "v8worker_events_lost": 0 }, "function_id": 1916364906, "function_name": "timers", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 27753 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4769, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 747, "failure_stats": 747, "latency_stats": 747, "lcb_exception_stats": 747, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4765, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27773": "2021-05-05T05:38:13Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27773": "2021-05-05T05:38:13Z" }, "v8worker_events_lost": 0 }, "function_id": 391676807, "function_name": "sbm", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 27773 } } ] 2021-05-05 11:08:19 | INFO | MainProcess | test_thread | [eventing_base.verify_doc_count_collections] Execution stats update_success: 0 update_failure: 0 delete_success: 0 delete_failure: 0 2021-05-05 11:08:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.112.206.104 port:8091 ssh_username:root] 2021-05-05 11:08:19 | INFO | MainProcess | test_thread | [eventing_base.print_eventing_stats_from_all_eventing_nodes] Stats for Node 10.112.206.104 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4900, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 773, "failure_stats": 773, "latency_stats": 773, "lcb_exception_stats": 773, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4896, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27753": "2021-05-05T05:38:19Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27753": "2021-05-05T05:38:19Z" }, "v8worker_events_lost": 0 }, "function_id": 1916364906, "function_name": "timers", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 27753 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4794, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 752, "failure_stats": 752, "latency_stats": 752, "lcb_exception_stats": 752, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4790, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27773": "2021-05-05T05:38:18Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27773": "2021-05-05T05:38:18Z" }, "v8worker_events_lost": 0 }, "function_id": 391676807, "function_name": "sbm", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 27773 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4688, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 731, "failure_stats": 731, "latency_stats": 731, "lcb_exception_stats": 731, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4684, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27793": "2021-05-05T05:38:18Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27793": "2021-05-05T05:38:18Z" }, "v8worker_events_lost": 0 }, "function_id": 4220563968, "function_name": "curl", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 27793 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4577, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 709, "failure_stats": 709, "latency_stats": 709, "lcb_exception_stats": 709, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4573, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27813": "2021-05-05T05:38:18Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27813": "2021-05-05T05:38:18Z" }, "v8worker_events_lost": 0 }, "function_id": 3965066002, "function_name": "n1ql", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 27813 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 5011, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 10240, "execution_stats": 795, "failure_stats": 795, "latency_stats": 795, "lcb_exception_stats": 795, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 5007, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "27734": "2021-05-05T05:38:18Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27734": "2021-05-05T05:38:18Z" }, "v8worker_events_lost": 0 }, "function_id": 2585655416, "function_name": "bucket_op", "gocb_creds_request_counter": 72, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 32, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.112.206.104:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.112.206.104:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 27734 } } ] 2021-05-05 11:08:19 | INFO | MainProcess | test_thread | [eventing_base.tearDown] ============== EventingBaseTest tearDown has started ============== 2021-05-05 11:08:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.112.206.104 port:8091 ssh_username:root] 2021-05-05 11:08:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 11:08:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root 2021-05-05 11:08:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 11:08:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 11:08:19 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.206.104:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-05-05 11:08:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.206.104: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.206.101:8091 => {'services': ['kv'], 'cpu_utilization': 74.58379578246392, 'mem_free': 1311289344, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.206.102:8091 => {'services': ['kv'], 'cpu_utilization': 80.6896551724138, 'mem_free': 1366847488, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.206.103:8091 => {'services': ['kv'], 'cpu_utilization': 48.20627802690583, 'mem_free': 1359515648, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.206.104:8091 => {'services': ['eventing'], 'cpu_utilization': 58.81032547699214, 'mem_free': 1340071936, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.206.105:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 54.3046357615894, 'mem_free': 1360023552, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.101 as root 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 11:08:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 11:08:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 11:08:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 11:08:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.103 as root 2021-05-05 11:08:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-05 11:08:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-05 11:08:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 05:38:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.104 as root ERROR ERROR ====================================================================== ERROR: test_eventing_rebalance_with_multiple_kv_nodes (eventing.eventing_rebalance_collection.EventingRebalanceCollection) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/eventing/eventing_rebalance_collection.py", line 349, in test_eventing_rebalance_with_multiple_kv_nodes self.verify_all_handler(self.docs_per_day * self.num_docs) File "pytests/eventing/eventing_rebalance_collection.py", line 79, in verify_all_handler self.verify_doc_count_collections("dst_bucket.scope_1.coll_0", number_of_docs) File "./pytests/eventing/eventing_base.py", line 1045, in verify_doc_count_collections raise Exception("missing data in destination bucket. Current : {0} " Exception: missing data in destination bucket. Current : 0 Expected : 40320 dcp_backlog : 0 TIMERS_IN_PAST : 0 lcb_exceptions : {'10.112.206.104': {}} ====================================================================== ERROR: test_eventing_rebalance_with_multiple_kv_nodes (eventing.eventing_rebalance_collection.EventingRebalanceCollection) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/eventing/eventing_rebalance_collection.py", line 57, in tearDown super(EventingRebalanceCollection, self).tearDown() File "./pytests/eventing/eventing_base.py", line 147, in tearDown super(EventingBaseTest, self).tearDown() File "./pytests/query_tests_helper.py", line 62, in tearDown super(QueryHelperTests, self).tearDown() File "pytests/basetestcase.py", line 527, in tearDown _tasks = self.cluster.async_log_scan(self.servers, self.log_scan_file_prefix+"_AFTER") File "lib/couchbase_helper/cluster.py", line 1411, in async_log_scan File "lib/tasks/task.py", line 6493, in __init__ self.log_scanner = LogScanner(server=self.server, exclude_keywords=exclude_keywords, File "./lib/log_scanner.py", line 56, in __init__ File "lib/remote/remote_util.py", line 267, in __init__ File "lib/remote/remote_util.py", line 3552, in extract_remote_info File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/paramiko/sftp_client.py", line 801, in get OSError: [Errno 24] Too many open files: 'etc-os-release-6cee45c4-83e9-4b5c-8526-2043271bfbd7' ---------------------------------------------------------------------- Ran 1 test in 1125.963s FAILED (errors=2) 2021-05-05 05:38:21 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#1 of 5 2021-05-05 05:38:21 | ERROR | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.101 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:21 | INFO | MainProcess | Thread-371 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#1 of 5 2021-05-05 05:38:21 | ERROR | MainProcess | Thread-371 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.102 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:21 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#1 of 5 2021-05-05 05:38:21 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 10 secs. 2021-05-05 05:38:21 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#1 of 5 2021-05-05 05:38:21 | ERROR | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.103 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:21 | INFO | MainProcess | Thread-371 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 10 secs. 2021-05-05 05:38:21 | ERROR | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.104 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:21 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 10 secs. 2021-05-05 05:38:21 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 10 secs. 2021-05-05 05:38:21 | INFO | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#1 of 5 2021-05-05 05:38:21 | ERROR | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.105 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:21 | INFO | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 10 secs. 2021-05-05 11:08:31 | INFO | MainProcess | Thread-371 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.102 with username:root, attempt#2 of 5 2021-05-05 11:08:31 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#2 of 5 2021-05-05 11:08:31 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#2 of 5 2021-05-05 11:08:31 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#2 of 5 2021-05-05 11:08:31 | INFO | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#2 of 5 2021-05-05 05:38:31 | ERROR | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.104 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:31 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 20 secs. 2021-05-05 05:38:31 | ERROR | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.105 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:31 | INFO | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 20 secs. 2021-05-05 05:38:31 | ERROR | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.101 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:31 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 20 secs. 2021-05-05 05:38:31 | ERROR | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.103 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:31 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 20 secs. 2021-05-05 05:38:31 | INFO | MainProcess | Thread-371 | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.206.102 as root 2021-05-05 05:38:51 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#3 of 5 2021-05-05 05:38:51 | ERROR | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.104 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:51 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 40 secs. 2021-05-05 05:38:51 | INFO | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#3 of 5 2021-05-05 05:38:51 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#3 of 5 2021-05-05 05:38:51 | ERROR | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.105 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:51 | ERROR | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.101 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:51 | INFO | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 40 secs. 2021-05-05 05:38:51 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 40 secs. 2021-05-05 05:38:51 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#3 of 5 2021-05-05 05:38:51 | ERROR | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.103 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:38:51 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 40 secs. 2021-05-05 05:39:31 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#4 of 5 2021-05-05 05:39:31 | ERROR | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.104 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:39:31 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 80 secs. 2021-05-05 05:39:31 | INFO | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#4 of 5 2021-05-05 05:39:31 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.101 with username:root, attempt#4 of 5 2021-05-05 05:39:31 | ERROR | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.105 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:39:31 | ERROR | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.101 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:39:31 | INFO | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 80 secs. 2021-05-05 05:39:31 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 80 secs. 2021-05-05 05:39:31 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.103 with username:root, attempt#4 of 5 2021-05-05 05:39:31 | ERROR | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.103 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:39:31 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 80 secs. 2021-05-05 11:10:51 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.104 with username:root, attempt#5 of 5 2021-05-05 11:10:51 | INFO | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.206.105 with username:root, attempt#5 of 5 2021-05-05 05:40:51 | ERROR | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 10.112.206.105 : [Errno 24] Too many open files with credentials root:couchbase 2021-05-05 05:40:51 | ERROR | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] -->No SSH connectivity to 10.112.206.105 even after 5 times! 2021-05-05 05:40:51 | ERROR | MainProcess | Thread-374 | [remote_util.ssh_connect_with_retries] Exit on failure: killing process /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d