INFO:root:__main__ INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: clitest.collectinfotest.CollectinfoTests.collectinfo_test,node_down=True,GROUP=P0,get-cbcollect-info=False,get-logs=False,get-coredumps=False INFO:__main__:Logs folder: /Users/joemitchelljones/tools/testrunner/logs/testrunner-21-Apr-23_13-30-36/test_1 *** TestRunner *** {'cluster_name': '1tool', 'get-cbcollect-info': 'False', 'get-coredumps': 'False', 'get-logs': 'False', 'ini': '1tool.ini', 'num_nodes': 1} Logs will be stored at /Users/joemitchelljones/tools/testrunner/logs/testrunner-21-Apr-23_13-30-36/test_1 ./testrunner -i 1tool.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t clitest.collectinfotest.CollectinfoTests.collectinfo_test,node_down=True,GROUP=P0,get-cbcollect-info=False,get-logs=False,get-coredumps=False Test Input params: {'node_down': 'True', 'GROUP': 'P0', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'ini': '1tool.ini', 'cluster_name': '1tool', 'num_nodes': 1, 'case_number': 1, 'logs_folder': '/Users/joemitchelljones/tools/testrunner/logs/testrunner-21-Apr-23_13-30-36/test_1'} Run before suite setup for clitest.collectinfotest.CollectinfoTests.collectinfo_test suite_setUp (unittest.loader._FailedTest) ... ERROR ====================================================================== ERROR: suite_setUp (unittest.loader._FailedTest) ---------------------------------------------------------------------- AttributeError: type object 'CollectinfoTests' has no attribute 'suite_setUp' ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (errors=1) collectinfo_test (clitest.collectinfotest.CollectinfoTests) We use cbcollect_info to automatically collect the logs for server node ... -->before_suite_name:clitest.collectinfotest.CollectinfoTests.suite_setUp,suite: ]> -->result: 2021-04-23 13:30:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.211.104 with username:root, attempt#1 of 5 2021-04-23 13:30:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.211.104 as root 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-4978-enterprise 2021-04-23 13:30:37 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://10.112.211.104:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [] 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 collectinfo_test============== 2021-04-23 13:30:37 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://10.112.211.104:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.211.104, nothing to delete 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.211.104:8091 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.211.104:8091 is running 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.211.104 with username:root, attempt#1 of 5 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.211.104 as root 2021-04-23 13:30:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-04-23 13:30:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-04-23 13:30:38 | ERROR | MainProcess | test_thread | [cli_base.tearDown] === need 2 cluster to setup xdcr in ini file === 2021-04-23 13:30:38 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://10.112.211.104:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-04-23 13:30:38 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-04-23 13:30:38 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-04-23 13:30:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.211.104 with username:root, attempt#1 of 5 2021-04-23 13:30:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.211.104 as root 2021-04-23 13:30:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-04-23 13:30:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-04-23 13:30:40 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 collectinfo_test ============== 2021-04-23 13:30:40 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://10.112.211.104:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-04-23 13:30:40 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.211.104, nothing to delete 2021-04-23 13:30:40 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-23 13:30:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.211.104:8091 2021-04-23 13:30:40 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-23 13:30:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.211.104:8091 is running 2021-04-23 13:30:40 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 collectinfo_test ============== 2021-04-23 13:30:40 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2021-04-23 13:30:40 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-04-23 13:30:40 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.211.104 port:8091 ssh_username:root, nodes/self 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '433', 'memoryTotal': 3876446208, 'memoryFree': 3408265216, 'mcdMemoryReserved': 2957, 'mcdMemoryAllocated': 2957, 'status': 'healthy', 'hostname': '10.112.211.104:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4978-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1552, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.211.104', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3696, 'curr_items': 0} 2021-04-23 13:30:41 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.211.104:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 10.112.211.104 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1725 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.211.104,8091,['kv', 'index', 'n1ql']) 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.211.104: 8091:hostname=10.112.211.104&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.211.104:8091:port=8091&username=Administrator&password=password 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.211.104 with username:root, attempt#1 of 5 2021-04-23 13:30:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.211.104 as root 2021-04-23 13:30:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-04-23 13:30:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-04-23 13:30:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.211.104: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-04-23 13:30:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-04-23 13:30:42 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-04-23 13:30:42 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.211.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-23 13:30:42 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.211.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-23 13:30:42 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-04-23 13:30:42 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.211.104 **** 2021-04-23 13:30:43 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://10.112.211.104:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-04-23 13:30:43 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-04-23 13:30:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-04-23 13:30:48 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-04-23 13:30:48 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2021-04-23 13:30:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.211.104 with username:root, attempt#1 of 5 2021-04-23 13:30:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.211.104 as root 2021-04-23 13:30:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-04-23 13:30:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-04-23 13:30:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.211.104: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-04-23 13:30:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-04-23 13:30:49 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-04-23 13:30:49 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.211.104 2021-04-23 13:30:50 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.112.211.104:8091/pools/default/buckets with param: name=default&ramQuotaMB=1725&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-04-23 13:30:50 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket default 2021-04-23 13:30:50 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 10.112.211.104 to accept set ops 2021-04-23 13:30:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:51 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-04-23 13:30:51 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 10.112.211.104 to accept set ops 2021-04-23 13:30:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:51 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-04-23 13:30:51 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 10.112.211.104 to accept set ops 2021-04-23 13:30:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:52 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-04-23 13:30:52 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 10.112.211.104 to accept set ops 2021-04-23 13:30:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:52 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-04-23 13:30:52 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 10.112.211.104 to accept set ops 2021-04-23 13:30:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:53 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-04-23 13:30:53 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 10.112.211.104 to accept set ops 2021-04-23 13:30:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:30:53 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-04-23 13:30:53 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 collectinfo_test ============== 2021-04-23 13:30:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.211.104 with username:root, attempt#1 of 5 2021-04-23 13:30:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.211.104 as root 2021-04-23 13:30:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-04-23 13:30:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-04-23 13:30:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-04-23 13:30:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.211.104:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.41988950276243, 'mem_free': 3308806144, 'mem_total': 3876446208, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-04-23 13:30:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-04-23 13:30:56 | ERROR | MainProcess | test_thread | [cli_base.setUp] === need 2 cluster to setup xdcr in ini file === 2021-04-23 13:30:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.211.104 with username:root, attempt#1 of 5 2021-04-23 13:30:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.211.104 as root 2021-04-23 13:30:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-04-23 13:30:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-04-23 13:30:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.211.104 with username:root, attempt#1 of 5 2021-04-23 13:30:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.211.104 as root 2021-04-23 13:30:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-04-23 13:30:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-04-23 13:30:57 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-4978-enterprise % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32 0 0 100 32 0 155 --:--:-- --:--:-- --:--:-- 154 100 52 100 20 100 32 57 92 --:--:-- --:--:-- --:--:-- 150 2021-04-23 13:30:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.211.104 with username:root, attempt#1 of 5 2021-04-23 13:30:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.211.104 as root 2021-04-23 13:30:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-04-23 13:30:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-04-23 13:30:58 | INFO | MainProcess | test_thread | [cli_base.setUp] Test command with absolute path 2021-04-23 13:30:58 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'default' user to node 10.112.211.104 **** 2021-04-23 13:30:58 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://10.112.211.104:8091/settings/rbac/users/local/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-04-23 13:30:58 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-04-23 13:30:58 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-04-23 13:31:03 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'default' user **** 2021-04-23 13:31:03 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.0-4978-enterprise'] 2021-04-23 13:31:03 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.211.104 port:8091 ssh_username:root] 2021-04-23 13:31:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.112.211.104:11210 default 2021-04-23 13:31:04 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '10.112.211.104:8091',default bucket 2021-04-23 13:31:04 | INFO | MainProcess | test_thread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=60&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&failoverServerGroup=false 2021-04-23 13:31:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.211.104: killall -9 memcached & killall -9 beam.smp 2021-04-23 13:31:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-04-23 13:31:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.211.104: /opt/couchbase/bin/cbcollect_info info.zip 2021-04-23 13:32:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-04-23 13:32:25 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.211.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-04-23 13:32:25 | INFO | MainProcess | test_thread | [remote_util.start_server] Running systemd start command on this server 2021-04-23 13:32:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.211.104: systemctl start couchbase-server.service 2021-04-23 13:32:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-04-23 13:34:25 | ERROR | MainProcess | test_thread | [rest_client._http_request] socket error while connecting to http://10.112.211.104:8091/nodes/self error timed out 2021-04-23 13:34:25 | ERROR | MainProcess | test_thread | [rest_client._http_request] Giving up due to timed out! Tried http://10.112.211.104:8091/nodes/self connect 1 times. 2021-04-23 13:34:25 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.112.211.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-04-23 13:34:25 | INFO | MainProcess | test_thread | [remote_util.start_server] Running systemd start command on this server 2021-04-23 13:34:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.211.104: systemctl start couchbase-server.service 2021-04-23 13:34:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-04-23 13:34:25 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-23 13:34:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.211.104 with username:root, attempt#1 of 5 2021-04-23 13:34:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.211.104 as root 2021-04-23 13:34:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-04-23 13:34:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-04-23 13:34:26 | ERROR | MainProcess | test_thread | [cli_base.tearDown] === need 2 cluster to setup xdcr in ini file === 2021-04-23 13:34:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-04-23 13:34:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.211.104:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 2.593360995850623, 'mem_free': 3221307392, 'mem_total': 3876446208, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-04-23 13:34:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-04-23 13:34:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.211.104 with username:root, attempt#1 of 5 2021-04-23 13:34:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.211.104 as root 2021-04-23 13:34:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-04-23 13:34:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-04-23 13:34:28 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 collectinfo_test ============== 2021-04-23 13:34:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 10.112.211.104 2021-04-23 13:34:29 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-04-23 13:34:29 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 10.112.211.104 existing buckets : [] 2021-04-23 13:34:29 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 10.112.211.104 2021-04-23 13:34:29 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-23 13:34:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.211.104:8091 2021-04-23 13:34:29 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-23 13:34:29 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.211.104:8091 is running 2021-04-23 13:34:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 collectinfo_test ============== 2021-04-23 13:34:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2021-04-23 13:34:29 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections downloading 10.112.211.104 ............................................ERROR ====================================================================== ERROR: collectinfo_test (clitest.collectinfotest.CollectinfoTests) We use cbcollect_info to automatically collect the logs for server node ---------------------------------------------------------------------- Traceback (most recent call last): File "lib/membase/api/rest_client.py", line 993, in _http_request params, headers) File "lib/membase/api/httplib2/__init__.py", line 1953, in request cachekey, File "lib/membase/api/httplib2/__init__.py", line 1618, in _request conn, request_uri, method, body, headers File "lib/membase/api/httplib2/__init__.py", line 1524, in _conn_request conn.connect() File "lib/membase/api/httplib2/__init__.py", line 1220, in connect raise socket_err File "lib/membase/api/httplib2/__init__.py", line 1194, in connect self.sock.connect((self.host, self.port) + sa[2:]) socket.timeout: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "pytests/clitest/collectinfotest.py", line 98, in collectinfo_test rest = RestConnection(self.master) File "lib/membase/api/rest_client.py", line 339, in __init__ http_res, success = self.init_http_request(api=self.baseUrl + "nodes/self") File "lib/membase/api/rest_client.py", line 445, in init_http_request status, content, header = self._http_request(api, 'GET', headers=headers, timeout=timeout) File "lib/membase/api/rest_client.py", line 1027, in _http_request raise ServerUnavailableException(ip=self.ip) membase.api.exception.ServerUnavailableException: unable to reach the host @ 10.112.211.104 ---------------------------------------------------------------------- Ran 1 test in 232.427s FAILED (errors=1) suite_tearDown (unittest.loader._FailedTest) ... *** TestRunner ***