2021-03-05 14:16:52,119 | test | INFO | MainProcess | MainThread | [basetestcase:_BaseTestCase__log_setup_status:483] ========= basetestcase setup started for test #1 test_failover_stop_server ========= 2021-03-05 14:16:53,213 | test | INFO | MainProcess | MainThread | [basetestcase:_BaseTestCase__log_setup_status:483] ========= basetestcase setup started for test #1 test_failover_stop_server ========= 2021-03-05 14:17:21,769 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1509] Rebalance done. Taken 25.1470000744 seconds to complete 2021-03-05 14:17:21,769 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1511] Sleep for 5 seconds after rebalance 2021-03-05 14:17:46,914 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:277] Initializing cluster 2021-03-05 14:17:47,230 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:514] RAM quota was defined less than 100 MB: 2021-03-05 14:17:47,959 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:299] Couchbase Cluster: default, Nodes: 10.112.190.101, 10.112.190.103, 10.112.190.104, 10.112.190.105 initialized 2021-03-05 14:17:47,980 | test | INFO | MainProcess | MainThread | [basetestcase:_BaseTestCase__log_setup_status:483] ========= basetestcase setup finished for test #1 test_failover_stop_server ========= 2021-03-05 14:17:48,299 | test | INFO | MainProcess | MainThread | [basetestcase:sleep:585] Reason: None. Sleep for 5 secs ... 2021-03-05 14:17:58,236 | test | INFO | MainProcess | pool-1-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+--------------+ | Nodes | Services | Status | +----------------+----------+--------------+ | 10.112.190.101 | kv | Cluster node | | 10.112.190.103 | None | <--- IN --- | | 10.112.190.104 | None | <--- IN --- | | 10.112.190.105 | None | <--- IN --- | +----------------+----------+--------------+ 2021-03-05 14:17:58,262 | test | INFO | MainProcess | pool-1-thread-5 | [task:check:304] Rebalance - status: running, progress: 0 2021-03-05 14:18:13,335 | test | INFO | MainProcess | pool-1-thread-5 | [task:check:304] Rebalance - status: none, progress: 100 2021-03-05 14:18:13,354 | test | INFO | MainProcess | pool-1-thread-5 | [task:check:363] Rebalance completed with progress: 100% in 15.1190001965 sec 2021-03-05 14:18:17,559 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+-----------+----------------------+-------------------+-----------------------+ | 10.112.190.101 | kv | 68.75 | 1042685952 | 606765056 | 7200768 / 1107292160 | 0 / 0 | 6.6.2-9559-enterprise | | 10.112.190.105 | kv | 59.375 | 1042685952 | 680161280 | 1523712 / 1107292160 | 0 / 0 | 6.6.2-9559-enterprise | | 10.112.190.104 | kv | 0 | 1042685952 | 740315136 | 1171456 / 1107292160 | 0 / 0 | 6.6.2-9559-enterprise | | 10.112.190.103 | kv | 61.8556701031 | 1042685952 | 679153664 | 2191360 / 1107292160 | 0 / 0 | 6.6.2-9559-enterprise | +----------------+----------+-----------------+------------+-----------+----------------------+-------------------+-----------------------+ 2021-03-05 14:18:17,621 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+---------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+---------+----------+------------+-----+-------+------------+-----------+-----------+ | default | membase | 3 | none | 0 | 0 | 2218786816 | 101168320 | 136 | +---------+---------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-03-05 14:18:17,674 | test | INFO | MainProcess | MainThread | [failoverbasetest:setUp:110] == FailoverBaseTest setup finished for test #1 test_failover_stop_server == 2021-03-05 14:18:17,677 | test | INFO | MainProcess | MainThread | [failovertests:common_test_body:70] Picking node 10.112.190.101 as reference node for test case 2021-03-05 14:18:17,680 | test | INFO | MainProcess | MainThread | [failovertests:print_test_params:510] num_replicas : 3 2021-03-05 14:18:17,680 | test | INFO | MainProcess | MainThread | [failovertests:print_test_params:511] recoveryType : ['delta'] 2021-03-05 14:18:17,681 | test | INFO | MainProcess | MainThread | [failovertests:print_test_params:512] failover_reason : stop_server 2021-03-05 14:18:17,681 | test | INFO | MainProcess | MainThread | [failovertests:print_test_params:513] num_failed_nodes : 3 2021-03-05 14:18:17,683 | test | INFO | MainProcess | MainThread | [failovertests:print_test_params:514] picking server : ip:10.112.190.101 port:8091 ssh_username:root as the master 2021-03-05 14:18:59,535 | test | INFO | MainProcess | pool-1-thread-8 | [table_view:display:72] Ops Trend for bucket 'default' +----------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................*............................................*.............*.....X | 3003.992 | | 2805.611 | ...........................................................................................*...X | 2899.202 | | 2575.150 | ...................................................................................*................X | 3041.916 | | 2897.796 | ..............................................................................................X | 2897.796 | | 2760.000 | .........................................................................................X | 2760.000 | | 2690.355 | .......................................................................................*.....*...X | 2932.068 | | 2655.311 | ......................................................................................X | 2655.311 | | 2235.529 | ........................................................................*..........X | 2527.528 | | 2260.000 | .........................................................................*...............*...X | 2804.519 | | 2641.988 | .....................................................................................*......X | 2815.631 | | 2352.352 | ............................................................................*..*....*..X | 2587.588 | | 2380.000 | .............................................................................*.........X | 2672.328 | | 2185.629 | ......................................................................*...........X | 2522.433 | | 2384.538 | .............................................................................X | 2384.538 | | 2370.000 | ............................................................................X | 2370.000 | | 1841.842 | ...........................................................X | 1841.842 | | 1638.362 | ....................................................X | 1638.362 | +----------+----------------------------------------------------------------------------------------------------------+----------+ 2021-03-05 14:19:21,460 | test | INFO | MainProcess | MainThread | [failovertests:run_failover_in_parallel_with_operations:527] 10 secs delay for membase-server to shutdown 2021-03-05 14:19:23,694 | test | INFO | MainProcess | MainThread | [failovertests:run_failover_in_parallel_with_operations:527] 10 secs delay for membase-server to shutdown 2021-03-05 14:19:26,339 | test | INFO | MainProcess | MainThread | [failovertests:run_failover_in_parallel_with_operations:527] 10 secs delay for membase-server to shutdown 2021-03-05 14:19:37,905 | test | INFO | MainProcess | MainThread | [rest_client:rebalance_reached:105] Rebalance reached >100% in 11.0629999638 seconds 2021-03-05 14:19:37,943 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1748] 10.112.190.103 - Node not part of cluster inactiveFailed 2021-03-05 14:19:37,944 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1748] 10.112.190.104 - Node not part of cluster inactiveFailed 2021-03-05 14:19:37,944 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1748] 10.112.190.105 - Node not part of cluster inactiveFailed 2021-03-05 14:19:37,973 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1748] 10.112.190.103 - Node not part of cluster inactiveFailed 2021-03-05 14:19:37,973 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1748] 10.112.190.104 - Node not part of cluster inactiveFailed 2021-03-05 14:19:37,974 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1748] 10.112.190.105 - Node not part of cluster inactiveFailed 2021-03-05 14:19:38,788 | test | INFO | MainProcess | MainThread | [basetestcase:sleep:585] Reason: after failover before invoking rebalance.... Sleep for 5 secs ... 2021-03-05 14:19:53,240 | test | INFO | MainProcess | pool-1-thread-6 | [table_view:display:72] Ops Trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *..................*........................................................*....................*......X | 13872.637 | | 13238.911 | ..............................................................................................X | 13238.911 | | 12690.783 | ..........................................................................................*.X | 12865.462 | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-03-05 14:20:08,594 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1509] Rebalance done. Taken 15.0400002003 seconds to complete 2021-03-05 14:20:08,595 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1511] Sleep for 5 seconds after rebalance 2021-03-05 14:20:13,605 | test | INFO | MainProcess | MainThread | [basetestcase:sleep:585] Reason: None. Sleep for 30 secs ... 2021-03-05 14:20:43,608 | test | INFO | MainProcess | MainThread | [failovertests:run_rebalance_after_failover_and_verify:300] Begin VERIFICATION for Rebalance after Failover Only 2021-03-05 14:20:50,447 | test | WARNING | MainProcess | MainThread | [bucket_ready_functions:verify_items_count:2243] Number of nodes is less than replica requires 2021-03-05 14:20:50,453 | test | INFO | MainProcess | MainThread | [basetestcase:sleep:585] Reason: None. Sleep for 60 secs ... 2021-03-05 14:21:50,881 | test | INFO | MainProcess | MainThread | [failovertests:run_rebalance_after_failover_and_verify:333] End VERIFICATION for Rebalance after Failover Only 2021-03-05 14:21:50,881 | test | INFO | MainProcess | MainThread | [failoverbasetest:tearDown:114] === FailoverBaseTest tearDown started for test #1 test_failover_stop_server === 2021-03-05 14:21:51,051 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3050] bash: killall: command not found 2021-03-05 14:21:51,055 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3050] bash: killall: command not found 2021-03-05 14:21:51,223 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3050] bash: killall: command not found 2021-03-05 14:21:51,232 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3050] bash: killall: command not found 2021-03-05 14:21:51,404 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3050] bash: killall: command not found 2021-03-05 14:21:51,421 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3050] bash: killall: command not found 2021-03-05 14:21:51,634 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3050] bash: killall: command not found 2021-03-05 14:21:51,641 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3050] bash: killall: command not found 2021-03-05 14:22:03,052 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:747] GET http://10.112.190.103:8091/pools/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-03-05 14:22:03,089 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:747] POST http://10.112.190.103:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2021-03-05 14:22:03,355 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:747] GET http://10.112.190.104:8091/pools/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-03-05 14:22:03,382 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:747] POST http://10.112.190.104:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2021-03-05 14:22:03,604 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:747] GET http://10.112.190.105:8091/pools/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-03-05 14:22:03,628 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:747] POST http://10.112.190.105:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2021-03-05 14:22:03,680 | test | INFO | MainProcess | MainThread | [basetestcase:tearDown:398] Checking for core_dumps on servers