2021-05-19 18:34:08,367 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-19 18:34:09,323 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-19 18:34:09,862 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:321] Initializing cluster 2021-05-19 18:34:15,345 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:17,368 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:19,391 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:21,413 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:23,436 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:25,459 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:27,482 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:34,612 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:36,634 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:38,657 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:40,680 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:42,703 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:44,726 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:46,755 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:54,214 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:56,237 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:34:58,259 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:00,282 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:02,305 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:04,332 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:06,354 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:15,334 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:17,355 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:19,377 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:21,401 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:23,423 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:25,443 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:31,171 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:33,197 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:35,220 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-19 18:35:52,025 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-19 18:36:06,450 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.201 initialized 2021-05-19 18:36:06,476 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-19 18:36:06,529 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-19 18:36:26,311 | test | INFO | MainProcess | pool-6-thread-6 | [table_view:display:72] Rebalance Overview +----------------+-----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-----------------+-----------------------+---------------+--------------+ | 172.23.98.196 | index, kv, n1ql | 7.0.0-5190-enterprise | 19.4500504541 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.201 | None | | | <--- IN --- | +----------------+-----------------+-----------------------+---------------+--------------+ 2021-05-19 18:36:26,349 | test | INFO | MainProcess | pool-6-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-19 18:36:36,440 | test | INFO | MainProcess | pool-6-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-19 18:36:36,476 | test | INFO | MainProcess | pool-6-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1649999619 sec 2021-05-19 18:36:36,815 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-19 18:37:31,655 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-19 18:37:31,969 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 431319360 | 425328920 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 285025880 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 452473344 | 431889957 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-19 18:37:31,970 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-19 18:38:28,026 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-19 18:38:28,042 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-19 18:38:48,937 | test | INFO | MainProcess | pool-6-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..........................**...........*....*..............................**.............................X | 105.324 | +-------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-19 18:38:49,457 | test | INFO | MainProcess | pool-6-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *......**...*........*.............................................**......................................X | 120.127 | +-------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-19 18:38:49,667 | test | INFO | MainProcess | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *............**...........*....................*.................................................**........X | 25686.195 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-19 18:38:49,937 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+-----------------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+-----------------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.98.196 | index, kv, n1ql | 89.5614260067 | 3693387776 | 2326982656 | 0 / 3909087232 | 53575 / 157727 | 7.0.0-5190-enterprise | | 172.23.98.195 | kv | 91.9969512195 | 3693387776 | 2618859520 | 0 / 3909087232 | 80625 / 237979 | 7.0.0-5190-enterprise | | 172.23.104.186 | kv | 91.3762401425 | 6078930944 | 4883402752 | 381784064 / 3758092288 | 61361 / 181821 | 7.0.0-5190-enterprise | | 172.23.120.201 | kv | 66.7562483203 | 3691520000 | 2670747648 | 0 / 3758092288 | 62010 / 189265 | 7.0.0-5190-enterprise | | 172.23.121.10 | kv | 99.975024975 | 4201836544 | 3057864704 | 0 / 3758092288 | 62477 / 149445 | 7.0.0-5190-enterprise | +----------------+-----------------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-05-19 18:39:04,513 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-19 18:40:58,237 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 206971928 | 344776386 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319662728 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 696018264 | 540211277 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-19 18:40:58,239 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-19 18:41:09,677 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_rebalance_out 2021-05-19 18:41:09,678 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_rebalance_out 2021-05-19 18:41:09,680 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:632] failing over nodes [ip:172.23.104.186 port:8091 ssh_username:root, ip:172.23.120.201 port:8091 ssh_username:root] 2021-05-19 18:41:31,765 | test | ERROR | MainProcess | pool-6-thread-3 | [rest_client:_http_request:748] POST http://172.23.98.196:8091/controller/failOver body: otpNode=ns_1%40172.23.104.186&allowUnsafe=false headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: status: 500, content: Unexpected server error: {failover_failed,"default", "Failed to get failover info for bucket \"default\": ['ns_1@172.23.120.201']"} Unexpected server error: {failover_failed,"default", "Failed to get failover info for bucket \"default\": ['ns_1@172.23.120.201']"} auth: Administrator:password 2021-05-19 18:41:31,769 | test | ERROR | MainProcess | pool-6-thread-3 | [rest_client:fail_over:1276] ns_1@172.23.104.186 - Failover error: Unexpected server error: {failover_failed,"default", "Failed to get failover info for bucket \"default\": ['ns_1@172.23.120.201']"} 2021-05-19 18:41:33,154 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-19 18:41:33,155 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-19 18:41:57,148 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-19 18:41:57,150 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-19 18:42:30,019 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-19 18:42:30,022 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-19 18:43:20,535 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1094] Running cbcollect on node ns_1@172.23.104.186,ns_1@172.23.120.201,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 2021-05-19 18:43:30,627 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1097] ns_1@172.23.104.186,ns_1@172.23.120.201,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 - cbcollect status: True 2021-05-19 18:43:30,628 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1102] Polling active_tasks to check cbcollect status 2021-05-19 18:50:32,528 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.104.186: Copying cbcollect ZIP file to Client 2021-05-19 18:50:35,868 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.120.201: Copying cbcollect ZIP file to Client 2021-05-19 18:50:38,924 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.121.10: Copying cbcollect ZIP file to Client 2021-05-19 18:50:42,170 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.98.195: Copying cbcollect ZIP file to Client 2021-05-19 18:50:45,698 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.98.196: Copying cbcollect ZIP file to Client 2021-05-19 18:50:49,858 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:492] Skipping get_trace !! 2021-05-19 18:51:08,440 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1519999504 seconds to complete 2021-05-19 18:51:08,461 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete