2021-03-01 17:01:12,885 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #56 test_data_load_collections_with_hard_failover_recovery ========= 2021-03-01 17:01:13,877 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #56 test_data_load_collections_with_hard_failover_recovery ========= 2021-03-01 17:01:14,408 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:318] Initializing cluster 2021-03-01 17:01:48,802 | test | ERROR | MainProcess | pool-57-thread-4 | [rest_client:_http_request:751] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-03-01 17:02:09,892 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:587] RAM quota was defined less than 100 MB: 2021-03-01 17:02:27,857 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-03-01 17:02:27,881 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #56 test_data_load_collections_with_hard_failover_recovery ========= 2021-03-01 17:02:27,926 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #56 test_data_load_collections_with_hard_failover_recovery ========= 2021-03-01 17:02:46,198 | test | INFO | MainProcess | pool-57-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.98.196 | kv | 7.0.0-4565-enterprise | 12.572436382 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-03-01 17:02:46,219 | test | INFO | MainProcess | pool-57-thread-6 | [task:check:330] Rebalance - status: running, progress: 0 2021-03-01 17:02:56,302 | test | INFO | MainProcess | pool-57-thread-6 | [task:check:330] Rebalance - status: none, progress: 100 2021-03-01 17:02:56,336 | test | INFO | MainProcess | pool-57-thread-6 | [task:check:389] Rebalance completed with progress: 100% in 10.138999939 sec 2021-03-01 17:02:56,683 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1790] Creating required buckets from template 2021-03-01 17:03:06,062 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2252] Waiting for all collections to be created 2021-03-01 17:03:06,519 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 524288000 | 4517392 | 12373148 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 524288000 | 63342368 | 136 | | default | couchbase | 3 | none | 0 | 0 | 7864320000 | 0 | 0 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-03-01 17:03:06,520 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:143] Creating required SDK clients for client_pool 2021-03-01 17:03:59,107 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4645] Performing scope/collection specific operations 2021-03-01 17:03:59,117 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4735] Done Performing scope/collection specific operations 2021-03-01 17:04:29,512 | test | INFO | MainProcess | pool-57-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...*.....*.......*......................*..........*............*...................................*......X | 23034.264 | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-01 17:04:30,394 | test | INFO | MainProcess | pool-57-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..................*....................*........................*......................................X | 2028.410 | | 1726.610 | ....................................................................................X | 1726.610 | | 1326.610 | ................................................................X | 1326.610 | | 497.880 | .......................**X | 499.980 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-03-01 17:04:31,473 | test | INFO | MainProcess | pool-57-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..............*.................*.......................*..............................................X | 2382.192 | | 2202.392 | ...........................................................................................X | 2202.392 | | 1698.192 | ......................................................................X | 1698.192 | | 263.598 | ..........X | 263.598 | | 55.898 | .*X | 55.898 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-03-01 17:04:31,585 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 90.4774061782 | 3693387776 | 2692485120 | 35467264 / 3909087232 | 73803 / 0 | 7.0.0-4565-enterprise | | 172.23.98.195 | kv | 90.3497574674 | 3693387776 | 2765803520 | 42897408 / 3909087232 | 99196 / 0 | 7.0.0-4565-enterprise | | 172.23.104.186 | kv | 65.4150702427 | 6078930944 | 4995043328 | 0 / 3758092288 | 112672 / 0 | 7.0.0-4565-enterprise | | 172.23.120.206 | kv | 67.4650161464 | 3691520000 | 2766114816 | 0 / 0 | 82591 / 0 | 7.0.0-4565-enterprise | | 172.23.121.10 | kv | 99.8001998002 | 4201836544 | 3208093696 | 2359296 / 3758092288 | 82603 / 0 | 7.0.0-4565-enterprise | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2021-03-01 17:04:35,628 | test | INFO | MainProcess | pool-57-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:42,335 | test | INFO | MainProcess | pool-57-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:42,641 | test | INFO | MainProcess | pool-57-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:43,421 | test | INFO | MainProcess | pool-57-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:44,207 | test | INFO | MainProcess | pool-57-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:44,674 | test | INFO | MainProcess | pool-57-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:44,865 | test | INFO | MainProcess | pool-57-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:45,234 | test | INFO | MainProcess | pool-57-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:49,470 | test | INFO | MainProcess | pool-57-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:50,253 | test | INFO | MainProcess | pool-57-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:50,631 | test | INFO | MainProcess | pool-57-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:50,877 | test | INFO | MainProcess | pool-57-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:51,309 | test | INFO | MainProcess | pool-57-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:51,971 | test | INFO | MainProcess | pool-57-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:56,463 | test | INFO | MainProcess | pool-57-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:56,628 | test | INFO | MainProcess | pool-57-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:56,858 | test | INFO | MainProcess | pool-57-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:58,058 | test | INFO | MainProcess | pool-57-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:04:58,434 | test | INFO | MainProcess | pool-57-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:00,387 | test | INFO | MainProcess | pool-57-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:02,563 | test | INFO | MainProcess | pool-57-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:02,783 | test | INFO | MainProcess | pool-57-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:03,470 | test | INFO | MainProcess | pool-57-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:04,739 | test | INFO | MainProcess | pool-57-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:06,572 | test | INFO | MainProcess | pool-57-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:08,476 | test | INFO | MainProcess | pool-57-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:08,798 | test | INFO | MainProcess | pool-57-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:08,803 | test | INFO | MainProcess | pool-57-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:11,542 | test | INFO | MainProcess | pool-57-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:14,092 | test | INFO | MainProcess | pool-57-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:16,924 | test | INFO | MainProcess | pool-57-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:05:21,926 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4402] Validating collection stats and item counts 2021-03-01 17:06:37,105 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 30000 | 524288000 | 152500024 | 144286641 | | bucket2 | ephemeral | 3 | none | 0 | 30000 | 524288000 | 102068024 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 7864320000 | 631963592 | 566049046 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-03-01 17:06:37,105 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #56 test_data_load_collections_with_hard_failover_recovery ========= 2021-03-01 17:06:46,917 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:119] Changed Rebalance settings: {u'rebalanceMovesPerNode': 64} 2021-03-01 17:06:46,917 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:772] Doing collection data load before hard_failover_recovery 2021-03-01 17:06:46,920 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4645] Performing scope/collection specific operations 2021-03-01 17:06:46,928 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4735] Done Performing scope/collection specific operations 2021-03-01 17:06:46,986 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:295] Starting rebalance operation of type : hard_failover_recovery 2021-03-01 17:06:46,986 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:672] failing over nodes [ip:172.23.104.186 port:8091 ssh_username:root] 2021-03-01 17:06:58,967 | test | INFO | MainProcess | pool-57-thread-8 | [rest_client:monitorRebalance:1460] Rebalance done. Taken 8.21000003815 seconds to complete 2021-03-01 17:06:58,980 | test | INFO | MainProcess | pool-57-thread-8 | [common_lib:sleep:22] Sleep 8.21000003815 seconds. Reason: Wait after rebalance complete 2021-03-01 17:07:10,815 | test | INFO | MainProcess | pool-57-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.*.....*.......**.................................................................**......................X | 18692.243 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-01 17:07:12,101 | test | INFO | MainProcess | pool-57-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **..**......*...............*..................................*....................*.......................X | 1097.730 | +-------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-03-01 17:07:12,869 | test | INFO | MainProcess | pool-57-thread-27 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...*........**.....................................................................**..............*......X | 1244.152 | +-------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-03-01 17:09:07,266 | test | INFO | MainProcess | MainThread | [collections_rebalance:wait_for_failover_or_assert:265] 1 nodes failed over as expected in 0.069000005722 seconds 2021-03-01 17:09:07,266 | test | INFO | MainProcess | MainThread | [collections_rebalance:data_load_after_failover:244] Starting a sync data load after failover 2021-03-01 17:09:07,267 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4645] Performing scope/collection specific operations 2021-03-01 17:09:07,276 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4735] Done Performing scope/collection specific operations 2021-03-01 17:09:26,220 | test | INFO | MainProcess | pool-57-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *..*................*....................*.........................*.....................................X | 20137.073 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-01 17:09:27,749 | test | INFO | MainProcess | pool-57-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.......*..............................*..........................*......................*...............X | 1194.758 | | 1125.784 | .............................................................................................X | 1125.784 | +----------+------------------------------------------------------------------------------------------------------------+----------+ 2021-03-01 17:09:28,186 | test | INFO | MainProcess | pool-57-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *...........*............................*.........................*..........*..........................*X | 1332.571 | +-------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-03-01 17:09:28,279 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1718] 172.23.104.186 - Node not part of cluster inactiveFailed 2021-03-01 17:09:28,339 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1718] 172.23.104.186 - Node not part of cluster inactiveFailed 2021-03-01 17:09:32,112 | test | INFO | MainProcess | pool-57-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:33,079 | test | INFO | MainProcess | pool-57-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:39,164 | test | INFO | MainProcess | pool-57-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:39,371 | test | INFO | MainProcess | pool-57-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:39,859 | test | INFO | MainProcess | pool-57-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:39,921 | test | INFO | MainProcess | pool-57-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:41,104 | test | INFO | MainProcess | pool-57-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:45,380 | test | INFO | MainProcess | pool-57-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:45,969 | test | INFO | MainProcess | pool-57-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:48,499 | test | INFO | MainProcess | pool-57-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:50,242 | test | INFO | MainProcess | pool-57-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:51,457 | test | INFO | MainProcess | pool-57-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:51,993 | test | INFO | MainProcess | pool-57-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:56,288 | test | INFO | MainProcess | pool-57-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:57,513 | test | INFO | MainProcess | pool-57-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:58,204 | test | INFO | MainProcess | pool-57-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:09:58,569 | test | INFO | MainProcess | pool-57-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:10:03,575 | test | INFO | MainProcess | pool-57-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:10:04,165 | test | INFO | MainProcess | pool-57-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:10:04,397 | test | INFO | MainProcess | pool-57-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:10:06,095 | test | INFO | MainProcess | pool-57-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:10:11,957 | test | INFO | MainProcess | pool-57-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:10:14,121 | test | INFO | MainProcess | pool-57-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-03-01 17:10:19,486 | test | INFO | MainProcess | pool-57-thread-22 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-4565-enterprise | 8.27831386491 | Cluster node | | 172.23.98.195 | kv | 7.0.0-4565-enterprise | 9.86875315497 | Cluster node | | 172.23.104.186 | kv | 7.0.0-4565-enterprise | 2.03109327984 | Cluster node | | 172.23.120.206 | kv | 7.0.0-4565-enterprise | 6.23272178939 | Cluster node | | 172.23.121.10 | kv | 7.0.0-4565-enterprise | 33.3758937692 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-03-01 17:10:19,507 | test | INFO | MainProcess | pool-57-thread-22 | [task:check:330] Rebalance - status: running, progress: 0 2021-03-01 17:10:24,579 | test | INFO | MainProcess | pool-57-thread-22 | [task:check:330] Rebalance - status: running, progress: 0 2021-03-01 17:10:29,670 | test | INFO | MainProcess | pool-57-thread-22 | [task:check:330] Rebalance - status: running, progress: 0 2021-03-01 17:10:34,750 | test | ERROR | MainProcess | pool-57-thread-22 | [rest_client:_rebalance_status_and_progress:1506] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2021-03-01 17:10:34,839 | test | INFO | MainProcess | pool-57-thread-22 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.98.196: 2021-03-01 17:10:34,841 | test | ERROR | MainProcess | pool-57-thread-22 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614647431240L, u'shortText': u'message', u'serverTime': u'2021-03-01T17:10:31.240Z', u'text': u'Rebalance exited with reason {{badmatch,\n {error,marking_as_warmed_failed,\n [\'ns_1@172.23.104.186\']}},\n [{ns_rebalancer,\n \'-apply_delta_recovery_buckets/3-fun-0-\',2,\n [{file,"src/ns_rebalancer.erl"},\n {line,1089}]},\n {lists,foreach,2,\n [{file,"lists.erl"},{line,1338}]},\n {ns_rebalancer,apply_delta_recovery_buckets,3,\n [{file,"src/ns_rebalancer.erl"},\n {line,1086}]},\n {ns_rebalancer,rebalance_body,5,\n [{file,"src/ns_rebalancer.erl"},\n {line,524}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,197}]}]}.\nRebalance Operation Id = 3249c2b20a0702b7af38f8db578af122'} 2021-03-01 17:10:34,842 | test | ERROR | MainProcess | pool-57-thread-22 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.104.186', u'tstamp': 1614647423880L, u'shortText': u'message', u'serverTime': u'2021-03-01T17:10:23.880Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.104.186\' in 3 seconds.'} 2021-03-01 17:10:34,842 | test | ERROR | MainProcess | pool-57-thread-22 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.104.186', u'tstamp': 1614647422334L, u'shortText': u'message', u'serverTime': u'2021-03-01T17:10:22.334Z', u'text': u'Bucket "bucket1" loaded on node \'ns_1@172.23.104.186\' in 1 seconds.'} 2021-03-01 17:10:34,842 | test | ERROR | MainProcess | pool-57-thread-22 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614647419425L, u'shortText': u'message', u'serverTime': u'2021-03-01T17:10:19.425Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.98.196','ns_1@172.23.98.195',\n 'ns_1@172.23.104.186','ns_1@172.23.120.206',\n 'ns_1@172.23.121.10'], EjectNodes = [], Failed over and being ejected nodes = [], Delta recovery nodes = ['ns_1@172.23.104.186'], Delta recovery buckets = all; Operation Id = 3249c2b20a0702b7af38f8db578af122"} 2021-03-01 17:10:34,844 | test | ERROR | MainProcess | pool-57-thread-22 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614647210615L, u'shortText': u'message', u'serverTime': u'2021-03-01T17:06:50.615Z', u'text': u'Failover completed successfully.\nRebalance Operation Id = 9ba468a98fc498635766051c156c8562'} 2021-03-01 17:10:34,844 | test | ERROR | MainProcess | pool-57-thread-22 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'failover', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614647210520L, u'shortText': u'message', u'serverTime': u'2021-03-01T17:06:50.520Z', u'text': u"Deactivating failed over nodes ['ns_1@172.23.104.186']"} 2021-03-01 17:10:34,845 | test | ERROR | MainProcess | pool-57-thread-22 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'failover', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1614647210500L, u'shortText': u'message', u'serverTime': u'2021-03-01T17:06:50.500Z', u'text': u"Failed over ['ns_1@172.23.104.186']: ok"} 2021-03-01 17:10:34,845 | test | ERROR | MainProcess | pool-57-thread-22 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.104.186', u'tstamp': 1614647210354L, u'shortText': u'message', u'serverTime': u'2021-03-01T17:06:50.354Z', u'text': u'Shutting down bucket "default" on \'ns_1@172.23.104.186\' for deletion'} 2021-03-01 17:10:34,845 | test | ERROR | MainProcess | pool-57-thread-22 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.104.186', u'tstamp': 1614647208977L, u'shortText': u'message', u'serverTime': u'2021-03-01T17:06:48.977Z', u'text': u'Shutting down bucket "bucket2" on \'ns_1@172.23.104.186\' for deletion'} 2021-03-01 17:10:34,845 | test | ERROR | MainProcess | pool-57-thread-22 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.104.186', u'tstamp': 1614647207885L, u'shortText': u'message', u'serverTime': u'2021-03-01T17:06:47.885Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.104.186\' for deletion'} 2021-03-01 17:10:34,845 | test | ERROR | MainProcess | pool-57-thread-22 | [task:call:242] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2021-03-01 17:10:34,891 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:119] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-03-01 17:10:36,239 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:90] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-03-01 17:10:36,240 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:93] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-03-01 17:10:38,003 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:90] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-03-01 17:10:38,003 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:93] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-03-01 17:11:03,734 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:90] Bucket: default, Active Resident ratio(DGM): 100% 2021-03-01 17:11:03,734 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:93] Bucket: default, Replica Resident ratio(DGM): 100% 2021-03-01 17:11:52,173 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1079] Running cbcollect on node ns_1@172.23.104.186,ns_1@172.23.120.206,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 2021-03-01 17:12:02,217 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1082] ns_1@172.23.104.186,ns_1@172.23.120.206,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 - cbcollect status: True 2021-03-01 17:12:02,219 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1087] Polling active_tasks to check cbcollect status 2021-03-01 17:17:13,516 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1114] ns_1@172.23.104.186: Copying cbcollect ZIP file to Client 2021-03-01 17:17:17,489 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1135] 172.23.104.186 node cb collect zip copied on client : True 2021-03-01 17:17:17,490 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1114] ns_1@172.23.120.206: Copying cbcollect ZIP file to Client 2021-03-01 17:17:20,871 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1135] 172.23.120.206 node cb collect zip copied on client : True 2021-03-01 17:17:20,871 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1114] ns_1@172.23.121.10: Copying cbcollect ZIP file to Client 2021-03-01 17:17:25,023 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1135] 172.23.121.10 node cb collect zip copied on client : True 2021-03-01 17:17:25,023 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1114] ns_1@172.23.98.195: Copying cbcollect ZIP file to Client 2021-03-01 17:17:28,503 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1135] 172.23.98.195 node cb collect zip copied on client : True 2021-03-01 17:17:28,505 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1114] ns_1@172.23.98.196: Copying cbcollect ZIP file to Client 2021-03-01 17:17:32,851 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1135] 172.23.98.196 node cb collect zip copied on client : True 2021-03-01 17:17:32,851 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:487] Skipping get_trace !! 2021-03-01 17:17:50,322 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1460] Rebalance done. Taken 11.138999939 seconds to complete 2021-03-01 17:17:50,341 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-03-01 17:17:55,394 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:751] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused