2021-06-23 21:06:12,661 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-06-23 21:06:13,776 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-06-23 21:06:14,305 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:322] Initializing cluster 2021-06-23 21:06:20,855 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:22,878 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:24,908 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:26,944 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:28,967 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:30,993 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:33,016 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:35,039 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:43,236 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:45,265 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:47,292 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:49,318 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:51,342 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:53,367 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:55,390 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:06:57,424 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:05,421 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:07,450 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:09,482 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:11,506 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:13,529 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:15,553 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:17,585 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:19,608 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:21,631 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:31,621 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:33,645 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:35,671 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:37,694 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:39,717 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:41,743 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:43,770 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:45,795 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:52,490 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:54,513 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:56,536 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:07:58,559 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-06-23 21:08:16,334 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:600] RAM quota was defined less than 100 MB: 2021-06-23 21:08:31,446 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:344] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.201 initialized 2021-06-23 21:08:38,980 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= BaseTestCase setup finished for test #3 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-06-23 21:08:39,056 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= ClusterSetup setup started for test #3 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-06-23 21:09:04,476 | test | INFO | MainProcess | pool-5-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.1.0-1036-enterprise | 5.39450466347 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.201 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-06-23 21:09:04,509 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-06-23 21:09:14,605 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-06-23 21:09:14,638 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1620001793 sec 2021-06-23 21:09:14,960 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 5.31914893617 | 3949948928 | 3229552640 | 3932160 / 3909087232 | 911714 / 1830361 | 7.1.0-1036-enterprise | | 172.23.98.195 | kv | 3.62173038229 | 3949948928 | 3336540160 | 3407872 / 3909087232 | 0 / 0 | 7.1.0-1036-enterprise | | 172.23.104.186 | kv | 1.35712490576 | 6078930944 | 5343076352 | 262156288 / 3758092288 | 0 / 0 | 7.1.0-1036-enterprise | | 172.23.120.201 | kv | 0.501756146513 | 3945754624 | 3319541760 | 2621440 / 3758092288 | 0 / 0 | 7.1.0-1036-enterprise | | 172.23.121.10 | kv | 3.97284385215 | 4201598976 | 3557789696 | 0 / 3758092288 | 0 / 0 | 7.1.0-1036-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-06-23 21:09:14,963 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= ClusterSetup setup complete for test #3 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-06-23 21:09:14,963 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= CollectionBase setup started for test #3 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-06-23 21:09:15,361 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1824] Creating required buckets from template 2021-06-23 21:09:34,526 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2287] Waiting for all collections to be created 2021-06-23 21:09:34,707 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------------------------------------------------------------------------------------------+-----------+----------+------------+-----+-------+-------------+-----------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------------------------------------------------------------------------------------------+-----------+----------+------------+-----+-------+-------------+-----------+------------+ | 4yZIQSEJJD6IzNnFF4kWkPgVRxWd0tC-1DBmRsF05LD1MVBbaPNwgg2vNjO8SCRd_syOC9QCrPVlNpP_x-15-414000 | couchbase | 2 | none | 0 | 0 | 10485760000 | 833751648 | 2557735565 | +---------------------------------------------------------------------------------------------+-----------+----------+------------+-----+-------+-------------+-----------+------------+ 2021-06-23 21:09:34,709 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:154] Creating required SDK clients for client_pool 2021-06-23 21:10:31,532 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4688] Performing scope/collection specific operations 2021-06-23 21:10:31,549 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4778] Done Performing scope/collection specific operations 2021-06-23 21:12:59,937 | test | INFO | MainProcess | pool-5-thread-9 | [table_view:display:72] Ops trend for bucket '4yZIQSEJJD6IzNnFF4kWkPgVRxWd0tC-1DBmRsF05LD1MVBbaPNwgg2vNjO8SCRd_syOC9QCrPVlNpP_x-15-414000' +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.....**................................*....................................*..............**X | 20280.300 | | 18573.800 | ...............................................................................X | 18573.800 | | 16940.900 | ........................................................................X | 16940.900 | | 15327.500 | .................................................................**......*...........**......**...*......*..**X | 23146.200 | | 22489.700 | ................................................................................................*X | 22489.700 | | 22221.800 | ...............................................................................................*X | 22221.800 | | 21569.800 | ............................................................................................X | 21569.800 | | 19866.500 | ....................................................................................*X | 19866.500 | | 19007.600 | .................................................................................*X | 19007.600 | | 18872.300 | ................................................................................X | 18872.300 | | 18661.400 | ...............................................................................***X | 18719.100 | | 18540.100 | ...............................................................................*...*....**...*..*X | 21455.800 | | 21184.600 | ..........................................................................................*X | 21184.600 | | 21181.600 | ..........................................................................................X | 21181.600 | | 20335.600 | ......................................................................................X | 20335.600 | | 19613.200 | ...................................................................................**..*.**....*X | 21094.900 | | 21016.100 | .........................................................................................*..X | 21355.900 | | 20919.600 | .........................................................................................**X | 20919.600 | | 20272.800 | ......................................................................................X | 20272.800 | | 20000.400 | .....................................................................................*.**...*.....*X | 22015.200 | | 21914.700 | .............................................................................................*X | 21914.700 | +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-23 21:13:01,667 | test | INFO | MainProcess | pool-5-thread-2 | [table_view:display:72] Ops trend for bucket '4yZIQSEJJD6IzNnFF4kWkPgVRxWd0tC-1DBmRsF05LD1MVBbaPNwgg2vNjO8SCRd_syOC9QCrPVlNpP_x-15-414000' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 20454.100 | ...................................................................................................X | 20454.100 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2021-06-23 21:13:01,803 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 96.4673227353 | 3949948928 | 2360655872 | 3932160 / 3909087232 | 538590 / 1094689 | 7.1.0-1036-enterprise | | 172.23.98.195 | kv | 97.0930232558 | 3949948928 | 2502381568 | 3407872 / 3909087232 | 557481 / 1123456 | 7.1.0-1036-enterprise | | 172.23.104.186 | kv | 94.6329113924 | 6078930944 | 4516446208 | 262156288 / 3758092288 | 541556 / 1068700 | 7.1.0-1036-enterprise | | 172.23.120.201 | kv | 72.0930232558 | 3945754624 | 2455887872 | 2621440 / 3758092288 | 572378 / 1134775 | 7.1.0-1036-enterprise | | 172.23.121.10 | kv | 99.8752183679 | 4201598976 | 2553303040 | 0 / 3758092288 | 358211 / 662246 | 7.1.0-1036-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-06-23 21:13:37,198 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4442] Validating collection stats and item counts 2021-06-23 21:14:21,836 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------------------------------------------------------------------------------------------+-----------+----------+------------+-----+---------+-------------+------------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------------------------------------------------------------------------------------------+-----------+----------+------------+-----+---------+-------------+------------+------------+ | 4yZIQSEJJD6IzNnFF4kWkPgVRxWd0tC-1DBmRsF05LD1MVBbaPNwgg2vNjO8SCRd_syOC9QCrPVlNpP_x-15-414000 | couchbase | 2 | none | 0 | 3000000 | 10485760000 | 2613283152 | 8185163251 | +---------------------------------------------------------------------------------------------+-----------+----------+------------+-----+---------+-------------+------------+------------+ 2021-06-23 21:14:21,838 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= CollectionBase setup complete for test #3 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-06-23 21:14:29,920 | test | INFO | MainProcess | MainThread | [collections_rebalance:setUp:59] Changing scrape interval to 2 2021-06-23 21:14:32,463 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 64} 2021-06-23 21:14:32,464 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:932] Doing collection data load after graceful_failover_rebalance_out 2021-06-23 21:14:32,466 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:390] Starting rebalance operation of type : graceful_failover_rebalance_out 2021-06-23 21:14:32,467 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:602] failing over nodes [ip:172.23.104.186 port:8091 ssh_username:root, ip:172.23.120.201 port:8091 ssh_username:root] 2021-06-23 21:14:54,015 | test | INFO | MainProcess | pool-5-thread-23 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 21.3380000591 seconds to complete 2021-06-23 21:14:54,023 | test | INFO | MainProcess | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-06-23 21:16:59,104 | test | INFO | MainProcess | MainThread | [collections_rebalance:wait_for_failover_or_assert:359] 1 nodes failed over as expected in 0.0729999542236 seconds 2021-06-23 21:17:20,703 | test | INFO | MainProcess | pool-5-thread-6 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 21.376999855 seconds to complete 2021-06-23 21:17:20,710 | test | INFO | MainProcess | pool-5-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-06-23 21:19:25,789 | test | INFO | MainProcess | MainThread | [collections_rebalance:wait_for_failover_or_assert:359] 2 nodes failed over as expected in 0.0729999542236 seconds 2021-06-23 21:19:25,789 | test | INFO | MainProcess | MainThread | [collections_rebalance:data_load_after_failover:338] Starting a sync data load after failover 2021-06-23 21:19:25,790 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4688] Performing scope/collection specific operations 2021-06-23 21:19:25,802 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4778] Done Performing scope/collection specific operations 2021-06-23 21:21:01,681 | test | INFO | MainProcess | pool-5-thread-28 | [table_view:display:72] Ops trend for bucket '4yZIQSEJJD6IzNnFF4kWkPgVRxWd0tC-1DBmRsF05LD1MVBbaPNwgg2vNjO8SCRd_syOC9QCrPVlNpP_x-15-414000' +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.........................*.....................................*.......................*...............X | 44670.000 | | 32627.500 | ........................................................................*.............*........X | 42317.500 | | 28743.000 | ...............................................................X | 28743.000 | | 26749.000 | ..........................................................*.............*.............X | 38102.000 | | 34802.000 | ............................................................................*.X | 35025.500 | | 32466.500 | .......................................................................*X | 32466.500 | | 27337.000 | ............................................................X | 27337.000 | | 25285.000 | .......................................................X | 25285.000 | | 24271.500 | .....................................................**.....X | 26709.000 | | 11958.000 | .........................X | 11958.000 | | 8583.000 | ..................***.....*...............X | 17432.750 | | 3748.000 | .......*.....................*..............................*.X | 27148.500 | | 20466.000 | ............................................*......................X | 30097.500 | | 24071.000 | ....................................................*...X | 25373.000 | | 23555.000 | ...................................................X | 23555.000 | | 18971.000 | .........................................*.......X | 22250.000 | | 16269.000 | ...................................*............**X | 21544.000 | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-06-23 21:21:03,378 | test | INFO | MainProcess | pool-5-thread-21 | [table_view:display:72] Ops trend for bucket '4yZIQSEJJD6IzNnFF4kWkPgVRxWd0tC-1DBmRsF05LD1MVBbaPNwgg2vNjO8SCRd_syOC9QCrPVlNpP_x-15-414000' +----------+------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------+----------+ | 9911.500 | ...................................................................................................X | 9911.500 | +----------+------------------------------------------------------------------------------------------------------+----------+ 2021-06-23 21:21:03,503 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1756] 172.23.104.186 - Node not part of cluster inactiveFailed 2021-06-23 21:21:03,503 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1756] 172.23.120.201 - Node not part of cluster inactiveFailed 2021-06-23 21:21:03,559 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1756] 172.23.104.186 - Node not part of cluster inactiveFailed 2021-06-23 21:21:03,561 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1756] 172.23.120.201 - Node not part of cluster inactiveFailed 2021-06-23 21:21:59,946 | test | INFO | MainProcess | pool-5-thread-11 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.98.196 | kv | 7.1.0-1036-enterprise | 10.5732484076 | Cluster node | | 172.23.98.195 | kv | 7.1.0-1036-enterprise | 24.7787610619 | Cluster node | | 172.23.104.186 | kv | 7.1.0-1036-enterprise | 1.00755667506 | --- OUT ---> | | 172.23.120.201 | kv | 7.1.0-1036-enterprise | 0.752823086575 | --- OUT ---> | | 172.23.121.10 | kv | 7.1.0-1036-enterprise | 12.2474747475 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-06-23 21:21:59,969 | test | INFO | MainProcess | pool-5-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-06-23 21:22:05,049 | test | ERROR | MainProcess | pool-5-thread-11 | [rest_client:_rebalance_status_and_progress:1548] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f0aad3f00adaee87bb1adbd0bebd9b01', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=ad9a439d75fbd35ab3c1a91d5f6e7f6f', u'status': u'notRunning'} - rebalance failed 2021-06-23 21:22:05,178 | test | INFO | MainProcess | pool-5-thread-11 | [rest_client:print_UI_logs:2694] Latest logs from UI on 172.23.98.196: 2021-06-23 21:22:05,180 | test | ERROR | MainProcess | pool-5-thread-11 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.98.196', u'tstamp': 1624508520866L, u'shortText': u'message', u'serverTime': u'2021-06-23T21:22:00.866Z', u'text': u'Rebalance exited with reason {badarg,\n [{io_lib,format,\n ["Rebalancing bucket ~p with config ~p",\n ["4yZIQSEJJD6IzNnFF4kWkPgVRxWd0tC-1DBmRsF05LD1MVBbaPNwgg2vNjO8SCRd_syOC9QCrPVlNpP_x-15-414000",\n [{auth_type,sasl},\n {num_replicas,2},\n {replica_index,true},\n {ram_quota,2097152000},\n {durability_min_level,none},\n {frag_percent,50},\n {pitr_enabled,false},\n {pitr_granularity,600},\n {pitr_max_history_age,86400},\n {autocompaction,false},\n {purge_interval,undefined},\n {flush_enabled,false},\n {num_threads,3},\n {eviction_policy,full_eviction},\n {conflict_resolution_type,seqno},\n {storage_mode,magma},\n {max_ttl,0},\n {compression_mode,active},\n {type,membase},\n {num_vbuckets,1024},\n {replication_topology,star},\n {repl_type,dcp},\n {servers,\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\']},\n {map_opts_hash,133465355},\n {map,\n [[\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',undefined,\n undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',undefined,\n undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.121.10\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',undefined,\n undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.121.10\',\n \'ns_1@172.23.98.195\'|...],\n [\'ns_1@172.23.121.10\'|...],\n [...]|...]},\n {fastForwardMap,undefined}]]],\n [{file,"io_lib.erl"},{line,202}]},\n {\'ale_logger-rebalance\',info,5,\n [{file,[]},{line,39}]},\n {ns_rebalancer,rebalance_bucket,6,\n [{file,"src/ns_rebalancer.erl"},{line,622}]},\n {lists,foreach,2,\n [{file,"lists.erl"},{line,1338}]},\n {ns_rebalancer,rebalance_kv,4,\n [{file,"src/ns_rebalancer.erl"},{line,605}]},\n {ns_rebalancer,rebalance_body,5,\n [{file,"src/ns_rebalancer.erl"},{line,556}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{...}]}]}.\nRebalance Operation Id = ...'} 2021-06-23 21:22:05,207 | test | ERROR | MainProcess | pool-5-thread-11 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1624508520725L, u'shortText': u'message', u'serverTime': u'2021-06-23T21:22:00.725Z', u'text': u'Started rebalancing bucket 4yZIQSEJJD6IzNnFF4kWkPgVRxWd0tC-1DBmRsF05LD1MVBbaPNwgg2vNjO8SCRd_syOC9QCrPVlNpP_x-15-414000'} 2021-06-23 21:22:05,207 | test | ERROR | MainProcess | pool-5-thread-11 | [rest_client:print_UI_logs:2696] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.98.195', u'tstamp': 1624508520628L, u'shortText': u'node down', u'serverTime': u'2021-06-23T21:22:00.628Z', u'text': u"Node 'ns_1@172.23.98.195' saw that node 'ns_1@172.23.120.201' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-06-23 21:22:05,209 | test | ERROR | MainProcess | pool-5-thread-11 | [rest_client:print_UI_logs:2696] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.98.196', u'tstamp': 1624508520626L, u'shortText': u'node down', u'serverTime': u'2021-06-23T21:22:00.626Z', u'text': u"Node 'ns_1@172.23.98.196' saw that node 'ns_1@172.23.120.201' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-06-23 21:22:05,210 | test | ERROR | MainProcess | pool-5-thread-11 | [rest_client:print_UI_logs:2696] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.121.10', u'tstamp': 1624508520626L, u'shortText': u'node down', u'serverTime': u'2021-06-23T21:22:00.626Z', u'text': u"Node 'ns_1@172.23.121.10' saw that node 'ns_1@172.23.120.201' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-06-23 21:22:05,210 | test | ERROR | MainProcess | pool-5-thread-11 | [rest_client:print_UI_logs:2696] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.120.201', u'tstamp': 1624508520540L, u'shortText': u'node down', u'serverTime': u'2021-06-23T21:22:00.540Z', u'text': u"Node 'ns_1@172.23.120.201' saw that node 'ns_1@172.23.104.186' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-06-23 21:22:05,210 | test | ERROR | MainProcess | pool-5-thread-11 | [rest_client:print_UI_logs:2696] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.98.195', u'tstamp': 1624508520538L, u'shortText': u'node down', u'serverTime': u'2021-06-23T21:22:00.538Z', u'text': u"Node 'ns_1@172.23.98.195' saw that node 'ns_1@172.23.104.186' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-06-23 21:22:05,211 | test | ERROR | MainProcess | pool-5-thread-11 | [rest_client:print_UI_logs:2696] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.121.10', u'tstamp': 1624508520535L, u'shortText': u'node down', u'serverTime': u'2021-06-23T21:22:00.535Z', u'text': u"Node 'ns_1@172.23.121.10' saw that node 'ns_1@172.23.104.186' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-06-23 21:22:05,213 | test | ERROR | MainProcess | pool-5-thread-11 | [rest_client:print_UI_logs:2696] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.98.196', u'tstamp': 1624508520535L, u'shortText': u'node down', u'serverTime': u'2021-06-23T21:22:00.535Z', u'text': u"Node 'ns_1@172.23.98.196' saw that node 'ns_1@172.23.104.186' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-06-23 21:22:05,213 | test | ERROR | MainProcess | pool-5-thread-11 | [rest_client:print_UI_logs:2696] {u'code': 1, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.120.201', u'tstamp': 1624508520528L, u'shortText': u'message', u'serverTime': u'2021-06-23T21:22:00.528Z', u'text': u"Node 'ns_1@172.23.120.201' is leaving cluster."} 2021-06-23 21:22:05,214 | test | ERROR | MainProcess | pool-5-thread-11 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f0aad3f00adaee87bb1adbd0bebd9b01', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=ad9a439d75fbd35ab3c1a91d5f6e7f6f', u'status': u'notRunning'} - rebalance failed 2021-06-23 21:22:05,220 | test | INFO | MainProcess | MainThread | [collections_rebalance:tearDown:85] Reverting prometheus settings back to default 2021-06-23 21:22:08,128 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-06-23 21:22:09,602 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:78] Bucket: 4yZIQSEJJD6IzNnFF4kWkPgVRxWd0tC-1DBmRsF05LD1MVBbaPNwgg2vNjO8SCRd_syOC9QCrPVlNpP_x-15-414000, Active Resident ratio(DGM): 100% 2021-06-23 21:22:09,604 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:81] Bucket: 4yZIQSEJJD6IzNnFF4kWkPgVRxWd0tC-1DBmRsF05LD1MVBbaPNwgg2vNjO8SCRd_syOC9QCrPVlNpP_x-15-414000, Replica Resident ratio(DGM): 100% 2021-06-23 21:22:53,627 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1094] Running cbcollect on node ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 2021-06-23 21:23:03,676 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1097] ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 - cbcollect status: True 2021-06-23 21:23:03,677 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1102] Polling active_tasks to check cbcollect status 2021-06-23 21:33:06,321 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.121.10: Copying cbcollect ZIP file to Client 2021-06-23 21:33:07,371 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1159] ns_1@172.23.121.10 - Failed to copy cb collect zip file 2021-06-23 21:33:07,371 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.98.195: Copying cbcollect ZIP file to Client 2021-06-23 21:33:10,013 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.98.196: Copying cbcollect ZIP file to Client 2021-06-23 21:33:12,950 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:495] Skipping get_trace !! 2021-06-23 21:33:43,318 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:253] DELETE http://172.23.98.196:8091/pools/default/buckets/4yZIQSEJJD6IzNnFF4kWkPgVRxWd0tC-1DBmRsF05LD1MVBbaPNwgg2vNjO8SCRd_syOC9QCrPVlNpP_x-15-414000 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-06-23 21:33:44,582 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3079] wc: /opt/couchbase/var/lib/couchbase/data: Is a directory 2021-06-23 21:33:44,584 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:1535] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-06-23 21:33:51,325 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 6.11699986458 seconds to complete 2021-06-23 21:33:51,345 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 6.11699986458 seconds. Reason: Wait after rebalance complete 2021-06-23 21:33:57,532 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-23 21:33:57,536 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-23 21:34:02,573 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-23 21:34:02,578 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-23 21:34:07,615 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-23 21:34:07,619 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-23 21:34:12,640 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:289] Node 172.23.121.10:8091 is in a broken state! 2021-06-23 21:34:12,641 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:cleanup_cluster:191] Can't create rest connection after rebalance out for ejected nodes, will retry after 10 seconds according to MB-8430: Unable to reach the host @ 172.23.121.10 2021-06-23 21:34:12,661 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: MB-8430 2021-06-23 21:34:22,684 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-23 21:34:22,687 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-23 21:34:27,729 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-23 21:34:27,733 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-23 21:34:32,772 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-23 21:34:32,776 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-23 21:34:37,793 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:289] Node 172.23.121.10:8091 is in a broken state! 2021-06-23 21:34:37,796 | test | WARNING | MainProcess | MainThread | [basetestcase:tearDownEverything:506] Killing memcached due to Unable to reach the host @ 172.23.121.10