2021-06-20 12:50:30,614 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= CollectionsRebalance setup started for test #12 test_data_load_collections_with_graceful_failover_recovery ========= 2021-06-20 12:50:31,713 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= BaseTestCase setup started for test #12 test_data_load_collections_with_graceful_failover_recovery ========= 2021-06-20 12:50:31,947 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:322] Initializing cluster 2021-06-20 12:50:38,138 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:50:40,147 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:50:42,155 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:50:44,164 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:50:46,174 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:50:48,184 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:50:50,193 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:50:52,201 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:50:59,957 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:01,967 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:03,976 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:05,984 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:07,996 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:10,006 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:12,015 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:14,023 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:16,032 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:23,375 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:25,384 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:27,392 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:29,401 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:31,411 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:33,420 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:35,428 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:37,437 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:39,447 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:47,742 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:49,752 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:51,760 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:53,769 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:55,779 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:57,788 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:51:59,796 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:52:01,805 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:52:03,815 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:52:09,384 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:52:11,394 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:52:13,404 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:52:15,413 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-06-20 12:52:32,555 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:600] RAM quota was defined less than 100 MB: 2021-06-20 12:52:47,434 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:344] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.201 initialized 2021-06-20 12:52:54,322 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= BaseTestCase setup finished for test #12 test_data_load_collections_with_graceful_failover_recovery ========= 2021-06-20 12:52:54,342 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= CollectionBase setup started for test #12 test_data_load_collections_with_graceful_failover_recovery ========= 2021-06-20 12:53:17,558 | test | INFO | MainProcess | pool-14-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.1.0-1029-enterprise | 1.03378719112 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.201 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-06-20 12:53:17,578 | test | INFO | MainProcess | pool-14-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-06-20 12:53:27,631 | test | INFO | MainProcess | pool-14-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-06-20 12:53:27,651 | test | INFO | MainProcess | pool-14-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.0939998627 sec 2021-06-20 12:53:27,884 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1794] Creating required buckets from template 2021-06-20 12:53:44,171 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2257] Waiting for all collections to be created 2021-06-20 12:53:44,267 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+----------+------------+-----+-------+-------------+-----------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-----------------+-----------+----------+------------+-----+-------+-------------+-----------+------------+ | TgO1z-27-919000 | couchbase | 2 | none | 0 | 0 | 10485760000 | 538713272 | 2473189140 | +-----------------+-----------+----------+------------+-----+-------+-------------+-----------+------------+ 2021-06-20 12:53:44,269 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:176] Creating required SDK clients for client_pool 2021-06-20 12:54:02,390 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4658] Performing scope/collection specific operations 2021-06-20 12:54:02,394 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4748] Done Performing scope/collection specific operations 2021-06-20 12:56:20,283 | test | INFO | MainProcess | pool-14-thread-9 | [table_view:display:72] Ops trend for bucket 'TgO1z-27-919000' +-----------+------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **...**..............*.................................................**..................**......*..........*X | 24517.443 | | 24157.443 | .................................................................................................*X | 24157.443 | | 23924.500 | ................................................................................................X | 23924.500 | | 22661.850 | ...........................................................................................*X | 22661.850 | | 21961.631 | ........................................................................................*X | 21961.631 | | 21206.531 | .....................................................................................X | 21206.531 | | 20452.048 | ..................................................................................****X | 20506.801 | | 20060.625 | ................................................................................**.**...*.....*..**.*..*X | 23486.940 | | 23197.008 | .............................................................................................*X | 23197.008 | | 22756.374 | ...........................................................................................X | 22756.374 | | 21476.249 | ......................................................................................**.X | 21818.682 | | 21336.075 | ......................................................................................*..*X | 21864.843 | | 21067.443 | ....................................................................................**..X | 21379.605 | | 20989.963 | ....................................................................................**..*X | 21529.579 | | 21343.816 | ......................................................................................**X | 21463.516 | | 21235.039 | .....................................................................................**..**.*X | 21968.851 | | 21906.561 | ........................................................................................*...**.*X | 22936.904 | +-----------+------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-20 12:56:20,354 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 97.7906100929 | 3949948928 | 2407419904 | 3407872 / 3909087232 | 520675 / 1048358 | 7.1.0-1029-enterprise | | 172.23.98.195 | kv | 96.1868686869 | 3949948928 | 2453090304 | 2621440 / 3909087232 | 555953 / 1119433 | 7.1.0-1029-enterprise | | 172.23.104.186 | kv | 94.9392712551 | 6078930944 | 4609646592 | 277803008 / 3758092288 | 544156 / 1069288 | 7.1.0-1029-enterprise | | 172.23.120.201 | kv | 70.057381325 | 3945754624 | 2513211392 | 2359296 / 3758092288 | 567073 / 1123301 | 7.1.0-1029-enterprise | | 172.23.121.10 | kv | 99.8750312422 | 4201611264 | 2628341760 | 0 / 3758092288 | 260212 / 569346 | 7.1.0-1029-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-06-20 12:56:55,704 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4412] Validating collection stats and item counts 2021-06-20 12:57:28,921 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+----------+------------+-----+---------+-------------+------------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-----------------+-----------+----------+------------+-----+---------+-------------+------------+------------+ | TgO1z-27-919000 | couchbase | 2 | none | 0 | 3000000 | 10485760000 | 2579109024 | 8037956876 | +-----------------+-----------+----------+------------+-----+---------+-------------+------------+------------+ 2021-06-20 12:57:28,921 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:548] ========= CollectionBase setup complete for test #12 test_data_load_collections_with_graceful_failover_recovery ========= 2021-06-20 12:57:36,742 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_recovery 2021-06-20 12:57:36,743 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-06-20 12:57:36,743 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.104.186 port:8091 ssh_username:root, ip:172.23.120.201 port:8091 ssh_username:root] 2021-06-20 12:57:58,119 | test | INFO | MainProcess | pool-14-thread-24 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 21.1889998913 seconds to complete 2021-06-20 12:57:58,124 | test | INFO | MainProcess | pool-14-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-06-20 13:00:03,161 | test | INFO | MainProcess | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0350000858307 seconds 2021-06-20 13:00:32,683 | test | INFO | MainProcess | pool-14-thread-21 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 29.3660001755 seconds to complete 2021-06-20 13:00:32,684 | test | INFO | MainProcess | pool-14-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-06-20 13:02:37,723 | test | INFO | MainProcess | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0360000133514 seconds 2021-06-20 13:02:37,723 | test | INFO | MainProcess | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-06-20 13:02:37,724 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4658] Performing scope/collection specific operations 2021-06-20 13:02:37,729 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4748] Done Performing scope/collection specific operations 2021-06-20 13:03:48,914 | test | INFO | MainProcess | pool-14-thread-27 | [table_view:display:72] Ops trend for bucket 'TgO1z-27-919000' +-----------+--------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...**...................*......................*.............................*......X | 33815.562 | | 33815.562 | ..............................................................................*...............X | 40240.762 | | 40240.762 | .............................................................................................X | 40240.762 | | 39940.762 | ............................................................................................*.......*X | 42779.788 | | 39082.088 | ..........................................................................................*X | 39082.088 | | 37631.627 | ......................................................................................X | 37631.627 | | 36598.973 | ....................................................................................*X | 36598.973 | | 26757.525 | .............................................................*X | 26757.525 | | 19840.525 | .............................................**....*.......**.........*X | 28517.754 | | 27875.784 | ................................................................X | 27875.784 | | 26602.584 | .............................................................*X | 26602.584 | | 25024.070 | .........................................................X | 25024.070 | +-----------+--------------------------------------------------------------------------------------------------------+-----------+ 2021-06-20 13:03:48,996 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1756] 172.23.104.186 - Node not part of cluster inactiveFailed 2021-06-20 13:03:48,997 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1756] 172.23.120.201 - Node not part of cluster inactiveFailed 2021-06-20 13:03:49,029 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1756] 172.23.104.186 - Node not part of cluster inactiveFailed 2021-06-20 13:03:49,029 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1756] 172.23.120.201 - Node not part of cluster inactiveFailed 2021-06-20 13:04:50,463 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4658] Performing scope/collection specific operations 2021-06-20 13:04:50,467 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4748] Done Performing scope/collection specific operations 2021-06-20 13:04:50,586 | test | INFO | MainProcess | pool-14-thread-12 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.98.196 | kv | 7.1.0-1029-enterprise | 9.11630269172 | Cluster node | | 172.23.98.195 | kv | 7.1.0-1029-enterprise | 8.6032388664 | Cluster node | | 172.23.104.186 | kv | 7.1.0-1029-enterprise | 0.978670012547 | Cluster node | | 172.23.120.201 | kv | 7.1.0-1029-enterprise | 0.652446675031 | Cluster node | | 172.23.121.10 | kv | 7.1.0-1029-enterprise | 68.0910240202 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-06-20 13:04:50,602 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-06-20 13:04:55,657 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-06-20 13:05:00,750 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.42 2021-06-20 13:05:05,890 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 3.07 2021-06-20 13:05:11,009 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 5.59 2021-06-20 13:05:16,134 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 8.66 2021-06-20 13:05:21,210 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 10.2 2021-06-20 13:05:26,293 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 11.73 2021-06-20 13:05:31,361 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 12.15 2021-06-20 13:05:36,466 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 12.99 2021-06-20 13:05:41,539 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 13.55 2021-06-20 13:05:46,598 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 14.8 2021-06-20 13:05:51,655 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 15.42 2021-06-20 13:05:56,723 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 16.56 2021-06-20 13:06:01,829 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 19.74 2021-06-20 13:06:06,904 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 23.32 2021-06-20 13:06:07,753 | test | INFO | MainProcess | pool-14-thread-17 | [table_view:display:72] Ops trend for bucket 'TgO1z-27-919000' +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***...***.........................................**...........................**......*....................***..X | 38603.904 | | 35082.333 | .........................................................................................X | 35082.333 | | 34554.433 | ........................................................................................*X | 34554.433 | | 34320.993 | .......................................................................................*.X | 34740.693 | | 33171.893 | ....................................................................................**...............X | 38802.300 | | 20474.149 | ...................................................X | 20474.149 | | 20474.149 | ...................................................X | 20474.149 | | 18660.919 | ...............................................*X | 18660.919 | | 17962.047 | .............................................*......................X | 26500.949 | | 26119.519 | ..................................................................*X | 26119.519 | | 25244.174 | ................................................................X | 25244.174 | | 23334.076 | ...........................................................X | 23334.076 | | 23334.076 | ...........................................................X | 23334.076 | | 21572.072 | ......................................................X | 21572.072 | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-20 13:06:11,976 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 27.34 2021-06-20 13:06:17,029 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 31.95 2021-06-20 13:06:22,068 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 34.72 2021-06-20 13:06:27,117 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 36.59 2021-06-20 13:06:32,157 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 39.97 2021-06-20 13:06:37,197 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 41.73 2021-06-20 13:06:42,270 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 46.59 2021-06-20 13:06:47,312 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 50.85 2021-06-20 13:06:52,354 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 53.61 2021-06-20 13:06:57,398 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 58.3 2021-06-20 13:07:02,437 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 61.66 2021-06-20 13:07:07,484 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.24 2021-06-20 13:07:12,535 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 71.35 2021-06-20 13:07:17,575 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 75.98 2021-06-20 13:07:22,622 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 81.1 2021-06-20 13:07:27,667 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 86.07 2021-06-20 13:07:32,713 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 90.18 2021-06-20 13:07:37,753 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 93.76 2021-06-20 13:07:42,793 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 95.6 2021-06-20 13:07:47,832 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 97.49 2021-06-20 13:07:52,869 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 98.83 2021-06-20 13:07:57,910 | test | INFO | MainProcess | pool-14-thread-12 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-06-20 13:08:02,947 | test | ERROR | MainProcess | pool-14-thread-12 | [rest_client:_rebalance_status_and_progress:1548] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'b6cfe0428ab0e839c75d6bfdb66b9db0', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=7215b92b5b18997d7995d0ec3c471f11', u'status': u'notRunning'} - rebalance failed 2021-06-20 13:08:02,996 | test | INFO | MainProcess | pool-14-thread-12 | [rest_client:print_UI_logs:2694] Latest logs from UI on 172.23.98.196: 2021-06-20 13:08:02,996 | test | ERROR | MainProcess | pool-14-thread-12 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.98.196', u'tstamp': 1624219677478L, u'shortText': u'message', u'serverTime': u'2021-06-20T13:07:57.478Z', u'text': u'Rebalance exited with reason bad_replicas.\nRebalance Operation Id = 876afb4ba3c08274d0c4c6786036d1e2'} 2021-06-20 13:08:02,997 | test | ERROR | MainProcess | pool-14-thread-12 | [rest_client:print_UI_logs:2696] {u'code': 2, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1624219677475L, u'shortText': u'message', u'serverTime': u'2021-06-20T13:07:57.475Z', u'text': u"Bad replicators after rebalance:\nMissing = [{'ns_1@172.23.104.186','ns_1@172.23.98.196',92}]\nExtras = []"} 2021-06-20 13:08:02,997 | test | ERROR | MainProcess | pool-14-thread-12 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1624219498374L, u'shortText': u'message', u'serverTime': u'2021-06-20T13:04:58.374Z', u'text': u'Bucket "TgO1z-27-919000" rebalance does not seem to be swap rebalance'} 2021-06-20 13:08:02,997 | test | ERROR | MainProcess | pool-14-thread-12 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.104.186', u'tstamp': 1624219496344L, u'shortText': u'message', u'serverTime': u'2021-06-20T13:04:56.344Z', u'text': u'Bucket "TgO1z-27-919000" loaded on node \'ns_1@172.23.104.186\' in 0 seconds.'} 2021-06-20 13:08:02,997 | test | ERROR | MainProcess | pool-14-thread-12 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.120.201', u'tstamp': 1624219496245L, u'shortText': u'message', u'serverTime': u'2021-06-20T13:04:56.245Z', u'text': u'Bucket "TgO1z-27-919000" loaded on node \'ns_1@172.23.120.201\' in 0 seconds.'} 2021-06-20 13:08:02,999 | test | ERROR | MainProcess | pool-14-thread-12 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1624219495941L, u'shortText': u'message', u'serverTime': u'2021-06-20T13:04:55.941Z', u'text': u'Started rebalancing bucket TgO1z-27-919000'} 2021-06-20 13:08:02,999 | test | ERROR | MainProcess | pool-14-thread-12 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.104.186', u'tstamp': 1624219491160L, u'shortText': u'message', u'serverTime': u'2021-06-20T13:04:51.160Z', u'text': u'Deleting old data files of bucket "TgO1z-27-919000"'} 2021-06-20 13:08:02,999 | test | ERROR | MainProcess | pool-14-thread-12 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.120.201', u'tstamp': 1624219491159L, u'shortText': u'message', u'serverTime': u'2021-06-20T13:04:51.159Z', u'text': u'Deleting old data files of bucket "TgO1z-27-919000"'} 2021-06-20 13:08:02,999 | test | ERROR | MainProcess | pool-14-thread-12 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1624219490520L, u'shortText': u'message', u'serverTime': u'2021-06-20T13:04:50.520Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.98.196','ns_1@172.23.98.195',\n 'ns_1@172.23.104.186','ns_1@172.23.120.201',\n 'ns_1@172.23.121.10'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 876afb4ba3c08274d0c4c6786036d1e2"} 2021-06-20 13:08:02,999 | test | ERROR | MainProcess | pool-14-thread-12 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1624219221441L, u'shortText': u'message', u'serverTime': u'2021-06-20T13:00:21.441Z', u'text': u'Graceful failover completed successfully.\nRebalance Operation Id = f1295272d022ddc972f66442c70deeb6'} 2021-06-20 13:08:03,000 | test | ERROR | MainProcess | pool-14-thread-12 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'b6cfe0428ab0e839c75d6bfdb66b9db0', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=7215b92b5b18997d7995d0ec3c471f11', u'status': u'notRunning'} - rebalance failed 2021-06-20 13:08:04,573 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:100] Bucket: TgO1z-27-919000, Active Resident ratio(DGM): 100% 2021-06-20 13:08:04,573 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:103] Bucket: TgO1z-27-919000, Replica Resident ratio(DGM): 100% 2021-06-20 13:08:48,757 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1094] Running cbcollect on node ns_1@172.23.104.186,ns_1@172.23.120.201,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 2021-06-20 13:08:58,785 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1097] ns_1@172.23.104.186,ns_1@172.23.120.201,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 - cbcollect status: True 2021-06-20 13:08:58,785 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1102] Polling active_tasks to check cbcollect status 2021-06-20 13:16:19,690 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.104.186: Copying cbcollect ZIP file to Client 2021-06-20 13:16:21,608 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.120.201: Copying cbcollect ZIP file to Client 2021-06-20 13:16:23,176 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.121.10: Copying cbcollect ZIP file to Client 2021-06-20 13:16:25,529 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.98.195: Copying cbcollect ZIP file to Client 2021-06-20 13:16:27,605 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.98.196: Copying cbcollect ZIP file to Client 2021-06-20 13:16:30,214 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:495] Skipping get_trace !! 2021-06-20 13:17:00,463 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:253] DELETE http://172.23.98.196:8091/pools/default/buckets/TgO1z-27-919000 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-06-20 13:17:01,625 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3079] wc: /opt/couchbase/var/lib/couchbase/data: Is a directory 2021-06-20 13:17:01,625 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:1505] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-06-20 13:17:11,016 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 9.08999991417 seconds to complete 2021-06-20 13:17:11,025 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 9.08999991417 seconds. Reason: Wait after rebalance complete 2021-06-20 13:17:20,186 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-20 13:17:20,187 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-20 13:17:25,206 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-20 13:17:25,207 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-20 13:17:30,223 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-20 13:17:30,226 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-20 13:17:35,234 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:289] Node 172.23.121.10:8091 is in a broken state! 2021-06-20 13:17:35,234 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:cleanup_cluster:191] Can't create rest connection after rebalance out for ejected nodes, will retry after 10 seconds according to MB-8430: Unable to reach the host @ 172.23.121.10 2021-06-20 13:17:35,242 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: MB-8430 2021-06-20 13:17:45,252 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-20 13:17:45,253 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-20 13:17:50,269 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-20 13:17:50,272 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-20 13:17:55,288 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:748] GET http://172.23.121.10:8091/nodes/self body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "Node is unknown to this cluster." auth: Administrator:password 2021-06-20 13:17:55,289 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:284] Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry 2021-06-20 13:18:00,299 | test | ERROR | MainProcess | MainThread | [rest_client:__init__:289] Node 172.23.121.10:8091 is in a broken state! 2021-06-20 13:18:00,299 | test | WARNING | MainProcess | MainThread | [basetestcase:tearDownEverything:506] Killing memcached due to Unable to reach the host @ 172.23.121.10