2021-07-20 22:03:26,621 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:561] ========= CollectionsDropRecreateRebalance setup started for test #1 test_data_load_collections_with_rebalance_out ========= 2021-07-20 22:03:27,036 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:561] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_rebalance_out ========= 2021-07-20 22:03:27,940 | test | INFO | MainProcess | MainThread | [basetestcase:initialize_cluster:350] Initializing cluster : C1 2021-07-20 22:03:31,030 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.135:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:03:33,459 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.135:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:03:35,480 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.135:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:03:37,500 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.135:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:03:43,252 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.136:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:03:45,270 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.136:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:03:47,289 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.136:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:03:49,305 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.136:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:03:55,275 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.139:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:03:57,289 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.139:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:03:59,303 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.139:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:01,318 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.139:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:06,407 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.140:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:08,421 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.140:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:10,438 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.140:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:12,451 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.140:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:18,000 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.141:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:20,016 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.141:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:22,030 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.141:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:24,045 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.141:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:29,384 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.48:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:31,398 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.48:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:33,413 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.48:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:04:35,427 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.48:8091/nodes/self error [Errno 111] Connection refused 2021-07-20 22:05:09,651 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:613] RAM quota was defined less than 100 MB: 2021-07-20 22:05:22,269 | test | INFO | MainProcess | MainThread | [basetestcase:initialize_cluster:375] Cluster C1 initialized 2021-07-20 22:05:25,865 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:561] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_rebalance_out ========= 2021-07-20 22:05:25,888 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:561] ========= ClusterSetup setup started for test #1 test_data_load_collections_with_rebalance_out ========= 2021-07-20 22:05:38,463 | test | INFO | MainProcess | pool-3-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.121.135 | kv | 7.1.0-1083-enterprise | 0.713749060856 | Cluster node | | 172.23.121.136 | None | | | <--- IN --- | | 172.23.121.139 | None | | | <--- IN --- | | 172.23.121.140 | None | | | <--- IN --- | | 172.23.121.141 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-07-20 22:05:38,479 | test | INFO | MainProcess | pool-3-thread-7 | [task:check:356] Rebalance - status: running, progress: 0.0 2021-07-20 22:05:48,516 | test | INFO | MainProcess | pool-3-thread-7 | [task:check:356] Rebalance - status: none, progress: 100 2021-07-20 22:05:48,530 | test | INFO | MainProcess | pool-3-thread-7 | [task:check:415] Rebalance completed with progress: 100% in 10.0679998398 sec 2021-07-20 22:05:48,661 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ | 172.23.121.140 | kv | 0.250344223307 | 11.45 GiB | 10.87 GiB | 0.0 Byte / 0.0 Byte | 0 / 0 | 7.1.0-1083-enterprise | | 172.23.121.136 | kv | 0.851170359244 | 11.45 GiB | 10.78 GiB | 108.84 MiB / 3.50 GiB | 0 / 0 | 7.1.0-1083-enterprise | | 172.23.121.139 | kv | 0.137758296807 | 11.45 GiB | 10.87 GiB | 0.0 Byte / 0.0 Byte | 0 / 0 | 7.1.0-1083-enterprise | | 172.23.121.135 | kv | 1.02989198694 | 11.45 GiB | 10.78 GiB | 122.95 MiB / 3.50 GiB | 0 / 0 | 7.1.0-1083-enterprise | | 172.23.121.141 | kv | 0.926969810848 | 11.45 GiB | 10.77 GiB | 0.0 Byte / 0.0 Byte | 0 / 0 | 7.1.0-1083-enterprise | +----------------+----------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ 2021-07-20 22:05:48,663 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:561] ========= ClusterSetup setup complete for test #1 test_data_load_collections_with_rebalance_out ========= 2021-07-20 22:05:48,664 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:561] ========= CollectionBase setup started for test #1 test_data_load_collections_with_rebalance_out ========= 2021-07-20 22:05:48,855 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1883] Creating required buckets from template 2021-07-20 22:06:00,490 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2363] Waiting for all collections to be created 2021-07-20 22:06:00,786 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +------------------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+------------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +------------------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+------------+-----+ | nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000 | couchbase | magma | 2 | none | 0 | 0 | 9.77 GiB | 329.38 MiB | 643.91 MiB | 100 | +------------------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+------------+-----+ 2021-07-20 22:06:00,788 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:157] Creating required SDK clients for client_pool 2021-07-20 22:06:14,045 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:06:14,059 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:06:45,951 | test | INFO | MainProcess | pool-3-thread-10 | [table_view:display:72] Ops trend for bucket 'nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000' +-------+-----------------------------------------------------------------------------------------------------------------------------+------------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------------------------+------------+ | 0.000 | *****..........................................................********....................................*******......**X | 102113.400 | +-------+-----------------------------------------------------------------------------------------------------------------------------+------------+ 2021-07-20 22:06:46,045 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ | 172.23.121.140 | kv | 89.9680102367 | 11.45 GiB | 10.11 GiB | 0.0 Byte / 0.0 Byte | 357020 / 724533 | 7.1.0-1083-enterprise | | 172.23.121.136 | kv | 90.9590090665 | 11.45 GiB | 10.05 GiB | 108.84 MiB / 3.50 GiB | 366671 / 690674 | 7.1.0-1083-enterprise | | 172.23.121.139 | kv | 89.5146871009 | 11.45 GiB | 10.09 GiB | 0.0 Byte / 0.0 Byte | 338279 / 675689 | 7.1.0-1083-enterprise | | 172.23.121.135 | kv | 88.9812092548 | 11.45 GiB | 10.07 GiB | 122.95 MiB / 3.50 GiB | 366003 / 716027 | 7.1.0-1083-enterprise | | 172.23.121.141 | kv | 89.8073733895 | 11.45 GiB | 10.05 GiB | 0.0 Byte / 0.0 Byte | 344330 / 699218 | 7.1.0-1083-enterprise | +----------------+----------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ 2021-07-20 22:06:55,075 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4478] Validating collection stats and item counts 2021-07-20 22:07:15,842 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +------------------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-----------+----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +------------------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-----------+----------+-----------+-----+ | nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000 | couchbase | magma | 2 | none | 0 | 3000000 | 9.77 GiB | 2.78 GiB | 3.10 GiB | 100 | +------------------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-----------+----------+-----------+-----+ 2021-07-20 22:07:15,844 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:561] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_rebalance_out ========= 2021-07-20 22:07:15,869 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:138] Changed Rebalance settings: {u'rebalanceMovesPerNode': 2} 2021-07-20 22:07:16,335 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:07:16,509 | test | INFO | MainProcess | pool-3-thread-4 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.121.140 | kv | 7.1.0-1083-enterprise | 4.8921224285 | Cluster node | | 172.23.121.136 | kv | 7.1.0-1083-enterprise | 4.82654600302 | --- OUT ---> | | 172.23.121.139 | kv | 7.1.0-1083-enterprise | 4.95037064958 | Cluster node | | 172.23.121.135 | kv | 7.1.0-1083-enterprise | 4.73737119879 | --- OUT ---> | | 172.23.121.141 | kv | 7.1.0-1083-enterprise | 4.77506911284 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-07-20 22:07:16,522 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 0.0 2021-07-20 22:07:21,584 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 0.45 2021-07-20 22:07:26,625 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 0.73 2021-07-20 22:07:31,661 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 0.73 2021-07-20 22:07:36,703 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 0.87 2021-07-20 22:07:37,578 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:07:37,579 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:07:41,740 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 0.87 2021-07-20 22:07:46,778 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:07:51,812 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:07:56,855 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:07:59,566 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:07:59,566 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:08:01,894 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:08:06,947 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:08:11,983 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:08:17,026 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:08:21,479 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:08:21,479 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:08:22,066 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:08:27,105 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:08:32,147 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:08:37,180 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:08:42,229 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:08:42,851 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:08:42,851 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:08:47,275 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:08:52,312 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:08:57,352 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:02,398 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:05,088 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:09:05,088 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:09:07,437 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:12,473 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:17,509 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:22,545 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:26,792 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:09:26,792 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:09:27,586 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:32,634 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:37,677 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:42,710 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:47,746 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:48,053 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:09:48,055 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:09:52,783 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:09:57,819 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:02,858 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:07,898 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:09,309 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:10:09,311 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:10:12,944 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:17,986 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:23,026 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:28,065 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:30,219 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:10:30,220 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:10:33,104 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:38,150 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:43,194 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:48,232 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:51,400 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:10:51,401 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:10:53,275 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:10:58,313 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:03,349 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:08,388 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:12,701 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:11:12,701 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:11:13,434 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:18,490 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:23,543 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:28,579 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:33,621 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:33,798 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:11:33,799 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:11:38,661 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:43,707 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:48,785 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:53,822 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:11:55,075 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:11:55,076 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:11:58,865 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:03,908 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:08,948 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:13,990 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:16,516 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:12:16,517 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:12:19,028 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:24,073 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:29,114 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:34,148 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:37,894 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:12:37,894 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:12:39,188 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:44,226 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:49,263 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:54,302 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:12:58,855 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:12:58,855 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:12:59,336 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:04,375 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:09,418 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:14,453 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:19,492 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:19,648 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:13:19,648 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:13:24,529 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:29,569 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:34,609 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:39,653 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:40,562 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:13:40,562 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:13:44,704 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:49,746 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:54,782 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:13:59,819 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:14:01,355 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:14:01,355 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:14:04,868 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:14:09,911 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:14:14,950 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:14:19,992 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:14:22,272 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:14:22,273 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:14:25,026 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:14:30,069 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:14:35,108 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:14:40,144 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:14:43,142 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:14:43,142 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:14:45,180 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:14:50,217 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:14:55,259 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:00,296 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:04,088 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:15:04,089 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:15:05,332 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:10,388 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:15,430 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:20,470 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:24,806 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:15:24,808 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:15:25,507 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:30,566 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:35,604 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:40,651 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:45,690 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:45,783 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:15:45,783 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:15:50,726 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:15:55,776 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:00,818 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:05,855 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:06,776 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:16:06,776 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:16:10,894 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:15,934 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:20,973 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:26,012 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:27,503 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:16:27,503 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:16:31,051 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:36,088 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:41,134 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:46,174 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:48,578 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:16:48,578 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:16:51,223 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:16:56,266 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:01,299 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:06,336 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:08,221 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:17:08,223 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:17:11,375 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:16,424 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:21,460 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:26,494 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:27,812 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:17:27,813 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:17:31,539 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:36,575 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:41,611 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:46,651 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:47,469 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:17:47,470 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:17:51,690 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:17:56,726 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:01,759 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:06,799 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:06,828 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:18:06,828 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:18:11,835 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:16,882 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:21,921 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:26,321 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:18:26,322 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:18:26,959 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:32,101 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:37,140 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:42,173 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:45,861 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:18:45,861 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:18:47,207 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:52,243 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:18:57,282 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:02,316 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:05,378 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:19:05,378 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:19:07,352 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:12,395 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:17,431 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:22,469 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:24,698 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:19:24,698 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:19:27,507 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:32,545 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:37,582 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:42,618 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:44,134 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:19:44,134 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:19:47,651 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:52,696 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:19:57,732 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:02,770 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:03,815 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:20:03,815 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:20:07,813 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:12,855 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:17,888 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:22,924 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:23,280 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:20:23,283 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:20:27,961 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:32,996 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:38,032 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:42,760 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:20:42,760 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:20:43,068 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:48,108 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:53,150 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:20:58,183 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:02,236 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:21:02,236 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:21:03,219 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:08,256 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:13,296 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:18,328 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:21,762 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:21:21,762 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:21:23,368 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:28,405 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:33,440 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:38,585 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:41,042 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:21:41,042 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:21:43,628 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:48,667 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:53,701 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:21:58,740 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:00,391 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:22:00,391 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:22:03,776 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:08,815 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:13,848 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:18,884 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:19,589 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:22:19,589 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:22:23,918 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:28,953 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:33,987 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:38,670 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:22:38,670 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:22:39,023 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:44,058 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:49,091 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:54,121 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:22:58,292 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:22:58,293 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:22:59,158 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:04,197 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:09,234 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:14,267 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:17,880 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:23:17,881 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:23:19,303 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:24,348 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:29,384 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:34,415 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:37,299 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:23:37,299 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:23:39,456 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:44,493 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:49,530 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:54,572 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:23:57,325 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:23:57,325 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:23:59,648 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:24:04,700 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:24:09,740 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:24:14,773 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:24:18,704 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:24:18,704 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:24:19,812 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:24:24,851 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:24:29,888 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:24:34,928 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:24:39,706 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:24:39,706 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:24:39,967 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:24:45,012 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:24:50,053 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:24:55,089 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:00,137 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:00,384 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:25:00,384 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:25:05,177 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:10,214 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:15,256 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:20,290 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:20,861 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:25:20,862 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:25:25,338 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:30,375 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:35,410 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:40,447 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:40,766 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:25:40,766 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:25:45,482 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:50,519 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:25:55,555 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:00,589 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:01,089 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:26:01,091 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:26:05,625 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:10,660 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:15,694 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:20,739 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:21,380 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:26:21,380 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:26:25,779 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:30,813 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:35,849 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:40,887 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:41,411 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:26:41,413 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:26:45,937 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:50,973 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:26:56,007 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:01,052 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:01,273 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:27:01,273 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:27:06,091 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:11,130 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:16,167 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:21,206 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:21,553 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:27:21,553 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:27:26,240 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:31,279 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:36,322 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:41,355 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:41,878 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:27:41,880 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:27:46,401 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:51,438 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:27:56,480 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:28:01,517 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:28:02,289 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:28:02,289 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4724] Performing scope/collection specific operations 2021-07-20 22:28:06,562 | test | INFO | MainProcess | pool-3-thread-4 | [task:check:356] Rebalance - status: running, progress: 1.03 2021-07-20 22:28:11,601 | test | ERROR | MainProcess | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1547] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'ef4929b52095d0e1b7f1ce7c96d01b84', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=a43d262e6e2c00c53e6aa53b0a22d187', u'status': u'notRunning'} - rebalance failed 2021-07-20 22:28:11,624 | test | INFO | MainProcess | pool-3-thread-4 | [rest_client:print_UI_logs:2693] Latest logs from UI on 172.23.121.135: 2021-07-20 22:28:11,624 | test | ERROR | MainProcess | pool-3-thread-4 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.121.135', u'tstamp': 1626845288009L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:28:08.009Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.10706.1>,\n {{dcp_wait_for_data_move_failed,\n "nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000",\n 203,\'ns_1@172.23.121.135\',\n [\'ns_1@172.23.121.141\',\n \'ns_1@172.23.121.140\',\n \'ns_1@172.23.121.139\'],\n {error,no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,459}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 49de3cd996b5984e6a69b46336b253fa'} 2021-07-20 22:28:11,625 | test | ERROR | MainProcess | pool-3-thread-4 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.121.135', u'tstamp': 1626845287930L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:28:07.930Z', u'text': u'Worker <0.10565.1> (for action {move,{203,\n [\'ns_1@172.23.121.135\',\n \'ns_1@172.23.121.141\',\n \'ns_1@172.23.121.140\'],\n [\'ns_1@172.23.121.141\',\n \'ns_1@172.23.121.140\',\n \'ns_1@172.23.121.139\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.10706.1>,\n {{dcp_wait_for_data_move_failed,\n "nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000",\n 203,\n \'ns_1@172.23.121.135\',\n [\'ns_1@172.23.121.141\',\n \'ns_1@172.23.121.140\',\n \'ns_1@172.23.121.139\'],\n {error,\n no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 459}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 234}]}]}}}'} 2021-07-20 22:28:11,625 | test | ERROR | MainProcess | pool-3-thread-4 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.121.135', u'tstamp': 1626844037048L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:07:17.048Z', u'text': u'Bucket "nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000" rebalance does not seem to be swap rebalance'} 2021-07-20 22:28:11,625 | test | ERROR | MainProcess | pool-3-thread-4 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.121.135', u'tstamp': 1626844036710L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:07:16.710Z', u'text': u'Started rebalancing bucket nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000'} 2021-07-20 22:28:11,625 | test | ERROR | MainProcess | pool-3-thread-4 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.121.135', u'tstamp': 1626844036461L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:07:16.461Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.140','ns_1@172.23.121.139',\n 'ns_1@172.23.121.141'], EjectNodes = ['ns_1@172.23.121.135',\n 'ns_1@172.23.121.136'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 49de3cd996b5984e6a69b46336b253fa"} 2021-07-20 22:28:11,627 | test | ERROR | MainProcess | pool-3-thread-4 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.121.140', u'tstamp': 1626843994955L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:06:34.955Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>] (repeated 1 times, last seen 56.584199 secs ago)'} 2021-07-20 22:28:11,627 | test | ERROR | MainProcess | pool-3-thread-4 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.121.139', u'tstamp': 1626843991844L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:06:31.844Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>] (repeated 1 times, last seen 53.47707 secs ago)'} 2021-07-20 22:28:11,627 | test | ERROR | MainProcess | pool-3-thread-4 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.121.136', u'tstamp': 1626843988851L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:06:28.851Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>] (repeated 1 times, last seen 50.484877 secs ago)'} 2021-07-20 22:28:11,627 | test | ERROR | MainProcess | pool-3-thread-4 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.121.135', u'tstamp': 1626843951980L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:05:51.980Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>] (repeated 1 times, last seen 13.615585 secs ago)'} 2021-07-20 22:28:11,628 | test | ERROR | MainProcess | pool-3-thread-4 | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'mb_master', u'type': u'info', u'node': u'ns_1@172.23.121.135', u'tstamp': 1626843951980L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:05:51.980Z', u'text': u"I'm the only node, so I'm the master. (repeated 1 times, last seen 25.847708 secs ago)"} 2021-07-20 22:28:11,628 | test | ERROR | MainProcess | pool-3-thread-4 | [task:call:268] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'ef4929b52095d0e1b7f1ce7c96d01b84', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=a43d262e6e2c00c53e6aa53b0a22d187', u'status': u'notRunning'} - rebalance failed 2021-07-20 22:28:11,664 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:138] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-07-20 22:28:22,914 | test | INFO | MainProcess | Thread-621 | [bucket_ready_functions:perform_tasks_from_spec:4814] Done Performing scope/collection specific operations 2021-07-20 22:28:22,915 | test | INFO | MainProcess | Thread-621 | [table_view:display:72] Data load details +-----------------------------------+---------------+ | Operation | Value | +-----------------------------------+---------------+ | Collections dropped and recreated | 250 | | Scopes dropped and recreated | 3 | | Cycles of data load | 62 | | Time Elapsed in secs | 1266.57999992 | +-----------------------------------+---------------+ 2021-07-20 22:28:25,053 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:78] Bucket: nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000, Active Resident ratio(DGM): 98% 2021-07-20 22:28:25,055 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:81] Bucket: nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000, Replica Resident ratio(DGM): 33% 2021-07-20 22:28:50,631 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1077] Running cbcollect on node ns_1@172.23.121.135,ns_1@172.23.121.136,ns_1@172.23.121.139,ns_1@172.23.121.140,ns_1@172.23.121.141 2021-07-20 22:29:00,661 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1080] ns_1@172.23.121.135,ns_1@172.23.121.136,ns_1@172.23.121.139,ns_1@172.23.121.140,ns_1@172.23.121.141 - cbcollect status: True 2021-07-20 22:29:00,661 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1085] Polling active_tasks to check cbcollect status 2021-07-20 22:35:21,381 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1112] ns_1@172.23.121.135: Copying cbcollect ZIP file to Client 2021-07-20 22:35:23,476 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1112] ns_1@172.23.121.136: Copying cbcollect ZIP file to Client 2021-07-20 22:35:24,647 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1112] ns_1@172.23.121.139: Copying cbcollect ZIP file to Client 2021-07-20 22:35:27,395 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1112] ns_1@172.23.121.140: Copying cbcollect ZIP file to Client 2021-07-20 22:35:28,555 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1112] ns_1@172.23.121.141: Copying cbcollect ZIP file to Client 2021-07-20 22:35:31,371 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:507] Skipping get_trace !! 2021-07-20 22:36:01,575 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:253] DELETE http://172.23.121.139:8091/pools/default/buckets/nm3B1VRZsUKP8%25uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-07-20 22:36:02,134 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3079] wc: /data: Is a directory 2021-07-20 22:36:02,134 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:1600] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-07-20 22:37:02,734 | test | ERROR | MainProcess | MainThread | [rest_client:_rebalance_status_and_progress:1547] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'c1c165f18c785319c05fa682bc2aecf1', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=d01941df2fd419d5d80e86d97ff5ae9c', u'status': u'notRunning'} - rebalance failed 2021-07-20 22:37:02,750 | test | INFO | MainProcess | MainThread | [rest_client:print_UI_logs:2693] Latest logs from UI on 172.23.121.139: 2021-07-20 22:37:02,750 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.121.135', u'tstamp': 1626845822325L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:37:02.325Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.121.139\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000"]}}}]}.\nRebalance Operation Id = 53db27c75ac95e7e4c470d9d9fdcf2d3'} 2021-07-20 22:37:02,750 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.121.135', u'tstamp': 1626845822323L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:37:02.323Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.121.139\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000"]}}}]\n'} 2021-07-20 22:37:02,750 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.121.135', u'tstamp': 1626845762320L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:36:02.320Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.139'], EjectNodes = ['ns_1@172.23.121.140',\n 'ns_1@172.23.121.136',\n 'ns_1@172.23.121.135',\n 'ns_1@172.23.121.141'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 53db27c75ac95e7e4c470d9d9fdcf2d3"} 2021-07-20 22:37:02,750 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2695] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.121.139', u'tstamp': 1626845762313L, u'shortText': u'client-side error report', u'serverTime': u'2021-07-20T22:36:02.313Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.121.139\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.121.140\', u\'ns_1@172.23.121.136\', u\'ns_1@172.23.121.135\', u\'ns_1@172.23.121.141\']'} 2021-07-20 22:37:02,752 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.139', u'tstamp': 1626845732268L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:35:32.268Z', u'text': u'Shutting down bucket "nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000" on \'ns_1@172.23.121.139\' for deletion'} 2021-07-20 22:37:02,752 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.141', u'tstamp': 1626845732053L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:35:32.053Z', u'text': u'Shutting down bucket "nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000" on \'ns_1@172.23.121.141\' for deletion'} 2021-07-20 22:37:02,752 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.140', u'tstamp': 1626845731986L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:35:31.986Z', u'text': u'Shutting down bucket "nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000" on \'ns_1@172.23.121.140\' for deletion'} 2021-07-20 22:37:02,752 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.135', u'tstamp': 1626845731986L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:35:31.986Z', u'text': u'Shutting down bucket "nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000" on \'ns_1@172.23.121.135\' for deletion'} 2021-07-20 22:37:02,752 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.136', u'tstamp': 1626845731963L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:35:31.963Z', u'text': u'Shutting down bucket "nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000" on \'ns_1@172.23.121.136\' for deletion'} 2021-07-20 22:37:02,753 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2695] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.121.141', u'tstamp': 1626845331271L, u'shortText': u'message', u'serverTime': u'2021-07-20T22:28:51.271Z', u'text': u'Bucket "nm3B1VRZsUKP8%uMu-txjkDFBi-NGJ4oIQx1jai4hU91hdLnYwwoOp4TKoathcSVXpoqeboBdQncke-48-878000" loaded on node \'ns_1@172.23.121.141\' in 10 seconds.'} 2021-07-20 22:37:02,753 | test | WARNING | MainProcess | MainThread | [basetestcase:tearDownEverything:519] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'c1c165f18c785319c05fa682bc2aecf1', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=d01941df2fd419d5d80e86d97ff5ae9c', u'status': u'notRunning'} - rebalance failed