2021-02-18 00:54:56,763 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-02-18 00:54:57,200 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-02-18 00:54:57,410 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:318] Initializing cluster 2021-02-18 00:55:27,736 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:_http_request:751] Socket error while connecting to http://172.23.106.225:8091/nodes/self error [Errno 98] Address already in use 2021-02-18 00:55:34,986 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:587] RAM quota was defined less than 100 MB: 2021-02-18 00:55:49,095 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.106.209, 172.23.106.225, 172.23.106.232, 172.23.106.239, 172.23.106.246 initialized 2021-02-18 00:55:49,108 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-02-18 00:55:49,127 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-02-18 00:55:55,351 | test | INFO | MainProcess | pool-5-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.106.209 | kv | 7.0.0-4486-enterprise | 1.16468378209 | Cluster node | | 172.23.106.225 | None | | | <--- IN --- | | 172.23.106.232 | None | | | <--- IN --- | | 172.23.106.239 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-18 00:55:55,358 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:330] Rebalance - status: running, progress: 0 2021-02-18 00:56:05,391 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:330] Rebalance - status: none, progress: 100 2021-02-18 00:56:05,405 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:389] Rebalance completed with progress: 100% in 10.0549998283 sec 2021-02-18 00:56:05,536 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1791] Creating required buckets from template 2021-02-18 00:56:15,487 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2247] Waiting for all collections to be created 2021-02-18 00:56:15,555 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | default | couchbase | 2 | none | 0 | 0 | 3355443200 | 231591872 | 0 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-02-18 00:56:15,555 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:143] Creating required SDK clients for client_pool 2021-02-18 00:56:28,710 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4640] Performing scope/collection specific operations 2021-02-18 00:56:28,714 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4730] Done Performing scope/collection specific operations 2021-02-18 00:56:36,700 | test | INFO | MainProcess | pool-5-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................**X | 12124.300 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-02-18 00:56:36,766 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-------------+-------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-------------+-------------+----------------+-------------------+-----------------------+ | 172.23.106.225 | kv | 34.3661609668 | 25084014592 | 23972839424 | 0 / 3758092288 | 30213 / 60197 | 7.0.0-4486-enterprise | | 172.23.106.239 | kv | 35.7660455487 | 25084014592 | 23947546624 | 0 / 3758092288 | 30337 / 60784 | 7.0.0-4486-enterprise | | 172.23.106.232 | kv | 37.4774774775 | 25084010496 | 23929245696 | 0 / 3758092288 | 30253 / 60840 | 7.0.0-4486-enterprise | | 172.23.106.209 | kv | 32.6190170666 | 25084014592 | 23947173888 | 0 / 3758092288 | 30348 / 59130 | 7.0.0-4486-enterprise | +----------------+----------+-----------------+-------------+-------------+----------------+-------------------+-----------------------+ 2021-02-18 00:56:39,436 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4397] Validating collection stats and item counts 2021-02-18 00:56:54,088 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | default | couchbase | 2 | none | 0 | 500000 | 3355443200 | 543871760 | 518856208 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-02-18 00:56:54,089 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-02-18 00:56:56,861 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:753] Doing collection data load during rebalance_in_out 2021-02-18 00:56:56,861 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_to_dgm:199] Loading docs with maxttl:0 in default collection in order to load bucket in dgm 2021-02-18 00:57:00,392 | test | INFO | MainProcess | pool-5-thread-11 | [task:call:2085] Starting DGM doc loading task 2021-02-18 00:57:01,641 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2069] DGM doc loading for 'default' to atleast 55% 2021-02-18 00:57:01,641 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:01,641 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:03,494 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:03,496 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:05,436 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:05,437 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:07,306 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:07,306 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:09,233 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:09,233 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:09,773 | infra | ERROR | MainProcess | pool-5-thread-11 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.209:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 98] Address already in use 2021-02-18 00:57:13,959 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:13,959 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:15,766 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:15,766 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:17,493 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:17,494 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:19,249 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:19,249 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:21,052 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:21,052 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:22,796 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:22,796 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:24,562 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:24,562 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:26,345 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:26,345 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:26,918 | infra | ERROR | MainProcess | pool-5-thread-11 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.209:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 98] Address already in use 2021-02-18 00:57:31,176 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:31,176 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:32,953 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:32,953 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:34,737 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:34,737 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:36,592 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:36,592 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:38,364 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:38,365 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:40,108 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:40,109 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:41,894 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:41,894 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:43,756 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:43,756 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:45,510 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:45,510 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:47,303 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:47,305 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:49,164 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:49,164 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:50,907 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:50,907 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:52,697 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:52,697 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:54,483 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:54,484 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:56,359 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:56,359 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:58,165 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:58,165 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:57:59,914 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:57:59,915 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:01,752 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:01,752 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:03,559 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:03,559 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:05,342 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:05,342 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:07,105 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:07,105 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:08,963 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:08,963 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:10,733 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:10,733 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:12,513 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:12,513 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:14,384 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:14,384 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:16,154 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:16,154 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:17,920 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:17,920 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:19,723 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:19,723 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:21,512 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:21,513 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:23,276 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:23,278 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:25,108 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:25,108 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:26,880 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:26,881 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:28,714 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:28,714 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:30,515 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:30,516 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:32,286 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:32,288 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:34,082 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:34,082 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:35,905 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:35,907 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:37,720 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:37,721 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:39,493 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:39,493 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:41,332 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:41,334 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:43,138 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:43,138 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:43,674 | infra | ERROR | MainProcess | pool-5-thread-11 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.209:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 98] Address already in use 2021-02-18 00:58:47,930 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:47,930 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:49,753 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:49,753 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:51,611 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:51,611 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:53,451 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:53,451 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:55,259 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:55,259 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:57,072 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:57,072 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:58:58,937 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:58:58,937 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:00,750 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:00,750 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:02,571 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:02,572 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:04,355 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:04,355 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:06,171 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:06,171 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:07,990 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:07,990 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:09,802 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:09,802 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:11,599 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:11,599 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:13,433 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:13,433 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:15,257 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:15,259 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:17,086 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:17,086 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:18,864 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:18,864 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:20,686 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:20,686 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:22,500 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:22,502 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:24,313 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:24,313 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:26,119 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:26,119 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:27,937 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:27,937 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:29,710 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:29,710 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:31,556 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:31,556 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:33,369 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:33,369 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:35,184 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:35,184 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:36,990 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:36,990 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:38,799 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:38,799 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:40,618 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:40,618 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:42,414 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:42,414 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:44,240 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:44,240 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:46,046 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:46,046 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:47,851 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:47,851 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:49,667 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:49,667 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:51,437 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:51,437 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:53,263 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:53,265 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:55,102 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:55,104 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:56,936 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:56,936 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 00:59:58,759 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 00:59:58,759 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 01:00:00,533 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:00,533 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 01:00:02,325 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:02,326 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 01:00:04,105 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:04,105 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 01:00:05,990 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:05,990 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 01:00:07,819 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:07,821 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 01:00:09,634 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:09,635 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 01:00:11,464 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:11,466 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 100 2021-02-18 01:00:13,292 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:13,293 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 88.7319013403 2021-02-18 01:00:15,134 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:15,134 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 56.9882555556 2021-02-18 01:00:16,967 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:16,967 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 56.9882555556 2021-02-18 01:00:18,832 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:18,832 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 56.9882555556 2021-02-18 01:00:20,690 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2073] Active_resident_items_ratio for default is 100 2021-02-18 01:00:20,690 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2075] Replica_resident_items_ratio for default is 56.9882555556 2021-02-18 01:00:22,506 | test | INFO | MainProcess | pool-5-thread-11 | [task:_load_bucket_into_dgm:2079] Active DGM 100% Replica DGM 45.4014552547% achieved for 'default'. Loaded docs: 4240000 2021-02-18 01:00:22,749 | test | INFO | MainProcess | pool-5-thread-11 | [task:call:2102] Done loading docs for DGM 2021-02-18 01:00:25,539 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+---------+------------+------------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+---------+------------+------------+------------+ | default | couchbase | 2 | none | 0 | 4700000 | 3355443200 | 2524055552 | 3195819117 | +---------+-----------+----------+------------+-----+---------+------------+------------+------------+ 2021-02-18 01:00:25,539 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:276] Starting rebalance operation of type : rebalance_in_out 2021-02-18 01:00:28,075 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4640] Performing scope/collection specific operations 2021-02-18 01:00:28,151 | test | INFO | MainProcess | pool-5-thread-2 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.106.225 | kv | 7.0.0-4486-enterprise | 29.8924180328 | Cluster node | | 172.23.106.246 | kv | 7.0.0-4486-enterprise | 0 | Cluster node | | 172.23.106.239 | kv | 7.0.0-4486-enterprise | 28.0962128966 | --- OUT ---> | | 172.23.106.232 | kv | 7.0.0-4486-enterprise | 28.5769132327 | Cluster node | | 172.23.106.209 | kv | 7.0.0-4486-enterprise | 37.9892555641 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-18 01:00:28,160 | test | INFO | MainProcess | pool-5-thread-2 | [task:check:330] Rebalance - status: running, progress: 0 2021-02-18 01:00:29,785 | infra | ERROR | MainProcess | ThreadPoolExecutor-45_7 | [Rest_Connection:_http_request:253] POST http://172.23.106.209:8091/pools/default/buckets/default/scopes/@ensureManifest/d4 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password 2021-02-18 01:00:29,788 | infra | ERROR | MainProcess | ThreadPoolExecutor-45_5 | [Rest_Connection:_http_request:253] POST http://172.23.106.209:8091/pools/default/buckets/default/scopes/@ensureManifest/d3 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password 2021-02-18 01:00:33,198 | test | INFO | MainProcess | pool-5-thread-2 | [task:check:330] Rebalance - status: running, progress: 2.22772617819 2021-02-18 01:00:38,227 | test | INFO | MainProcess | pool-5-thread-2 | [task:check:330] Rebalance - status: running, progress: 11.7204269866 2021-02-18 01:00:39,229 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4730] Done Performing scope/collection specific operations 2021-02-18 01:00:43,273 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:_rebalance_status_and_progress:1494] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2021-02-18 01:00:43,309 | test | INFO | MainProcess | pool-5-thread-2 | [rest_client:print_UI_logs:2599] Latest logs from UI on 172.23.106.209: 2021-02-18 01:00:43,309 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:print_UI_logs:2601] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.106.209', u'tstamp': 1613638839746L, u'shortText': u'message', u'serverTime': u'2021-02-18T01:00:39.746Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.24171.1>,\n {{dcp_wait_for_data_move_failed,"default",\n 481,\'ns_1@172.23.106.225\',\n [\'ns_1@172.23.106.246\',\n \'ns_1@172.23.106.232\'],\n {error,no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,465}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 375f8fa5c330e3fc0660b209a82b1784'} 2021-02-18 01:00:43,311 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:print_UI_logs:2601] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.106.209', u'tstamp': 1613638839653L, u'shortText': u'message', u'serverTime': u'2021-02-18T01:00:39.653Z', u'text': u'Worker <0.24154.1> (for action {move,{481,\n [\'ns_1@172.23.106.225\',\n \'ns_1@172.23.106.239\',\n \'ns_1@172.23.106.232\'],\n [\'ns_1@172.23.106.225\',\n \'ns_1@172.23.106.246\',\n \'ns_1@172.23.106.232\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.24171.1>,\n {{dcp_wait_for_data_move_failed,\n "default",\n 481,\n \'ns_1@172.23.106.225\',\n [\'ns_1@172.23.106.246\',\n \'ns_1@172.23.106.232\'],\n {error,\n no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 465}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 234}]}]}}}'} 2021-02-18 01:00:43,311 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:print_UI_logs:2601] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.106.209', u'tstamp': 1613638830542L, u'shortText': u'message', u'serverTime': u'2021-02-18T01:00:30.542Z', u'text': u'Bucket "default" rebalance appears to be swap rebalance'} 2021-02-18 01:00:43,312 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:print_UI_logs:2601] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.106.246', u'tstamp': 1613638829324L, u'shortText': u'message', u'serverTime': u'2021-02-18T01:00:29.324Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.106.246\' in 0 seconds.'} 2021-02-18 01:00:43,312 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:print_UI_logs:2601] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.106.209', u'tstamp': 1613638829192L, u'shortText': u'message', u'serverTime': u'2021-02-18T01:00:29.192Z', u'text': u'Started rebalancing bucket default'} 2021-02-18 01:00:43,312 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:print_UI_logs:2601] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.106.246', u'tstamp': 1613638828137L, u'shortText': u'message', u'serverTime': u'2021-02-18T01:00:28.137Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]'} 2021-02-18 01:00:43,312 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:print_UI_logs:2601] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.106.209', u'tstamp': 1613638828131L, u'shortText': u'message', u'serverTime': u'2021-02-18T01:00:28.131Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.106.225','ns_1@172.23.106.246',\n 'ns_1@172.23.106.232','ns_1@172.23.106.209'], EjectNodes = ['ns_1@172.23.106.239'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 375f8fa5c330e3fc0660b209a82b1784"} 2021-02-18 01:00:43,312 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:print_UI_logs:2601] {u'code': 3, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.106.246', u'tstamp': 1613638828032L, u'shortText': u'message', u'serverTime': u'2021-02-18T01:00:28.032Z', u'text': u'Node ns_1@172.23.106.246 joined cluster'} 2021-02-18 01:00:43,312 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:print_UI_logs:2601] {u'code': 1, u'module': u'menelaus_web_sup', u'type': u'info', u'node': u'ns_1@172.23.106.246', u'tstamp': 1613638827998L, u'shortText': u'web start ok', u'serverTime': u'2021-02-18T01:00:27.998Z', u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.106.246\'. Version: "7.0.0-4486-enterprise".'} 2021-02-18 01:00:43,312 | test | ERROR | MainProcess | pool-5-thread-2 | [rest_client:print_UI_logs:2601] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.106.239', u'tstamp': 1613638825968L, u'shortText': u'node up', u'serverTime': u'2021-02-18T01:00:25.968Z', u'text': u"Node 'ns_1@172.23.106.239' saw that node 'ns_1@172.23.106.246' came up. Tags: []"} 2021-02-18 01:00:43,313 | test | ERROR | MainProcess | pool-5-thread-2 | [task:call:242] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2021-02-18 01:00:44,015 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:90] Bucket: default, Active Resident ratio(DGM): 100% 2021-02-18 01:00:44,016 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:93] Bucket: default, Replica Resident ratio(DGM): 50% 2021-02-18 01:00:58,767 | infra | ERROR | MainProcess | pool-5-thread-12 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.209:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 98] Address already in use 2021-02-18 01:01:16,598 | infra | ERROR | MainProcess | pool-5-thread-12 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.209:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 98] Address already in use 2021-02-18 01:01:51,144 | test | INFO | MainProcess | pool-5-thread-1 | [table_view:display:72] Keys failed in default:_default:_default +--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1039940 | {'error': 'com.couchbase.client.core.error.UnambiguousTimeoutException | GetRequest, Reason: TIMEOUT | 172.23.106.246:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-1039905 | {'error': 'com.couchbase.client.core.error.UnambiguousTimeoutException | GetRequest, Reason: TIMEOUT | 172.23.106.246:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | +--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-18 01:01:51,151 | test | INFO | MainProcess | pool-5-thread-18 | [table_view:display:72] Keys failed in default:_default:_default +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-227381 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.106.246:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-227254 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.106.246:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-227247 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.106.246:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-227211 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.106.246:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | +-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-18 01:01:51,280 | infra | ERROR | MainProcess | pool-5-thread-12 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.106.209:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno 98] Address already in use 2021-02-18 01:01:53,848 | test | ERROR | MainProcess | pool-5-thread-24 | [task:call:3765] Failed to load 2 docs from 848500 to 1527300 2021-02-18 01:01:58,730 | test | INFO | MainProcess | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-02-18 01:00:39, Test's start timestamp is 2021-02-18 00:54:56.763000 2021-02-18 01:01:58,732 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:789] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-02-18 01:01:58,732 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:791] 172.23.106.225: Found ' ERROR ' logs - ['2021-02-18T01:00:39.636961-08:00 ERROR 1582: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.106.225->ns_1@172.23.106.246:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":0,"cas":0,"datatype":"raw","extlen":0,"keylen":0,"magic":"ClientResponse","opaque":55,"opcode":"DCP_SYSTEM_EVENT","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.106.225->ns_1@172.23.106.246:default, vb:481, state:in-memory\n'] 2021-02-18 01:02:00,470 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1079] Running cbcollect on node ns_1@172.23.106.209,ns_1@172.23.106.225,ns_1@172.23.106.232,ns_1@172.23.106.239,ns_1@172.23.106.246 2021-02-18 01:02:10,490 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1082] ns_1@172.23.106.209,ns_1@172.23.106.225,ns_1@172.23.106.232,ns_1@172.23.106.239,ns_1@172.23.106.246 - cbcollect status: True 2021-02-18 01:02:10,492 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1087] Polling active_tasks to check cbcollect status 2021-02-18 01:02:50,569 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:751] Socket error while connecting to http://172.23.106.209:8091/pools/default/tasks error [Errno 98] Address already in use 2021-02-18 01:04:02,714 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1110] ns_1@172.23.106.209: Copying cbcollect ZIP file to Client 2021-02-18 01:04:03,684 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1131] 172.23.106.209 node cb collect zip copied on client : True 2021-02-18 01:04:03,686 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1110] ns_1@172.23.106.225: Copying cbcollect ZIP file to Client 2021-02-18 01:04:04,526 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1131] 172.23.106.225 node cb collect zip copied on client : True 2021-02-18 01:04:04,526 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1110] ns_1@172.23.106.232: Copying cbcollect ZIP file to Client 2021-02-18 01:04:05,371 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1131] 172.23.106.232 node cb collect zip copied on client : True 2021-02-18 01:04:05,371 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1110] ns_1@172.23.106.239: Copying cbcollect ZIP file to Client 2021-02-18 01:04:06,138 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1131] 172.23.106.239 node cb collect zip copied on client : True 2021-02-18 01:04:06,138 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1110] ns_1@172.23.106.246: Copying cbcollect ZIP file to Client 2021-02-18 01:04:06,805 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1131] 172.23.106.246 node cb collect zip copied on client : True 2021-02-18 01:04:06,819 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:487] Skipping get_trace !! 2021-02-18 01:04:16,223 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:751] Socket error while connecting to http://172.23.106.209:8091/pools/default/rebalanceProgress error [Errno 98] Address already in use 2021-02-18 01:04:21,259 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1448] Rebalance done. Taken 13.0800001621 seconds to complete 2021-02-18 01:04:21,267 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-02-18 01:04:26,361 | test | WARNING | MainProcess | MainThread | [basetestcase:tearDown:449] CRASH | CRITICAL | WARN messages found in cb_logs