2021-05-10 20:17:49,630 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-10 20:17:50,582 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-10 20:17:51,144 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:320] Initializing cluster 2021-05-10 20:17:56,719 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:17:58,739 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:00,763 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:02,782 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:04,805 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:06,831 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:08,852 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:10,878 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:12,903 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:20,255 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:22,279 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:24,301 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:26,323 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:28,351 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:30,375 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:32,401 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:34,421 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:36,441 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:43,947 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:45,969 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:47,990 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:50,013 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:52,039 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:54,061 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:56,085 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:18:58,105 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:00,128 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:02,148 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:10,743 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:12,766 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:14,789 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:16,815 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:18,836 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:20,859 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:22,884 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:24,903 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:26,918 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:28,940 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:30,963 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:32,983 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:35,003 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:37,026 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:41,930 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:43,951 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:45,973 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:47,996 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:50,019 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:52,042 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:57,424 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:19:59,447 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:20:01,470 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:20:03,492 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:20:05,515 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:20:07,540 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 20:20:24,900 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-10 20:20:39,631 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206, 172.23.120.201 initialized 2021-05-10 20:20:39,657 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-10 20:20:39,701 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-10 20:21:19,243 | test | INFO | MainProcess | pool-5-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5133-enterprise | 3.35251438579 | Cluster node | | 172.23.98.195 | ['fts'] | | | <--- IN --- | | 172.23.121.10 | ['kv'] | | | <--- IN --- | | 172.23.104.186 | ['kv'] | | | <--- IN --- | | 172.23.120.206 | ['kv'] | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-10 20:21:19,280 | test | INFO | MainProcess | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-10 20:21:29,368 | test | INFO | MainProcess | pool-5-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-10 20:21:29,408 | test | INFO | MainProcess | pool-5-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 10.1659998894 sec 2021-05-10 20:21:29,760 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-10 20:22:08,581 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-10 20:22:08,936 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 838860800 | 696424504 | 151703296 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 838860800 | 260236976 | 136 | | default | couchbase | 3 | none | 0 | 0 | 8388608000 | 680247536 | 134598484 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-10 20:22:08,937 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-10 20:22:54,910 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-10 20:22:54,923 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-10 20:23:17,028 | test | INFO | MainProcess | pool-5-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................*..................................................................**......*........X | 132.387 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-10 20:23:17,316 | test | INFO | MainProcess | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.................................................................................................***...X | 130.454 | | 117.588 | .........................................................................................X | 117.588 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-10 20:23:17,566 | test | INFO | MainProcess | pool-5-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.......*....................................**..................*.......................................X | 24290.985 | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-10 20:23:17,832 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 92.4720244151 | 3693387776 | 2514300928 | 0 / 3909087232 | 82387 / 244055 | 7.0.0-5133-enterprise | | 172.23.98.195 | fts | 3.16582914573 | 3693387776 | 3123052544 | 0 / 3909087232 | 0 / 0 | 7.0.0-5133-enterprise | | 172.23.104.186 | kv | 96.5534591195 | 6078930944 | 4767055872 | 0 / 3758092288 | 89463 / 263770 | 7.0.0-5133-enterprise | | 172.23.120.206 | kv | 73.9715189873 | 3691520000 | 2622566400 | 0 / 0 | 91042 / 272865 | 7.0.0-5133-enterprise | | 172.23.121.10 | kv | 99.8 | 4201836544 | 3046752256 | 0 / 3758092288 | 90780 / 115398 | 7.0.0-5133-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-10 20:23:38,359 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-10 20:25:08,065 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 838860800 | 205419072 | 300403800 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 838860800 | 316814512 | 136 | | default | couchbase | 3 | none | 0 | 500000 | 8388608000 | 755490896 | 590212284 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-10 20:25:08,066 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-10 20:25:16,298 | test | INFO | MainProcess | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-10 20:25:18,355 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-10 20:25:18,355 | test | INFO | MainProcess | MainThread | [collections_rebalance:setUp:78] Creating metakv entries start 2021-05-10 20:27:37,470 | test | INFO | MainProcess | MainThread | [collections_rebalance:setUp:80] Creating metakv entries end 2021-05-10 20:27:37,470 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before hard_failover_rebalance_out 2021-05-10 20:27:37,474 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-10 20:27:44,384 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-10 20:27:44,589 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_rebalance_out 2021-05-10 20:27:44,591 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:632] failing over nodes [ip:172.23.104.186 port:8091 ssh_username:root, ip:172.23.120.206 port:8091 ssh_username:root] 2021-05-10 20:27:54,937 | test | ERROR | MainProcess | pool-5-thread-9 | [rest_client:_http_request:748] POST http://172.23.98.196:8091/controller/failOver body: otpNode=ns_1%40172.23.104.186&allowUnsafe=false headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password 2021-05-10 20:27:54,940 | test | ERROR | MainProcess | pool-5-thread-9 | [rest_client:fail_over:1276] ns_1@172.23.104.186 - Failover error: ["Unexpected server error, request logged."] 2021-05-10 20:27:55,016 | test | INFO | MainProcess | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-10 20:27:59,956 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-10 20:28:02,733 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-10 20:28:02,734 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-10 20:28:27,127 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-10 20:28:27,128 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-10 20:28:46,312 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-10 20:28:46,313 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-10 20:29:06,059 | test | INFO | MainProcess | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket2:_default:vl-29-815000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.120.206:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-2 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.104.186:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-10 20:29:08,315 | test | INFO | MainProcess | pool-5-thread-22 | [table_view:display:72] Keys failed in bucket2:_default:J7JKmLpb4E-29-816000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.120.206:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-2 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.104.186:11210 | reason:TIMEOUT | retryAttempts:63 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.121.10:11210 | reason:TIMEOUT | retryAttempts:61 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-10 20:29:08,556 | test | INFO | MainProcess | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket2:_default:Iy-29-815000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.120.206:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-2 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.104.186:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.121.10:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-10 20:29:08,559 | test | INFO | MainProcess | pool-5-thread-8 | [table_view:display:72] Keys failed in bucket2:_default:EF38Wl831-29-814000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.120.206:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-2 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.104.186:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.AmbiguousTimeoutException | RemoveRequest, Reason: TIMEOUT | 172.23.121.10:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-10 20:29:11,890 | infra | ERROR | MainProcess | pool-5-thread-29 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.98.196:8091/pools/default/buckets/bucket2/stats?zoom=minute. Error [Errno -1] Unmapped exception: java.lang.InterruptedException 2021-05-10 20:29:16,887 | infra | ERROR | MainProcess | MainThread | [task_manager:shutdown:68] Pool did not terminate 2021-05-10 20:29:17,121 | test | INFO | MainProcess | pool-5-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *...........*..............................................................................*...........*X | 3196.767 | | 125.580 | ..*X | 125.580 | | 72.380 | .X | 72.380 | | 66.686 | .**X | 66.686 | | 52.686 | X | 52.686 | | 12.903 | *X | 12.903 | | 12.803 | X | 12.803 | | 12.684 | ******X | 12.917 | | 12.597 | *X | 12.597 | | 12.389 | ****X | 22.507 | +---------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-10 20:29:18,888 | infra | ERROR | MainProcess | MainThread | [task_manager:shutdown:68] Pool did not terminate 2021-05-10 20:29:41,355 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-10 20:29:41,502 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1094] Running cbcollect on node ns_1@172.23.104.186,ns_1@172.23.120.206,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 2021-05-10 20:29:51,555 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1097] ns_1@172.23.104.186,ns_1@172.23.120.206,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 - cbcollect status: True 2021-05-10 20:29:51,556 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1102] Polling active_tasks to check cbcollect status 2021-05-10 20:36:13,612 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.104.186: Copying cbcollect ZIP file to Client 2021-05-10 20:36:16,861 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.120.206: Copying cbcollect ZIP file to Client 2021-05-10 20:36:19,839 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.121.10: Copying cbcollect ZIP file to Client 2021-05-10 20:36:24,232 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.98.195: Copying cbcollect ZIP file to Client 2021-05-10 20:36:27,937 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.98.196: Copying cbcollect ZIP file to Client 2021-05-10 20:36:32,046 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:491] Skipping get_trace !! 2021-05-10 20:36:58,882 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 20.3499999046 seconds to complete 2021-05-10 20:36:58,911 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-10 20:37:04,086 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused