2021-01-27 22:26:35,076 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:552] ========= BaseTestCase setup started for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2021-01-27 22:26:35,572 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:552] ========= BaseTestCase setup started for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2021-01-27 22:26:35,842 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:328] Initializing cluster 2021-01-27 22:26:58,686 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:597] RAM quota was defined less than 100 MB: 2021-01-27 22:27:14,618 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:350] Couchbase Cluster: default, Nodes: 172.23.105.52, 172.23.105.53, 172.23.105.59, 172.23.105.64, 172.23.105.79 initialized 2021-01-27 22:27:14,631 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:552] ========= BaseTestCase setup finished for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2021-01-27 22:27:14,648 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:552] ========= CollectionBase setup started for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2021-01-27 22:27:48,578 | test | INFO | MainProcess | pool-6-thread-6 | [table_view:display:72] Rebalance Overview +---------------+----------+--------------+ | Nodes | Services | Status | +---------------+----------+--------------+ | 172.23.105.52 | kv | Cluster node | | 172.23.105.53 | None | <--- IN --- | | 172.23.105.59 | None | <--- IN --- | | 172.23.105.64 | None | <--- IN --- | | 172.23.105.79 | None | <--- IN --- | +---------------+----------+--------------+ 2021-01-27 22:27:48,586 | test | INFO | MainProcess | pool-6-thread-6 | [task:check:322] Rebalance - status: running, progress: 0 2021-01-27 22:27:58,622 | test | INFO | MainProcess | pool-6-thread-6 | [task:check:322] Rebalance - status: none, progress: 100 2021-01-27 22:27:58,638 | test | INFO | MainProcess | pool-6-thread-6 | [task:check:379] Rebalance completed with progress: 100% in 10.0620000362 sec 2021-01-27 22:27:58,822 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1773] Creating required buckets from template 2021-01-27 22:28:27,170 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2242] Waiting for all collections to be created 2021-01-27 22:28:27,346 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 276199992 | 212257642 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 1048576000 | 179180888 | 170 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 282456152 | 178286060 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-01-27 22:28:27,346 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:143] Creating required SDK clients for client_pool 2021-01-27 22:28:44,187 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4613] Performing scope/collection specific operations 2021-01-27 22:28:44,194 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4738] Done Performing scope/collection specific operations 2021-01-27 22:29:13,838 | test | INFO | MainProcess | pool-6-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................**................................................................X | 231.415 | | 166.013 | ......................................................................*X | 166.013 | | 40.711 | ................X | 40.711 | | 35.908 | ..............X | 35.908 | | 35.612 | ..............X | 35.612 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-01-27 22:29:15,914 | test | INFO | MainProcess | pool-6-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...........................................*...............................*..........................X | 185.124 | | 147.923 | ..............................................................................*X | 147.923 | | 43.014 | ......................*...*X | 48.718 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-01-27 22:29:16,023 | test | INFO | MainProcess | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **........*..............*...............................................*...........**.................*...*X | 19775.941 | +-------+----------------------------------------------------------------------------------------------------------------+-----------+ 2021-01-27 22:29:16,184 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +---------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +---------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.105.52 | kv | 90.1940755873 | 4201840640 | 2653777920 | 35651584 / 3758092288 | 79998 / 156961 | 7.0.0-4325-enterprise | | 172.23.105.59 | kv | 76.0311284047 | 4339109888 | 2774638592 | 0 / 3758092288 | 70207 / 141125 | 7.0.0-4325-enterprise | | 172.23.105.64 | kv | 97.5106864471 | 4340109312 | 2733023232 | 0 / 3758092288 | 101600 / 201400 | 7.0.0-4325-enterprise | | 172.23.105.79 | kv | 96.9513731418 | 4219441152 | 2567389184 | 0 / 3758092288 | 87116 / 175874 | 7.0.0-4325-enterprise | | 172.23.105.53 | kv | 90.3390262554 | 4201840640 | 2611232768 | 0 / 3758092288 | 71627 / 144986 | 7.0.0-4325-enterprise | +---------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2021-01-27 22:29:17,457 | test | INFO | MainProcess | pool-6-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:19,144 | test | INFO | MainProcess | pool-6-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:20,316 | test | INFO | MainProcess | pool-6-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:22,716 | test | INFO | MainProcess | pool-6-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:23,125 | test | INFO | MainProcess | pool-6-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:24,875 | test | INFO | MainProcess | pool-6-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:25,188 | test | INFO | MainProcess | pool-6-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:25,881 | test | INFO | MainProcess | pool-6-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:28,848 | test | INFO | MainProcess | pool-6-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:29,105 | test | INFO | MainProcess | pool-6-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:30,615 | test | INFO | MainProcess | pool-6-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:31,446 | test | INFO | MainProcess | pool-6-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:31,667 | test | INFO | MainProcess | pool-6-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:34,630 | test | INFO | MainProcess | pool-6-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:35,441 | test | INFO | MainProcess | pool-6-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:36,279 | test | INFO | MainProcess | pool-6-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:37,022 | test | INFO | MainProcess | pool-6-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:38,164 | test | INFO | MainProcess | pool-6-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:40,322 | test | INFO | MainProcess | pool-6-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:41,857 | test | INFO | MainProcess | pool-6-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:41,957 | test | INFO | MainProcess | pool-6-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:42,598 | test | INFO | MainProcess | pool-6-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:44,650 | test | INFO | MainProcess | pool-6-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:48,239 | test | INFO | MainProcess | pool-6-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:51,088 | test | INFO | MainProcess | pool-6-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:29:56,088 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4376] Validating collection stats and item counts 2021-01-27 22:31:12,081 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 166409032 | 272972898 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 1048576000 | 245390712 | 170 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 532025560 | 414479662 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-01-27 22:31:12,082 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:552] ========= CollectionBase setup complete for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2021-01-27 22:31:17,335 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:710] Doing collection data load during hard_failover_recovery 2021-01-27 22:31:17,335 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:242] Starting rebalance operation of type : hard_failover_recovery 2021-01-27 22:31:27,210 | test | INFO | MainProcess | pool-6-thread-27 | [rest_client:monitorRebalance:1438] Rebalance done. Taken 8.0529999733 seconds to complete 2021-01-27 22:31:27,213 | test | INFO | MainProcess | pool-6-thread-27 | [common_lib:sleep:22] Sleep 8.0529999733 seconds. Reason: Wait after rebalance complete 2021-01-27 22:33:35,296 | test | INFO | MainProcess | MainThread | [collections_rebalance:wait_for_failover_or_assert:213] 1 nodes failed over as expected in 0.0280001163483 seconds 2021-01-27 22:33:35,296 | test | INFO | MainProcess | MainThread | [collections_rebalance:data_load_after_failover:192] Starting a sync data load after failover 2021-01-27 22:33:35,296 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4613] Performing scope/collection specific operations 2021-01-27 22:33:40,217 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4738] Done Performing scope/collection specific operations 2021-01-27 22:33:56,061 | test | INFO | MainProcess | pool-6-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................**X | 1437.637 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-01-27 22:33:56,522 | test | INFO | MainProcess | pool-6-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................**X | 744.516 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-01-27 22:33:57,578 | test | INFO | MainProcess | pool-6-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................***X | 6886.098 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-01-27 22:33:57,638 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1710] 172.23.105.79 - Node not part of cluster inactiveFailed 2021-01-27 22:33:57,661 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1710] 172.23.105.79 - Node not part of cluster inactiveFailed 2021-01-27 22:33:58,888 | test | INFO | MainProcess | pool-6-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:00,648 | test | INFO | MainProcess | pool-6-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:01,940 | test | INFO | MainProcess | pool-6-thread-15 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:04,124 | test | INFO | MainProcess | pool-6-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:04,563 | test | INFO | MainProcess | pool-6-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:06,341 | test | INFO | MainProcess | pool-6-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:07,482 | test | INFO | MainProcess | pool-6-thread-15 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:10,221 | test | INFO | MainProcess | pool-6-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:10,476 | test | INFO | MainProcess | pool-6-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:11,999 | test | INFO | MainProcess | pool-6-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:13,095 | test | INFO | MainProcess | pool-6-thread-15 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:15,881 | test | INFO | MainProcess | pool-6-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:16,861 | test | INFO | MainProcess | pool-6-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:17,673 | test | INFO | MainProcess | pool-6-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:18,631 | test | INFO | MainProcess | pool-6-thread-15 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:21,569 | test | INFO | MainProcess | pool-6-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:23,355 | test | INFO | MainProcess | pool-6-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:23,642 | test | INFO | MainProcess | pool-6-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:24,161 | test | INFO | MainProcess | pool-6-thread-15 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:29,325 | test | INFO | MainProcess | pool-6-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-27 22:34:34,398 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4613] Performing scope/collection specific operations 2021-01-27 22:34:34,517 | test | INFO | MainProcess | pool-6-thread-10 | [table_view:display:72] Rebalance Overview +---------------+----------+--------------+ | Nodes | Services | Status | +---------------+----------+--------------+ | 172.23.105.52 | kv | Cluster node | | 172.23.105.59 | kv | Cluster node | | 172.23.105.64 | kv | Cluster node | | 172.23.105.79 | kv | Cluster node | | 172.23.105.53 | kv | Cluster node | +---------------+----------+--------------+ 2021-01-27 22:34:34,565 | test | INFO | MainProcess | pool-6-thread-10 | [task:check:322] Rebalance - status: running, progress: 0 2021-01-27 22:34:39,414 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4738] Done Performing scope/collection specific operations 2021-01-27 22:34:39,658 | test | INFO | MainProcess | pool-6-thread-10 | [task:check:322] Rebalance - status: running, progress: 0.291011591103 2021-01-27 22:34:40,868 | test | INFO | MainProcess | pool-6-thread-11 | [table_view:display:72] Keys failed in default:eknRs-38-382000:lLPlg6BABPnXUUfoEj-38-470000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:40,868 | test | INFO | MainProcess | pool-6-thread-11 | [table_view:display:72] Keys failed in default:eknRs-38-382000:lLPlg6BABPnXUUfoEj-38-470000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-162 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:44,553 | test | INFO | MainProcess | pool-6-thread-27 | [table_view:display:72] Keys failed in default:R-58-864000:Nb_J-58-865000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1019 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:44,789 | test | INFO | MainProcess | pool-6-thread-10 | [task:check:322] Rebalance - status: running, progress: 3.27214598089 2021-01-27 22:34:46,244 | test | INFO | MainProcess | pool-6-thread-16 | [table_view:display:72] Keys failed in bucket1:8DHpiU-58-857000:E_FU84laFqqujNU-58-857000 +--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:49,835 | test | ERROR | MainProcess | pool-6-thread-10 | [rest_client:_rebalance_status_and_progress:1484] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2021-01-27 22:34:49,875 | test | INFO | MainProcess | pool-6-thread-8 | [table_view:display:72] Keys failed in default:R-58-864000:J%-58-865000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-655 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-541 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:49,888 | test | INFO | MainProcess | pool-6-thread-10 | [rest_client:print_UI_logs:2593] Latest logs from UI on 172.23.105.52: 2021-01-27 22:34:49,888 | test | ERROR | MainProcess | pool-6-thread-10 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.59', u'tstamp': 1611815687006L, u'shortText': u'message', u'serverTime': u'2021-01-27T22:34:47.006Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.105.59\' in 1 seconds.'} 2021-01-27 22:34:49,888 | test | ERROR | MainProcess | pool-6-thread-10 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.59', u'tstamp': 1611815686505L, u'shortText': u'message', u'serverTime': u'2021-01-27T22:34:46.505Z', u'text': u'Bucket "bucket1" loaded on node \'ns_1@172.23.105.59\' in 1 seconds.'} 2021-01-27 22:34:49,888 | test | ERROR | MainProcess | pool-6-thread-10 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.59', u'tstamp': 1611815685212L, u'shortText': u'message', u'serverTime': u'2021-01-27T22:34:45.212Z', u'text': u'Bucket "bucket2" loaded on node \'ns_1@172.23.105.59\' in 0 seconds.'} 2021-01-27 22:34:49,888 | test | ERROR | MainProcess | pool-6-thread-10 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.52', u'tstamp': 1611815684369L, u'shortText': u'message', u'serverTime': u'2021-01-27T22:34:44.369Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.19228.3>,\n {{{{{child_interrupted,\n {\'EXIT\',<16930.25550.1>,socket_closed}},\n [{dcp_replicator,spawn_and_wait,1,\n [{file,"src/dcp_replicator.erl"},\n {line,265}]},\n {dcp_replicator,handle_call,3,\n [{file,"src/dcp_replicator.erl"},\n {line,127}]},\n {gen_server,try_handle_call,4,\n [{file,"gen_server.erl"},{line,661}]},\n {gen_server,handle_msg,6,\n [{file,"gen_server.erl"},{line,690}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,249}]}]},\n {gen_server,call,\n [<16930.25545.1>,get_partitions,\n infinity]}},\n {gen_server,call,\n [\'dcp_replication_manager-default\',\n {get_replicator_pid,847},\n infinity]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.105.79\'},\n {if_rebalance,<0.13642.3>,\n {update_vbucket_state,1000,active,\n undefined,undefined,\n [[\'ns_1@172.23.105.79\',\n \'ns_1@172.23.105.64\',\n \'ns_1@172.23.105.53\']]}},\n infinity]}}}}}.\nRebalance Operation Id = 1b7d5afd9cb8f5023b863eedd0e2a941'} 2021-01-27 22:34:49,890 | test | ERROR | MainProcess | pool-6-thread-10 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.105.52', u'tstamp': 1611815684211L, u'shortText': u'message', u'serverTime': u'2021-01-27T22:34:44.211Z', u'text': u'Worker <0.18637.3> (for action {move,{1000,\n [\'ns_1@172.23.105.64\',\n \'ns_1@172.23.105.53\',undefined],\n [\'ns_1@172.23.105.79\',\n \'ns_1@172.23.105.64\',\n \'ns_1@172.23.105.53\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.19228.3>,\n {{{{{child_interrupted,\n {\'EXIT\',\n <16930.25550.1>,\n socket_closed}},\n [{dcp_replicator,\n spawn_and_wait,\n 1,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 265}]},\n {dcp_replicator,\n handle_call,\n 3,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 127}]},\n {gen_server,\n try_handle_call,\n 4,\n [{file,\n "gen_server.erl"},\n {line,\n 661}]},\n {gen_server,\n handle_msg,\n 6,\n [{file,\n "gen_server.erl"},\n {line,\n 690}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 249}]}]},\n {gen_server,\n call,\n [<16930.25545.1>,\n get_partitions,\n infinity]}},\n {gen_server,\n call,\n [\'dcp_replication_manager-default\',\n {get_replicator_pid,\n 847},\n infinity]}},\n {gen_server,\n call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.105.79\'},\n {if_rebalance,\n <0.13642.3>,\n {update_vbucket_state,\n 1000,\n active,\n undefined,\n undefined,\n [[\'ns_1@172.23.105.79\',\n \'ns_1@172.23.105.64\',\n \'ns_1@172.23.105.53\']]}},\n infinity]}}}}'} 2021-01-27 22:34:49,892 | test | ERROR | MainProcess | pool-6-thread-10 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.105.59', u'tstamp': 1611815684194L, u'shortText': u'message', u'serverTime': u'2021-01-27T22:34:44.194Z', u'text': u"Service 'memcached' exited with status 139. Restarting. Messages:\n2021-01-27T22:34:44.125021-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fb90ce7b000+0x188fbb]\n2021-01-27T22:34:44.125027-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fb90ce7b000+0x16dc13]\n2021-01-27T22:34:44.125032-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fb90ce7b000+0x167c92]\n2021-01-27T22:34:44.125041-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fb90ce7b000+0x2e71d6]\n2021-01-27T22:34:44.125048-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fb90ce7b000+0x2cf6ca]\n2021-01-27T22:34:44.125055-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fb90ce7b000+0x2ea2f9]\n2021-01-27T22:34:44.125061-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fb90ce7b000+0x1660d3]\n2021-01-27T22:34:44.125094-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fb909551000+0xb9dcf]\n2021-01-27T22:34:44.125099-08:00 CRITICAL /lib64/libpthread.so.0() [0x7fb908e1c000+0x7dc5]\n2021-01-27T22:34:44.125128-08:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fb908a5b000+0xf776d]"} 2021-01-27 22:34:49,892 | test | ERROR | MainProcess | pool-6-thread-10 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.59', u'tstamp': 1611815684177L, u'shortText': u'message', u'serverTime': u'2021-01-27T22:34:44.177Z', u'text': u"Control connection to memcached on 'ns_1@172.23.105.59' disconnected. Check logs for details."} 2021-01-27 22:34:49,892 | test | ERROR | MainProcess | pool-6-thread-10 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.105.52', u'tstamp': 1611815678054L, u'shortText': u'message', u'serverTime': u'2021-01-27T22:34:38.054Z', u'text': u'Bucket "default" rebalance does not seem to be swap rebalance'} 2021-01-27 22:34:49,894 | test | ERROR | MainProcess | pool-6-thread-10 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.79', u'tstamp': 1611815676043L, u'shortText': u'message', u'serverTime': u'2021-01-27T22:34:36.043Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.105.79\' in 0 seconds.'} 2021-01-27 22:34:49,894 | test | ERROR | MainProcess | pool-6-thread-10 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.105.52', u'tstamp': 1611815675839L, u'shortText': u'message', u'serverTime': u'2021-01-27T22:34:35.839Z', u'text': u'Started rebalancing bucket default'} 2021-01-27 22:34:49,894 | test | ERROR | MainProcess | pool-6-thread-10 | [task:call:238] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2021-01-27 22:34:50,957 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:90] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-01-27 22:34:50,959 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:93] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-01-27 22:34:52,326 | test | INFO | MainProcess | pool-6-thread-28 | [table_view:display:72] Keys failed in bucket1:UniB2-37-794000:BAk0sJhW8pd5yCdXhX_c-37-909000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4819 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4977 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4898 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4810 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4932 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4855 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4937 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4859 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4893 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4852 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4896 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4972 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4928 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4968 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4924 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4880 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4885 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4841 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4961 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4955 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4835 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4832 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4877 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4910 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4915 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4830 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4875 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4950 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4908 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4906 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4823 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4989 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4868 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4821 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4866 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4943 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4948 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4946 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4903 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4826 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4980 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4863 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4864 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:52,328 | test | INFO | MainProcess | pool-6-thread-28 | [table_view:display:72] Keys failed in bucket1:UniB2-37-794000:BAk0sJhW8pd5yCdXhX_c-37-909000 +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4939 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4819 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4977 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4898 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4810 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4932 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4855 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4937 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4817 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4979 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4859 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4893 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4852 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4896 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4972 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4928 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4924 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4968 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4804 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4880 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4841 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4885 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4961 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4955 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4835 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4832 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4877 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4910 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4915 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4870 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4830 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4875 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4950 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4994 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4908 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4906 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4823 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4868 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4989 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4821 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4987 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4866 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4943 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4948 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4946 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4903 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4826 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4980 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4863 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4864 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:52,417 | test | INFO | MainProcess | pool-6-thread-13 | [table_view:display:72] Keys failed in bucket1:PJN-aOkuTeM-38-133000:YojpeQdIHVToK9-38-265000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4658 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4734 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4695 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4654 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4611 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4773 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4697 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4647 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4769 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4722 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4602 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4729 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4683 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4684 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4681 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4688 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4686 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4713 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4636 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4714 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4711 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4799 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4756 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4718 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4716 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4673 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4792 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4676 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4753 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4754 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4633 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4751 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4669 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4625 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4700 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4707 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4705 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4660 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4781 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4742 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4665 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4740 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4620 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:52,420 | test | INFO | MainProcess | pool-6-thread-13 | [table_view:display:72] Keys failed in bucket1:PJN-aOkuTeM-38-133000:YojpeQdIHVToK9-38-265000 +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4658 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4734 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4695 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4731 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4654 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4611 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4773 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4697 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4774 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4690 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4609 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4647 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4769 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4722 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4767 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4602 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4729 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4649 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4683 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4684 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4681 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4688 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4686 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4713 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4636 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4714 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4799 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4711 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4756 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4718 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4638 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4716 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4673 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4792 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4676 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4753 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4633 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4754 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4751 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4669 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4625 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4700 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4745 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4707 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4705 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4660 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4781 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4742 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4665 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4740 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4620 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:52,426 | test | INFO | MainProcess | pool-6-thread-30 | [table_view:display:72] Keys failed in bucket1:PJN-aOkuTeM-38-133000:wxZbmnXyL%uOekUCq-38-265000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4658 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4734 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4695 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4654 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4611 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4773 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4697 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4647 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4769 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4722 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4602 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4729 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4683 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4684 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4681 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4688 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4686 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4713 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4636 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4714 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4799 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4711 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4756 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4718 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4716 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4673 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4792 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4676 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4753 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4633 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4754 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4751 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4669 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4625 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4700 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4707 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4705 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4781 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4660 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4742 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4665 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4740 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4620 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:52,428 | test | INFO | MainProcess | pool-6-thread-30 | [table_view:display:72] Keys failed in bucket1:PJN-aOkuTeM-38-133000:wxZbmnXyL%uOekUCq-38-265000 +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4658 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4734 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4695 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4731 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4654 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4611 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4773 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4697 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4774 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4690 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4609 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4647 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4769 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4722 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4602 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4767 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4729 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4649 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4683 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4684 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4681 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4688 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4686 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4713 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4636 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4714 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4799 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4711 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4756 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4718 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4638 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4716 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4673 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4792 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4676 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4753 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4754 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4633 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4751 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4669 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4625 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4700 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4745 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4707 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4705 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4781 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4660 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4742 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4665 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4740 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4620 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:53,496 | test | INFO | MainProcess | pool-6-thread-28 | [table_view:display:72] Keys failed in bucket2:scope1:ZTj-58-846000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:54,229 | test | INFO | MainProcess | pool-6-thread-16 | [table_view:display:72] Keys failed in bucket2:4Vx_O-aoD9Z%0vEz1X6a-58-848000:fAN0rGax-58-851000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:54,789 | test | INFO | MainProcess | pool-6-thread-12 | [table_view:display:72] Keys failed in default:_default:DuAEO0beOi-58-859000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1230 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:54,792 | test | INFO | MainProcess | pool-6-thread-4 | [table_view:display:72] Keys failed in default:_default:X-58-860000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1230 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:54,792 | test | INFO | MainProcess | pool-6-thread-23 | [table_view:display:72] Keys failed in default:R-58-864000:hXibo-58-865000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1230 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:54,793 | test | INFO | MainProcess | pool-6-thread-25 | [table_view:display:72] Keys failed in default:R-58-864000:J%-58-865000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1230 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:54,798 | test | INFO | MainProcess | pool-6-thread-14 | [table_view:display:72] Keys failed in default:scope1:Z-58-861000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1230 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:54,799 | test | INFO | MainProcess | pool-6-thread-27 | [table_view:display:72] Keys failed in default:R-58-864000:Nb_J-58-865000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1230 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:54,802 | test | INFO | MainProcess | pool-6-thread-29 | [table_view:display:72] Keys failed in default:scope2:X-JtbIuXtU-58-862000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1230 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:54,999 | test | INFO | MainProcess | pool-6-thread-21 | [table_view:display:72] Keys failed in bucket2:G9Ho3X-36-904000:8nl3oMkK5A5z-37-4000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4460 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4516 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4414 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4513 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4436 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4558 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4518 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4571 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4495 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4451 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4478 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4599 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4534 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4556 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4597 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4553 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4476 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4433 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4490 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4449 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4505 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4527 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4425 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4569 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4409 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4407 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4529 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4584 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4562 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4540 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4442 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4486 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4420 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4483 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4522 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4500 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4567 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4545 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4465 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4488 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:55,000 | test | INFO | MainProcess | pool-6-thread-21 | [table_view:display:72] Keys failed in bucket2:G9Ho3X-36-904000:8nl3oMkK5A5z-37-4000 +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4460 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4516 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4513 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4414 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4436 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4558 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4518 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4571 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4451 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4495 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4473 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4599 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4478 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4556 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4534 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4553 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4476 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4597 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4433 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4490 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4449 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4505 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4527 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4425 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4569 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4409 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4407 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4529 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4562 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4584 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4540 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4420 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4442 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4486 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4483 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4522 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4500 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4567 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4545 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4465 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4488 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.105.59:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:55,094 | test | INFO | MainProcess | pool-6-thread-14 | [table_view:display:72] Keys failed in bucket2:scope1:9EubiC-58-847000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:55,211 | test | INFO | MainProcess | pool-6-thread-21 | [table_view:display:72] Keys failed in bucket2:G9Ho3X-36-904000:8nl3oMkK5A5z-37-4000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4636 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4669 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4713 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4625 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4769 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4799 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4756 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4729 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4683 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4673 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4660 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4665 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4676 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4633 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4620 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:55,213 | test | INFO | MainProcess | pool-6-thread-21 | [table_view:display:72] Keys failed in bucket2:G9Ho3X-36-904000:8nl3oMkK5A5z-37-4000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4609 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4713 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4636 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4625 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4669 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4769 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4799 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4756 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4729 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4683 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4673 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4660 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4731 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4676 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4665 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4633 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4774 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4620 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:55,405 | test | INFO | MainProcess | pool-6-thread-16 | [table_view:display:72] Keys failed in bucket2:4Vx_O-aoD9Z%0vEz1X6a-58-848000:VG3sBEEc6nlO-58-852000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:55,492 | test | INFO | MainProcess | pool-6-thread-21 | [table_view:display:72] Keys failed in bucket2:G9Ho3X-36-904000:8nl3oMkK5A5z-37-4000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4819 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4977 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4955 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4989 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4835 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4943 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4910 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4932 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4948 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4915 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4946 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4859 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4903 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4893 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4880 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4885 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4852 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4950 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:55,493 | test | INFO | MainProcess | pool-6-thread-21 | [table_view:display:72] Keys failed in bucket2:G9Ho3X-36-904000:8nl3oMkK5A5z-37-4000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4819 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4955 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4977 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4989 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4835 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4987 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4932 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4910 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4943 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4915 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4948 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4817 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4946 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4903 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4804 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4859 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4870 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4893 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4880 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4852 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4885 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4994 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4950 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:34:56,200 | test | INFO | MainProcess | pool-6-thread-23 | [table_view:display:72] Keys failed in bucket2:4Vx_O-aoD9Z%0vEz1X6a-58-848000:dwHAu6AJ-58-852000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-27 22:35:03,000 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:90] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-01-27 22:35:03,000 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:93] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-01-27 22:35:20,098 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:90] Bucket: default, Active Resident ratio(DGM): 100% 2021-01-27 22:35:20,099 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:93] Bucket: default, Replica Resident ratio(DGM): 100% 2021-01-27 22:35:40,911 | infra | ERROR | MainProcess | pool-6-thread-18 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.105.52:8091/pools/default/buckets/bucket1/stats?zoom=minute. Error [Errno -1] Unmapped exception: java.lang.InterruptedException 2021-01-27 22:35:40,911 | infra | ERROR | MainProcess | pool-6-thread-7 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.105.52:8091/pools/default/buckets/bucket2/stats?zoom=minute. Error [Errno -1] Unmapped exception: java.lang.InterruptedException 2021-01-27 22:35:45,910 | infra | ERROR | MainProcess | MainThread | [task_manager:shutdown:68] Pool did not terminate 2021-01-27 22:35:46,125 | test | INFO | MainProcess | pool-6-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...........................*.........................................................................**X | 3355.290 | | 2546.390 | ..........................................................................X | 2546.390 | | 1121.833 | ................................X | 1121.833 | | 225.533 | .....*X | 225.533 | | 152.908 | ...X | 152.908 | | 110.475 | ..X | 110.475 | | 106.075 | ..**X | 106.075 | | 57.253 | X | 57.253 | | 53.722 | **X | 53.722 | | 53.712 | X | 53.712 | | 53.589 | ***X | 53.589 | | 53.531 | X | 53.531 | | 53.300 | *X | 53.300 | +----------+------------------------------------------------------------------------------------------------------------+----------+ 2021-01-27 22:35:46,155 | test | INFO | MainProcess | pool-6-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...........................................................................**.........................**X | 2160.066 | | 841.162 | .....................................X | 841.162 | | 473.062 | ....................****..................................................*........*X | 1709.262 | | 1233.800 | ........................................................X | 1233.800 | | 321.800 | .............X | 321.800 | | 0.000 | *********X | 0.000 | +----------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-01-27 22:35:47,911 | infra | ERROR | MainProcess | MainThread | [task_manager:shutdown:68] Pool did not terminate 2021-01-27 22:35:51,494 | test | WARNING | MainProcess | MainThread | [basetestcase:check_coredump_exist:756] 172.23.105.52: No 'sanitizers.log.*' files found 2021-01-27 22:35:54,084 | test | WARNING | MainProcess | MainThread | [basetestcase:check_coredump_exist:756] 172.23.105.53: No 'sanitizers.log.*' files found 2021-01-27 22:35:54,453 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:728] 172.23.105.59: 1 core dump seen 2021-01-27 22:35:54,746 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:729] 172.23.105.59: Stack Trace of first crash - 3b61ddfd-4093-40f6-2881ddbc-b152fe13.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007fb90cef3d66 in std::_Hashtable, std::pair const, index_entry>, MemoryTrackingAllocator const, index_entry> >, std::__detail::_Select1st, std::equal_to, std::hash >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::~_Hashtable() () from /opt/couchbase/bin/../lib/libep.so #0 0x00007fb90cef3d66 in std::_Hashtable, std::pair const, index_entry>, MemoryTrackingAllocator const, index_entry> >, std::__detail::_Select1st, std::equal_to, std::hash >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::~_Hashtable() () from /opt/couchbase/bin/../lib/libep.so No symbol table info available. #1 0x00007fb90cef2b1a in Checkpoint::~Checkpoint() () from /opt/couchbase/bin/../lib/libep.so No symbol table info available. #2 0x00007fb90cefbfd8 in CheckpointManager::removeClosedUnrefCheckpoints(VBucket&, bool&, unsigned long) () from /opt/couchbase/bin/../lib/libep.so No symbol table info available. #3 0x00007fb90cf06382 in CheckpointVisitor::visitBucket(std::shared_ptr const&) () from /opt/couchbase/bin/../lib/libep.so No symbol table info available. #4 0x00007fb90d003fbb in VBCBAdaptor::run() () from /opt/couchbase/bin/../lib/libep.so No symbol table info available. #5 0x00007fb90cfe8c13 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so No symbol table info available. #6 0x00007fb90cfe2c92 in void folly::detail::function::FunctionTraits::callSmall(folly::detail::function::Data&) () from /opt/couchbase/bin/../lib/libep.so No symbol table info available. #7 0x00007fb90d1621d6 in folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) () from /opt/couchbase/bin/../lib/libep.so No symbol table info available. #8 0x00007fb90d14a6ca in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () from /opt/couchbase/bin/../lib/libep.so No symbol table info available. #9 0x00007fb90d1652f9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () from /opt/couchbase/bin/../lib/libep.so No symbol table info available. #10 0x00007fb90cfe10d3 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () from /opt/couchbase/bin/../lib/libep.so No symbol table info available. #11 0x00007fb90960adcf in execute_native_thread_routine () from /opt/couchbase/bin/../lib/libstdc++.so.6 No symbol table info available. #12 0x00007fb908e23dc5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #13 0x00007fb908b5276d in clone () from /lib64/libc.so.6 No symbol table info available. 2021-01-27 22:35:55,209 | test | WARNING | MainProcess | MainThread | [basetestcase:check_coredump_exist:756] 172.23.105.59: No 'sanitizers.log.*' files found 2021-01-27 22:35:55,948 | test | WARNING | MainProcess | MainThread | [basetestcase:check_coredump_exist:756] 172.23.105.64: No 'sanitizers.log.*' files found 2021-01-27 22:35:57,282 | test | WARNING | MainProcess | MainThread | [basetestcase:check_coredump_exist:756] 172.23.105.79: No 'sanitizers.log.*' files found 2021-01-27 22:35:57,329 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1073] Running cbcollect on node ns_1@172.23.105.52,ns_1@172.23.105.53,ns_1@172.23.105.59,ns_1@172.23.105.64,ns_1@172.23.105.79 2021-01-27 22:36:07,355 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1076] ns_1@172.23.105.52,ns_1@172.23.105.53,ns_1@172.23.105.59,ns_1@172.23.105.64,ns_1@172.23.105.79 - cbcollect status: True 2021-01-27 22:36:07,357 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1081] Polling active_tasks to check cbcollect status 2021-01-27 22:39:37,849 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1104] ns_1@172.23.105.52: Copying cbcollect ZIP file to Client 2021-01-27 22:39:39,108 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] 172.23.105.52 node cb collect zip copied on client : True 2021-01-27 22:39:39,109 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1104] ns_1@172.23.105.53: Copying cbcollect ZIP file to Client 2021-01-27 22:39:40,667 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] 172.23.105.53 node cb collect zip copied on client : True 2021-01-27 22:39:40,667 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1104] ns_1@172.23.105.59: Copying cbcollect ZIP file to Client 2021-01-27 22:39:41,526 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] 172.23.105.59 node cb collect zip copied on client : True 2021-01-27 22:39:41,528 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1104] ns_1@172.23.105.64: Copying cbcollect ZIP file to Client 2021-01-27 22:39:42,539 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] 172.23.105.64 node cb collect zip copied on client : True 2021-01-27 22:39:42,539 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1104] ns_1@172.23.105.79: Copying cbcollect ZIP file to Client 2021-01-27 22:39:43,881 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] 172.23.105.79 node cb collect zip copied on client : True 2021-01-27 22:39:43,894 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:497] Skipping get_trace !! 2021-01-27 22:39:59,638 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1438] Rebalance done. Taken 11.0680000782 seconds to complete 2021-01-27 22:39:59,648 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete