2021-02-02 11:02:35,117 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:552] ========= BaseTestCase setup started for test #12 test_data_load_collections_with_graceful_failover_recovery ========= 2021-02-02 11:02:35,579 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:552] ========= BaseTestCase setup started for test #12 test_data_load_collections_with_graceful_failover_recovery ========= 2021-02-02 11:02:35,766 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:747] GET http://172.23.107.94:8091/pools/default/remoteClusters/ body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password 2021-02-02 11:02:35,904 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:328] Initializing cluster 2021-02-02 11:02:46,421 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:597] RAM quota was defined less than 100 MB: 2021-02-02 11:03:00,730 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:350] Couchbase Cluster: default, Nodes: 172.23.107.94, 172.23.107.95, 172.23.107.97, 172.23.107.98, 172.23.107.99 initialized 2021-02-02 11:03:00,742 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:552] ========= BaseTestCase setup finished for test #12 test_data_load_collections_with_graceful_failover_recovery ========= 2021-02-02 11:03:00,763 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:552] ========= CollectionBase setup started for test #12 test_data_load_collections_with_graceful_failover_recovery ========= 2021-02-02 11:03:09,250 | test | INFO | MainProcess | pool-13-thread-6 | [table_view:display:72] Rebalance Overview +---------------+----------+--------------+ | Nodes | Services | Status | +---------------+----------+--------------+ | 172.23.107.94 | kv | Cluster node | | 172.23.107.95 | None | <--- IN --- | | 172.23.107.97 | None | <--- IN --- | | 172.23.107.98 | None | <--- IN --- | | 172.23.107.99 | None | <--- IN --- | +---------------+----------+--------------+ 2021-02-02 11:03:09,259 | test | INFO | MainProcess | pool-13-thread-6 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-02 11:03:19,295 | test | INFO | MainProcess | pool-13-thread-6 | [task:check:322] Rebalance - status: none, progress: 100 2021-02-02 11:03:19,312 | test | INFO | MainProcess | pool-13-thread-6 | [task:check:379] Rebalance completed with progress: 100% in 10.0610001087 sec 2021-02-02 11:03:19,470 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1778] Creating required buckets from template 2021-02-02 11:05:10,063 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2247] Waiting for all collections to be created 2021-02-02 11:05:10,276 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-----------+-----------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-----------+-----------+------------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 524288000 | 52254346 | 2411128212 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 524288000 | 360716158 | 170 | | default | couchbase | 2 | none | 0 | 0 | 524288000 | 54270115 | 1601317931 | +---------+-----------+----------+------------+-----+-------+-----------+-----------+------------+ 2021-02-02 11:05:10,278 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:143] Creating required SDK clients for client_pool 2021-02-02 11:05:26,359 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4624] Performing scope/collection specific operations 2021-02-02 11:05:26,369 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4749] Done Performing scope/collection specific operations 2021-02-02 11:05:37,250 | test | INFO | MainProcess | pool-13-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *...............................................................................*.....................*X | 346.800 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-02-02 11:05:38,131 | test | INFO | MainProcess | pool-13-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+-------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-------+ | 0.000 | ***....................................................................................................**X | 6.600 | +-------+------------------------------------------------------------------------------------------------------------+-------+ 2021-02-02 11:05:38,864 | test | INFO | MainProcess | pool-13-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *...........................................................................*.........................**X | 295.000 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-02-02 11:05:39,170 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +---------------+----------+-----------------+-------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +---------------+----------+-----------------+-------------+------------+----------------+-------------------+-----------------------+ | 172.23.107.97 | kv | 49.8270339526 | 12294119424 | 8691580928 | 0 / 3758092288 | 712 / 2931 | 7.0.0-4356-enterprise | | 172.23.107.94 | kv | 53.4244824942 | 12294119424 | 8564563968 | 0 / 3758092288 | 1412 / 1861 | 7.0.0-4356-enterprise | | 172.23.107.98 | kv | 52.3706070288 | 12294119424 | 8720912384 | 0 / 3758092288 | 1534 / 2219 | 7.0.0-4356-enterprise | | 172.23.107.95 | kv | 51.0758196721 | 12294119424 | 8593735680 | 0 / 3758092288 | 1534 / 1983 | 7.0.0-4356-enterprise | | 172.23.107.99 | kv | 51.2538382805 | 12294115328 | 8661524480 | 0 / 3758092288 | 1290 / 3850 | 7.0.0-4356-enterprise | +---------------+----------+-----------------+-------------+------------+----------------+-------------------+-----------------------+ 2021-02-02 11:05:40,358 | test | INFO | MainProcess | pool-13-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:41,831 | test | INFO | MainProcess | pool-13-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:43,398 | test | INFO | MainProcess | pool-13-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:44,625 | test | INFO | MainProcess | pool-13-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:46,045 | test | INFO | MainProcess | pool-13-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:46,049 | test | INFO | MainProcess | pool-13-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:47,509 | test | INFO | MainProcess | pool-13-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:49,045 | test | INFO | MainProcess | pool-13-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:50,312 | test | INFO | MainProcess | pool-13-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:51,736 | test | INFO | MainProcess | pool-13-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:51,740 | test | INFO | MainProcess | pool-13-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:53,165 | test | INFO | MainProcess | pool-13-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:54,687 | test | INFO | MainProcess | pool-13-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:55,996 | test | INFO | MainProcess | pool-13-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:57,448 | test | INFO | MainProcess | pool-13-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:57,483 | test | INFO | MainProcess | pool-13-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:05:58,836 | test | INFO | MainProcess | pool-13-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:06:00,355 | test | INFO | MainProcess | pool-13-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:06:01,644 | test | INFO | MainProcess | pool-13-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:06:03,121 | test | INFO | MainProcess | pool-13-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:06:03,243 | test | INFO | MainProcess | pool-13-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:06:04,539 | test | INFO | MainProcess | pool-13-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:06:06,019 | test | INFO | MainProcess | pool-13-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:06:07,311 | test | INFO | MainProcess | pool-13-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:06:08,782 | test | INFO | MainProcess | pool-13-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:06:13,783 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4381] Validating collection stats and item counts 2021-02-02 11:08:40,836 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-----------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-----------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 6000 | 524288000 | 17905625 | 576815208 | | bucket2 | ephemeral | 2 | none | 0 | 6000 | 524288000 | 394912810 | 170 | | default | couchbase | 2 | none | 0 | 10000 | 524288000 | 21256007 | 577933078 | +---------+-----------+----------+------------+-----+-------+-----------+-----------+-----------+ 2021-02-02 11:08:40,838 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:552] ========= CollectionBase setup complete for test #12 test_data_load_collections_with_graceful_failover_recovery ========= 2021-02-02 11:08:44,651 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:710] Doing collection data load during graceful_failover_recovery 2021-02-02 11:08:44,651 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:242] Starting rebalance operation of type : graceful_failover_recovery 2021-02-02 11:09:20,986 | test | INFO | MainProcess | pool-13-thread-28 | [rest_client:monitorRebalance:1438] Rebalance done. Taken 36.2109999657 seconds to complete 2021-02-02 11:09:20,990 | test | INFO | MainProcess | pool-13-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-02-02 11:11:26,023 | test | INFO | MainProcess | MainThread | [collections_rebalance:wait_for_failover_or_assert:213] 1 nodes failed over as expected in 0.0309998989105 seconds 2021-02-02 11:11:26,023 | test | INFO | MainProcess | MainThread | [collections_rebalance:data_load_after_failover:192] Starting a sync data load after failover 2021-02-02 11:11:26,115 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4624] Performing scope/collection specific operations 2021-02-02 11:11:36,622 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4749] Done Performing scope/collection specific operations 2021-02-02 11:11:55,390 | test | INFO | MainProcess | pool-13-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................**X | 530.800 | | 530.800 | ...................**............................................................*....................X | 2567.554 | +---------+---------------------------------------------------------------------------------------------------------+----------+ 2021-02-02 11:11:56,812 | test | INFO | MainProcess | pool-13-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...............*****................................................................*.....................*X | 3076.750 | +-------+----------------------------------------------------------------------------------------------------------------+----------+ 2021-02-02 11:11:57,073 | test | INFO | MainProcess | pool-13-thread-27 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.................*****.............................................................*......................*X | 2983.644 | +-------+----------------------------------------------------------------------------------------------------------------+----------+ 2021-02-02 11:11:57,217 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1710] 172.23.107.99 - Node not part of cluster inactiveFailed 2021-02-02 11:11:57,240 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1710] 172.23.107.99 - Node not part of cluster inactiveFailed 2021-02-02 11:11:58,407 | test | INFO | MainProcess | pool-13-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:11:59,862 | test | INFO | MainProcess | pool-13-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:01,336 | test | INFO | MainProcess | pool-13-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:02,770 | test | INFO | MainProcess | pool-13-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:04,071 | test | INFO | MainProcess | pool-13-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:05,507 | test | INFO | MainProcess | pool-13-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:06,977 | test | INFO | MainProcess | pool-13-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:08,411 | test | INFO | MainProcess | pool-13-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:09,730 | test | INFO | MainProcess | pool-13-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:11,184 | test | INFO | MainProcess | pool-13-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:12,625 | test | INFO | MainProcess | pool-13-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:14,092 | test | INFO | MainProcess | pool-13-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:15,378 | test | INFO | MainProcess | pool-13-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:16,885 | test | INFO | MainProcess | pool-13-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:18,256 | test | INFO | MainProcess | pool-13-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:19,746 | test | INFO | MainProcess | pool-13-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:21,016 | test | INFO | MainProcess | pool-13-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:22,568 | test | INFO | MainProcess | pool-13-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:23,898 | test | INFO | MainProcess | pool-13-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:25,431 | test | INFO | MainProcess | pool-13-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-02 11:12:30,513 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4624] Performing scope/collection specific operations 2021-02-02 11:12:30,592 | test | INFO | MainProcess | pool-13-thread-15 | [table_view:display:72] Rebalance Overview +---------------+----------+--------------+ | Nodes | Services | Status | +---------------+----------+--------------+ | 172.23.107.97 | kv | Cluster node | | 172.23.107.94 | kv | Cluster node | | 172.23.107.98 | kv | Cluster node | | 172.23.107.95 | kv | Cluster node | | 172.23.107.99 | kv | Cluster node | +---------------+----------+--------------+ 2021-02-02 11:12:30,601 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-02 11:12:35,674 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 0.70224358543 2021-02-02 11:12:40,760 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 3.51223519528 2021-02-02 11:12:43,572 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4749] Done Performing scope/collection specific operations 2021-02-02 11:12:46,598 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 7.868345713 2021-02-02 11:12:50,217 | test | INFO | MainProcess | pool-13-thread-16 | [table_view:display:72] Keys failed in default:_default:SW-19-562000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-31 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:50,219 | test | INFO | MainProcess | pool-13-thread-16 | [table_view:display:72] Keys failed in default:_default:SW-19-562000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-31 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:51,496 | test | INFO | MainProcess | pool-13-thread-7 | [table_view:display:72] Keys failed in default:UVSHwQDZZ1gxhvRr-Tu_9sUiidDe9pE-pKsYq4LwWa%IGRzmgtg%6-3-gmJdCG8LEq%8SfJHmwfHgBog_KQV%J07QIw1E%9ZGCQscFrEQrBqHCKqb81FVvcH0D1ic7ZelABgwdatOitJ0mz9pFC4Tn_ZejjzH-33-737000:iT_1CpvS4UWUBEWXjaOpUsfpdzhVF5W0AbfU9tRMJ9udBzD%UN%XdMRPrnaNaqjXLV8QU4I4zb7yy9u21m7a_9-gMpylnrse-NSpHbByIjCEYE-pHYkypfWm-LhUyTJdOsmXfll3FAngRH33bCbdGNGkDZjFBPjJDEjnhert3VH3f0IvZfhP4VH6CDHi47-33-899000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1149 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:51,645 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 11.8865980882 2021-02-02 11:12:51,878 | test | INFO | MainProcess | pool-13-thread-10 | [table_view:display:72] Keys failed in default:GJb_cy7e1Ew4cmtzV52IjDIzhpY7qoQW05eAGgJ4efDa%6OzaPMWES%rOh0a8VZTaKTLUeGMct6LKQ0iLgRV5mYMcYkIcCKOYiApDdsEX4H7YYvivGGVHykem0_w5wgmUp-33-333000:oPV2sCjM_oy3Zo%ZeOwJeyUk2ECif5eopInHHuF_6PK-33-496000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5057 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:51,878 | test | INFO | MainProcess | pool-13-thread-10 | [table_view:display:72] Keys failed in default:GJb_cy7e1Ew4cmtzV52IjDIzhpY7qoQW05eAGgJ4efDa%6OzaPMWES%rOh0a8VZTaKTLUeGMct6LKQ0iLgRV5mYMcYkIcCKOYiApDdsEX4H7YYvivGGVHykem0_w5wgmUp-33-333000:oPV2sCjM_oy3Zo%ZeOwJeyUk2ECif5eopInHHuF_6PK-33-496000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5057 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:52,668 | test | INFO | MainProcess | pool-13-thread-7 | [table_view:display:72] Keys failed in default:UVSHwQDZZ1gxhvRr-Tu_9sUiidDe9pE-pKsYq4LwWa%IGRzmgtg%6-3-gmJdCG8LEq%8SfJHmwfHgBog_KQV%J07QIw1E%9ZGCQscFrEQrBqHCKqb81FVvcH0D1ic7ZelABgwdatOitJ0mz9pFC4Tn_ZejjzH-33-737000:iT_1CpvS4UWUBEWXjaOpUsfpdzhVF5W0AbfU9tRMJ9udBzD%UN%XdMRPrnaNaqjXLV8QU4I4zb7yy9u21m7a_9-gMpylnrse-NSpHbByIjCEYE-pHYkypfWm-LhUyTJdOsmXfll3FAngRH33bCbdGNGkDZjFBPjJDEjnhert3VH3f0IvZfhP4VH6CDHi47-33-899000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1656 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:52,683 | test | INFO | MainProcess | pool-13-thread-30 | [table_view:display:72] Keys failed in default:4hezn07XvgQSFd_TgzmPxMb0t6-19-578000:KsSxGmyTL_yEdau%oE8yKSJcXYigL-TusuB1CY4Dfx1XhIS7BCl5eP4OaHEb0SS0N-bpHZBI3%PW8NlRYe-36-333000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1656 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:52,688 | test | INFO | MainProcess | pool-13-thread-13 | [table_view:display:72] Keys failed in default:GJb_cy7e1Ew4cmtzV52IjDIzhpY7qoQW05eAGgJ4efDa%6OzaPMWES%rOh0a8VZTaKTLUeGMct6LKQ0iLgRV5mYMcYkIcCKOYiApDdsEX4H7YYvivGGVHykem0_w5wgmUp-33-333000:oPV2sCjM_oy3Zo%ZeOwJeyUk2ECif5eopInHHuF_6PK-33-496000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1656 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:53,302 | test | INFO | MainProcess | pool-13-thread-13 | [table_view:display:72] Keys failed in default:4hezn07XvgQSFd_TgzmPxMb0t6-19-578000:WiwM%tVYHOTQGmvzpDYLXvfZl5Ts-b0Omme1qfql_FvY2w7DCqUVemd4Z-whO8CMBlkIfl8nMVKvvASfQzC4tR9Qo0O-22qMMz7G4b9vjcHZebeWYqfRY5fWCLls3-hby7KMRp-19-579000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:53,404 | test | INFO | MainProcess | pool-13-thread-24 | [table_view:display:72] Keys failed in default:iOBXwQ0Y-cAfQ44cQYDqo2n9m-m2K0D4YgTWSb-_X4L%MFe5ljw-41-703000:ik46cUl7oxZl4xCR3HKZPa7IgC1b3WpU_KANF7hgpoEnjJGmsHnEF9GcDIS_2t9a6ciiKqWALPXnNhGe2D4GsmeugrUCkk3qmmWCTPt9LaVRexTkDXpU%IfOCGwqXCwey2AdfjAzGgRVUde07bMGzOtVb-41-841000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-134 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:53,404 | test | INFO | MainProcess | pool-13-thread-28 | [table_view:display:72] Keys failed in default:PxbzJRWpznXZ0k_QhZqC1FkVbAaGiprR7aKixUjdFGEGuAYPAjwcvaxP2l%zBW9Uja29X%y-r5lP3L03VXVRfv0btLVmsH-O5HNfmyY6hQQDIex6WnRIODAuSyf7YI4lBotFvYVvjukwHxgQYk3UVgzG1qFMfY98ENwIh3-OTjI76dcKn%vQwijZuTVaf3qgnpVhl-42-939000:ruaxOJgjALP8TFrRF6n_1x7yx25J3P%eRZ_moywLibp_g4l4752T-p39rstRYAwK4VCgbD49PwvNuMb1dZxE3-XZd5GjyggSvHRKJMenw_weeWPEy1Q-43-108000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-134 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:53,404 | test | INFO | MainProcess | pool-13-thread-24 | [table_view:display:72] Keys failed in default:iOBXwQ0Y-cAfQ44cQYDqo2n9m-m2K0D4YgTWSb-_X4L%MFe5ljw-41-703000:ik46cUl7oxZl4xCR3HKZPa7IgC1b3WpU_KANF7hgpoEnjJGmsHnEF9GcDIS_2t9a6ciiKqWALPXnNhGe2D4GsmeugrUCkk3qmmWCTPt9LaVRexTkDXpU%IfOCGwqXCwey2AdfjAzGgRVUde07bMGzOtVb-41-841000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-134 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:53,404 | test | INFO | MainProcess | pool-13-thread-28 | [table_view:display:72] Keys failed in default:PxbzJRWpznXZ0k_QhZqC1FkVbAaGiprR7aKixUjdFGEGuAYPAjwcvaxP2l%zBW9Uja29X%y-r5lP3L03VXVRfv0btLVmsH-O5HNfmyY6hQQDIex6WnRIODAuSyf7YI4lBotFvYVvjukwHxgQYk3UVgzG1qFMfY98ENwIh3-OTjI76dcKn%vQwijZuTVaf3qgnpVhl-42-939000:ruaxOJgjALP8TFrRF6n_1x7yx25J3P%eRZ_moywLibp_g4l4752T-p39rstRYAwK4VCgbD49PwvNuMb1dZxE3-XZd5GjyggSvHRKJMenw_weeWPEy1Q-43-108000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-134 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:54,769 | test | INFO | MainProcess | pool-13-thread-1 | [table_view:display:72] Keys failed in default:7NE2mM9Dm8onD-6VNaVcb_gNYO2lfkoLUE5ang8NTzG-zvPyEEj2xQX7IL6TGGCGnCuDTBiq4qsJoaGeIA1klVUZF045wO2I%a-OJVhVb-V9uWpKOsl8pm-to-U%TqRgnPnEk-vqNdbihGZVtSNGrq43LdU-19-581000:ppgd0GXrABCN-eDZzmON8js835mFN6taBhGl8_cjLRZ3%e3ysWaqrg5Wt7YJjaxGzIYzKCpnOsyYRmTPLLt3kL-tzy2UhEFxMjV8d21pp2O-19-584000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:54,769 | test | INFO | MainProcess | pool-13-thread-7 | [table_view:display:72] Keys failed in default:_default:ZlmhtOqnVutH4yUcOnZ3yxc8XBnhp7ejAiztON6O-R4nnC_DjWiL7DXwbRFfvN%HsXA2sSbc3YXze5EIwk1vVtLQYmEm8jg76Q0bi4mTmu5_5nXGPHNS1520ES0vGRZaZzcVgWvMMxTPPUXdMSDzC45L-7isXkDCe6FLZzlCfRE4TzHg%BhmtdgoaV7mqYhl-wDkxJi%goQ_bVB3dV2W-lAtI-19-561000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:54,778 | test | INFO | MainProcess | pool-13-thread-12 | [table_view:display:72] Keys failed in default:7NE2mM9Dm8onD-6VNaVcb_gNYO2lfkoLUE5ang8NTzG-zvPyEEj2xQX7IL6TGGCGnCuDTBiq4qsJoaGeIA1klVUZF045wO2I%a-OJVhVb-V9uWpKOsl8pm-to-U%TqRgnPnEk-vqNdbihGZVtSNGrq43LdU-19-581000:qbEgOsYXpQVEju%MYNOxYhIPLaRwAHYr2Ba08SeksE51hBdTatp7vV8JxfWD84wt3MRWfllqJVFkcrS42mfYtLEIz0bLgqtGCvqSpW8wdLfYj5pk_tzzTcvw_ayIpVNX65FoN72Gmw9RAHRY30znVnnFZcDl-IKu5WaFKJI3VJ5AFZ2YPE%%FFyeYAc700R-XIR-88FifmzB5j96ifNs8l9jUV5I0RZVyo-19-584000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:54,854 | test | INFO | MainProcess | pool-13-thread-10 | [table_view:display:72] Keys failed in default:GJb_cy7e1Ew4cmtzV52IjDIzhpY7qoQW05eAGgJ4efDa%6OzaPMWES%rOh0a8VZTaKTLUeGMct6LKQ0iLgRV5mYMcYkIcCKOYiApDdsEX4H7YYvivGGVHykem0_w5wgmUp-33-333000:oPV2sCjM_oy3Zo%ZeOwJeyUk2ECif5eopInHHuF_6PK-33-496000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5462 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5521 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:54,854 | test | INFO | MainProcess | pool-13-thread-10 | [table_view:display:72] Keys failed in default:GJb_cy7e1Ew4cmtzV52IjDIzhpY7qoQW05eAGgJ4efDa%6OzaPMWES%rOh0a8VZTaKTLUeGMct6LKQ0iLgRV5mYMcYkIcCKOYiApDdsEX4H7YYvivGGVHykem0_w5wgmUp-33-333000:oPV2sCjM_oy3Zo%ZeOwJeyUk2ECif5eopInHHuF_6PK-33-496000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5462 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5521 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-02 11:12:54,901 | test | ERROR | MainProcess | pool-13-thread-6 | [task:call:3540] Failed to load 1 docs from 1000 to 1960 2021-02-02 11:12:56,694 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 14.632675486 2021-02-02 11:13:01,737 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 17.4102600954 2021-02-02 11:13:06,809 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 19.5447110042 2021-02-02 11:13:07,851 | test | ERROR | MainProcess | pool-13-thread-6 | [task:call:3540] Failed to load 1 docs from 0 to 5000 2021-02-02 11:13:08,328 | test | ERROR | MainProcess | pool-13-thread-6 | [task:call:3540] Failed to load 1 docs from 0 to 5000 2021-02-02 11:13:08,338 | test | ERROR | MainProcess | pool-13-thread-6 | [task:call:3540] Failed to load 3 docs from 5000 to 5800 2021-02-02 11:13:08,342 | test | ERROR | MainProcess | pool-13-thread-6 | [task:call:3540] Failed to load 1 docs from 29 to 33 2021-02-02 11:13:08,345 | test | ERROR | MainProcess | pool-13-thread-6 | [task:call:3540] Failed to load 2 docs from 1000 to 1960 2021-02-02 11:13:08,345 | test | ERROR | MainProcess | pool-13-thread-6 | [task:call:3540] Failed to load 1 docs from 1000 to 1960 2021-02-02 11:13:08,348 | test | ERROR | MainProcess | pool-13-thread-6 | [task:call:3540] Failed to load 1 docs from 5 to 9 2021-02-02 11:13:11,260 | test | ERROR | MainProcess | pool-13-thread-6 | [task:call:3540] Failed to load 1 docs from 5 to 9 2021-02-02 11:13:11,262 | test | ERROR | MainProcess | pool-13-thread-6 | [task:call:3540] Failed to load 1 docs from 5 to 9 2021-02-02 11:13:11,262 | test | ERROR | MainProcess | pool-13-thread-6 | [task:call:3540] Failed to load 1 docs from 5 to 9 2021-02-02 11:13:11,842 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 22.1985908471 2021-02-02 11:13:16,891 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 25.1149187694 2021-02-02 11:13:22,025 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 28.9132370832 2021-02-02 11:13:27,200 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 33.2816537468 2021-02-02 11:13:30,575 | test | INFO | MainProcess | pool-13-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***...................................................................................................*****X | 3530.189 | | 1211.246 | .................................****X | 1211.246 | | 379.773 | .........X | 379.773 | | 379.773 | .........***X | 379.773 | | 314.600 | .......***X | 314.600 | +----------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-02-02 11:13:31,329 | test | INFO | MainProcess | pool-13-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+--------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***..........................................................******..........................................****X | 2406.695 | | 1603.718 | .................................................................****X | 1603.718 | | 981.600 | .......................................X | 981.600 | | 981.600 | .......................................***X | 981.600 | +----------+--------------------------------------------------------------------------------------------------------------------+----------+ 2021-02-02 11:13:32,236 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 35.967433139 2021-02-02 11:13:32,332 | test | INFO | MainProcess | pool-13-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +---------+-------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***........................*.......*****...............................................................*****......****X | 1901.591 | | 973.600 | ..................................................****X | 973.600 | +---------+-------------------------------------------------------------------------------------------------------------------------+----------+ 2021-02-02 11:13:37,296 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 42.2897738458 2021-02-02 11:13:42,354 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 49.393250426 2021-02-02 11:13:47,413 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 56.7819574486 2021-02-02 11:13:52,467 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 64.2732098299 2021-02-02 11:13:57,510 | test | INFO | MainProcess | pool-13-thread-15 | [task:check:322] Rebalance - status: running, progress: 68.5398587708 2021-02-02 11:14:02,551 | test | ERROR | MainProcess | pool-13-thread-15 | [rest_client:_rebalance_status_and_progress:1484] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2021-02-02 11:14:02,581 | test | INFO | MainProcess | pool-13-thread-15 | [rest_client:print_UI_logs:2593] Latest logs from UI on 172.23.107.94: 2021-02-02 11:14:02,581 | test | ERROR | MainProcess | pool-13-thread-15 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.107.94', u'tstamp': 1612293239898L, u'shortText': u'message', u'serverTime': u'2021-02-02T11:13:59.898Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.17273.10>,\n {{wait_seqno_persisted_failed,"bucket2",407,\n 844,\n [{\'ns_1@172.23.107.99\',\n {\'EXIT\',\n {{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,158}]},\n {mc_client_binary,\n wait_for_seqno_persistence,3,\n [{file,"src/mc_client_binary.erl"},\n {line,763}]},\n {ns_memcached,\n \'-wait_for_seqno_persistence/3-fun-0-\',\n 3,\n [{file,"src/ns_memcached.erl"},\n {line,1391}]},\n {ns_memcached,\n \'-perform_very_long_call/3-fun-0-\',2,\n [{file,"src/ns_memcached.erl"},\n {line,324}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,92}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},\n {line,197}]}]},\n {gen_server,call,\n [{\'janitor_agent-bucket2\',\n \'ns_1@172.23.107.99\'},\n {if_rebalance,<0.10836.10>,\n {wait_seqno_persisted,407,844}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,488}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 4552e1ed4b74f427f47c1a4b524157c5'} 2021-02-02 11:14:02,581 | test | ERROR | MainProcess | pool-13-thread-15 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.107.99', u'tstamp': 1612293239838L, u'shortText': u'message', u'serverTime': u'2021-02-02T11:13:59.838Z', u'text': u"Control connection to memcached on 'ns_1@172.23.107.99' disconnected. Check logs for details."} 2021-02-02 11:14:02,582 | test | ERROR | MainProcess | pool-13-thread-15 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.107.99', u'tstamp': 1612293239833L, u'shortText': u'message', u'serverTime': u'2021-02-02T11:13:59.833Z', u'text': u"Service 'memcached' exited with status 134. Restarting. Messages:\n2021-02-02T11:13:59.495768-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so(_ZN5folly6detail8function14FunctionTraitsIFvvEE7callBigIZNS_21PriorityThreadFactory9newThreadEONS_8FunctionIS3_EEEUlvE_EEvRNS1_4DataE+0x109) [0x7f34b9e78000+0x2e1ac59]\n2021-02-02T11:13:59.495835-08:00 CRITICAL /opt/couchbase/bin/../lib/libep.so(_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5folly18NamedThreadFactory9newThreadEONS3_8FunctionIFvvEEEEUlvE_EEEEE6_M_runEv+0x154) [0x7f34b9e78000+0x2e15704]\n2021-02-02T11:13:59.495873-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f34b3c3b000+0xb9dcf]\n2021-02-02T11:13:59.495884-08:00 CRITICAL /lib64/libpthread.so.0() [0x7f34b27e8000+0x7ea5]\n2021-02-02T11:13:59.495917-08:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f34b241a000+0xfe8dd]\n[*** LOG ERROR ***] [2021-02-02 11:13:59] [spdlog_file_logger] async log: thread pool doesn't exist anymore\n/home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/logger/custom_rotating_file_sink.cc:84:22: runtime error: member call on null pointer of type 'struct pattern_formatter'"} 2021-02-02 11:14:02,582 | test | ERROR | MainProcess | pool-13-thread-15 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.107.94', u'tstamp': 1612293239833L, u'shortText': u'message', u'serverTime': u'2021-02-02T11:13:59.833Z', u'text': u'Worker <0.17169.10> (for action {move,{407,\n [\'ns_1@172.23.107.95\',\n \'ns_1@172.23.107.98\',undefined],\n [\'ns_1@172.23.107.95\',\n \'ns_1@172.23.107.99\',\n \'ns_1@172.23.107.98\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.17273.10>,\n {{wait_seqno_persisted_failed,\n "bucket2",\n 407,844,\n [{\'ns_1@172.23.107.99\',\n {\'EXIT\',\n {{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 158}]},\n {mc_client_binary,\n wait_for_seqno_persistence,\n 3,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 763}]},\n {ns_memcached,\n \'-wait_for_seqno_persistence/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 1391}]},\n {ns_memcached,\n \'-perform_very_long_call/3-fun-0-\',\n 2,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 324}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 92}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 197}]}]},\n {gen_server,\n call,\n [{\'janitor_agent-bucket2\',\n \'ns_1@172.23.107.99\'},\n {if_rebalance,\n <0.10836.10>,\n {wait_seqno_persisted,\n 407,\n 844}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 488}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 234}]}]}}}'} 2021-02-02 11:14:02,582 | test | ERROR | MainProcess | pool-13-thread-15 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.107.94', u'tstamp': 1612293235364L, u'shortText': u'message', u'serverTime': u'2021-02-02T11:13:55.364Z', u'text': u'Bucket "bucket2" rebalance does not seem to be swap rebalance'} 2021-02-02 11:14:02,582 | test | ERROR | MainProcess | pool-13-thread-15 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.107.99', u'tstamp': 1612293234111L, u'shortText': u'message', u'serverTime': u'2021-02-02T11:13:54.111Z', u'text': u'Bucket "bucket2" loaded on node \'ns_1@172.23.107.99\' in 0 seconds.'} 2021-02-02 11:14:02,582 | test | ERROR | MainProcess | pool-13-thread-15 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.107.94', u'tstamp': 1612293234002L, u'shortText': u'message', u'serverTime': u'2021-02-02T11:13:54.002Z', u'text': u'Started rebalancing bucket bucket2'} 2021-02-02 11:14:02,582 | test | ERROR | MainProcess | pool-13-thread-15 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.107.94', u'tstamp': 1612293208937L, u'shortText': u'message', u'serverTime': u'2021-02-02T11:13:28.937Z', u'text': u'Bucket "bucket1" rebalance does not seem to be swap rebalance'} 2021-02-02 11:14:02,582 | test | ERROR | MainProcess | pool-13-thread-15 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.107.99', u'tstamp': 1612293207645L, u'shortText': u'message', u'serverTime': u'2021-02-02T11:13:27.645Z', u'text': u'Bucket "bucket1" loaded on node \'ns_1@172.23.107.99\' in 0 seconds.'} 2021-02-02 11:14:02,582 | test | ERROR | MainProcess | pool-13-thread-15 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.107.94', u'tstamp': 1612293207495L, u'shortText': u'message', u'serverTime': u'2021-02-02T11:13:27.495Z', u'text': u'Started rebalancing bucket bucket1'} 2021-02-02 11:14:02,584 | test | ERROR | MainProcess | pool-13-thread-15 | [task:call:238] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2021-02-02 11:14:03,216 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:90] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-02-02 11:14:03,216 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:93] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-02-02 11:14:27,315 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.107.94:8091/pools/default/buckets/bucket2/scopes/-MN-_HyuU_uUwn4IkXfRmoJfELZj%25tPAEpjEzw-19-525000/collections/ID6vmSlsCsqpZY6z3hLdA9By2SGLZMPXETPPdyIWgLpPH%25dV1TYupiy_9MIEqsN6JiUaGtTzIUN1Cdys0b_8E5_pAYzmXmLdQIYKlYnCZROyrHuvT5bEIIu-C25aRj7R6%25irlafs0DtZNtBY-19-528000. Error [Errno 98] Address already in use 2021-02-02 11:14:30,332 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.107.94:8091/pools/default/buckets/bucket2/scopes/-MN-_HyuU_uUwn4IkXfRmoJfELZj%25tPAEpjEzw-19-525000/collections/ID6vmSlsCsqpZY6z3hLdA9By2SGLZMPXETPPdyIWgLpPH%25dV1TYupiy_9MIEqsN6JiUaGtTzIUN1Cdys0b_8E5_pAYzmXmLdQIYKlYnCZROyrHuvT5bEIIu-C25aRj7R6%25irlafs0DtZNtBY-19-528000. Error [Errno 98] Address already in use 2021-02-02 11:14:38,032 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:90] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-02-02 11:14:38,032 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:93] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-02-02 11:15:16,477 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:90] Bucket: default, Active Resident ratio(DGM): 100% 2021-02-02 11:15:16,479 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:93] Bucket: default, Replica Resident ratio(DGM): 100% 2021-02-02 11:15:57,302 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_if_new_messages:696] does not match any timestamp 2021-02-02 11:15:57,302 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:799] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.js-evaluator.104957 2021-02-02 11:15:57,302 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:801] 172.23.107.94: Found '^' logs - ['==js-evaluator==104957==WARNING: AddressSanitizer failed to allocate 0x33ff1278c37000 bytes\n', "==js-evaluator==104957==AddressSanitizer's allocator is terminating the process instead of returning 0\n", "==js-evaluator==104957==If you don't like this behavior set allocator_may_return_null=1\n", '==js-evaluator==104957==AddressSanitizer CHECK failed: /tmp/deploy/objdir/../gcc-7.3.0/libsanitizer/sanitizer_common/sanitizer_allocator.cc:218 "((0)) != (0)" (0x0, 0x0)\n', ' #0 0x7fe75d0c9fe5 (/opt/couchbase/bin/../lib/libasan.so.4+0xe3fe5)\n', ' #1 0x7fe75d0e6b25 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/opt/couchbase/bin/../lib/libasan.so.4+0x100b25)\n', ' #2 0x7fe75d0ce8c2 (/opt/couchbase/bin/../lib/libasan.so.4+0xe88c2)\n', ' #3 0x7fe75d00d896 (/opt/couchbase/bin/../lib/libasan.so.4+0x27896)\n', ' #4 0x7fe75d0bf8bb in __interceptor_malloc (/opt/couchbase/bin/../lib/libasan.so.4+0xd98bb)\n', ' #5 0x61a20d (/opt/couchbase/bin/js-evaluator+0x61a20d)\n', ' #6 0x4539e2 (/opt/couchbase/bin/js-evaluator+0x4539e2)\n', ' #7 0x4543de (/opt/couchbase/bin/js-evaluator+0x4543de)\n', ' #8 0x45736e (/opt/couchbase/bin/js-evaluator+0x45736e)\n', ' #9 0x4a7589 (/opt/couchbase/bin/js-evaluator+0x4a7589)\n', ' #10 0x43b22c (/opt/couchbase/bin/js-evaluator+0x43b22c)\n', ' #11 0x50407d (/opt/couchbase/bin/js-evaluator+0x50407d)\n', ' #12 0x4f3417 (/opt/couchbase/bin/js-evaluator+0x4f3417)\n', ' #13 0x40d2ed (/opt/couchbase/bin/js-evaluator+0x40d2ed)\n', ' #14 0x7fe75a613554 in __libc_start_main (/lib64/libc.so.6+0x22554)\n', ' #15 0x40ec20 (/opt/couchbase/bin/js-evaluator+0x40ec20)\n', '\n'] 2021-02-02 11:15:58,015 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_if_new_messages:696] does not match any timestamp 2021-02-02 11:15:58,015 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:799] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.js-evaluator.100378 2021-02-02 11:15:58,015 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:801] 172.23.107.95: Found '^' logs - ['==js-evaluator==100378==WARNING: AddressSanitizer failed to allocate 0x33ffb5e0745800 bytes\n', "==js-evaluator==100378==AddressSanitizer's allocator is terminating the process instead of returning 0\n", "==js-evaluator==100378==If you don't like this behavior set allocator_may_return_null=1\n", '==js-evaluator==100378==AddressSanitizer CHECK failed: /tmp/deploy/objdir/../gcc-7.3.0/libsanitizer/sanitizer_common/sanitizer_allocator.cc:218 "((0)) != (0)" (0x0, 0x0)\n', ' #0 0x7ff8836e9fe5 (/opt/couchbase/bin/../lib/libasan.so.4+0xe3fe5)\n', ' #1 0x7ff883706b25 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/opt/couchbase/bin/../lib/libasan.so.4+0x100b25)\n', ' #2 0x7ff8836ee8c2 (/opt/couchbase/bin/../lib/libasan.so.4+0xe88c2)\n', ' #3 0x7ff88362d896 (/opt/couchbase/bin/../lib/libasan.so.4+0x27896)\n', ' #4 0x7ff8836df8bb in __interceptor_malloc (/opt/couchbase/bin/../lib/libasan.so.4+0xd98bb)\n', ' #5 0x61a20d (/opt/couchbase/bin/js-evaluator+0x61a20d)\n', ' #6 0x4539e2 (/opt/couchbase/bin/js-evaluator+0x4539e2)\n', ' #7 0x4543de (/opt/couchbase/bin/js-evaluator+0x4543de)\n', ' #8 0x45736e (/opt/couchbase/bin/js-evaluator+0x45736e)\n', ' #9 0x4a7589 (/opt/couchbase/bin/js-evaluator+0x4a7589)\n', ' #10 0x43b22c (/opt/couchbase/bin/js-evaluator+0x43b22c)\n', ' #11 0x50407d (/opt/couchbase/bin/js-evaluator+0x50407d)\n', ' #12 0x4f3417 (/opt/couchbase/bin/js-evaluator+0x4f3417)\n', ' #13 0x40d2ed (/opt/couchbase/bin/js-evaluator+0x40d2ed)\n', ' #14 0x7ff880c33554 in __libc_start_main (/lib64/libc.so.6+0x22554)\n', ' #15 0x40ec20 (/opt/couchbase/bin/js-evaluator+0x40ec20)\n', '\n'] 2021-02-02 11:15:58,658 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_if_new_messages:696] does not match any timestamp 2021-02-02 11:15:58,658 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:799] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.js-evaluator.100324 2021-02-02 11:15:58,658 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:801] 172.23.107.97: Found '^' logs - ['==js-evaluator==100324==WARNING: AddressSanitizer failed to allocate 0x33ffe23bee4000 bytes\n', "==js-evaluator==100324==AddressSanitizer's allocator is terminating the process instead of returning 0\n", "==js-evaluator==100324==If you don't like this behavior set allocator_may_return_null=1\n", '==js-evaluator==100324==AddressSanitizer CHECK failed: /tmp/deploy/objdir/../gcc-7.3.0/libsanitizer/sanitizer_common/sanitizer_allocator.cc:218 "((0)) != (0)" (0x0, 0x0)\n', ' #0 0x7f14e87a6fe5 (/opt/couchbase/bin/../lib/libasan.so.4+0xe3fe5)\n', ' #1 0x7f14e87c3b25 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/opt/couchbase/bin/../lib/libasan.so.4+0x100b25)\n', ' #2 0x7f14e87ab8c2 (/opt/couchbase/bin/../lib/libasan.so.4+0xe88c2)\n', ' #3 0x7f14e86ea896 (/opt/couchbase/bin/../lib/libasan.so.4+0x27896)\n', ' #4 0x7f14e879c8bb in __interceptor_malloc (/opt/couchbase/bin/../lib/libasan.so.4+0xd98bb)\n', ' #5 0x61a20d (/opt/couchbase/bin/js-evaluator+0x61a20d)\n', ' #6 0x4539e2 (/opt/couchbase/bin/js-evaluator+0x4539e2)\n', ' #7 0x4543de (/opt/couchbase/bin/js-evaluator+0x4543de)\n', ' #8 0x45736e (/opt/couchbase/bin/js-evaluator+0x45736e)\n', ' #9 0x4a7589 (/opt/couchbase/bin/js-evaluator+0x4a7589)\n', ' #10 0x43b22c (/opt/couchbase/bin/js-evaluator+0x43b22c)\n', ' #11 0x50407d (/opt/couchbase/bin/js-evaluator+0x50407d)\n', ' #12 0x4f3417 (/opt/couchbase/bin/js-evaluator+0x4f3417)\n', ' #13 0x40d2ed (/opt/couchbase/bin/js-evaluator+0x40d2ed)\n', ' #14 0x7f14e5cf0554 in __libc_start_main (/lib64/libc.so.6+0x22554)\n', ' #15 0x40ec20 (/opt/couchbase/bin/js-evaluator+0x40ec20)\n', '\n'] 2021-02-02 11:15:59,311 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_if_new_messages:696] does not match any timestamp 2021-02-02 11:15:59,312 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:799] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.js-evaluator.100346 2021-02-02 11:15:59,312 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:801] 172.23.107.98: Found '^' logs - ['==js-evaluator==100346==WARNING: AddressSanitizer failed to allocate 0x33ffd9e19b5800 bytes\n', "==js-evaluator==100346==AddressSanitizer's allocator is terminating the process instead of returning 0\n", "==js-evaluator==100346==If you don't like this behavior set allocator_may_return_null=1\n", '==js-evaluator==100346==AddressSanitizer CHECK failed: /tmp/deploy/objdir/../gcc-7.3.0/libsanitizer/sanitizer_common/sanitizer_allocator.cc:218 "((0)) != (0)" (0x0, 0x0)\n', ' #0 0x7efe4dcfefe5 (/opt/couchbase/bin/../lib/libasan.so.4+0xe3fe5)\n', ' #1 0x7efe4dd1bb25 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/opt/couchbase/bin/../lib/libasan.so.4+0x100b25)\n', ' #2 0x7efe4dd038c2 (/opt/couchbase/bin/../lib/libasan.so.4+0xe88c2)\n', ' #3 0x7efe4dc42896 (/opt/couchbase/bin/../lib/libasan.so.4+0x27896)\n', ' #4 0x7efe4dcf48bb in __interceptor_malloc (/opt/couchbase/bin/../lib/libasan.so.4+0xd98bb)\n', ' #5 0x61a20d (/opt/couchbase/bin/js-evaluator+0x61a20d)\n', ' #6 0x4539e2 (/opt/couchbase/bin/js-evaluator+0x4539e2)\n', ' #7 0x4543de (/opt/couchbase/bin/js-evaluator+0x4543de)\n', ' #8 0x45736e (/opt/couchbase/bin/js-evaluator+0x45736e)\n', ' #9 0x4a7589 (/opt/couchbase/bin/js-evaluator+0x4a7589)\n', ' #10 0x43b22c (/opt/couchbase/bin/js-evaluator+0x43b22c)\n', ' #11 0x50407d (/opt/couchbase/bin/js-evaluator+0x50407d)\n', ' #12 0x4f3417 (/opt/couchbase/bin/js-evaluator+0x4f3417)\n', ' #13 0x40d2ed (/opt/couchbase/bin/js-evaluator+0x40d2ed)\n', ' #14 0x7efe4b248554 in __libc_start_main (/lib64/libc.so.6+0x22554)\n', ' #15 0x40ec20 (/opt/couchbase/bin/js-evaluator+0x40ec20)\n', '\n'] 2021-02-02 11:15:59,822 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:728] 172.23.107.99: 1 core dump seen 2021-02-02 11:16:20,163 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_if_new_messages:696] does not match any timestamp 2021-02-02 11:16:20,163 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:799] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.js-evaluator.106546 2021-02-02 11:16:20,163 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:801] 172.23.107.99: Found '^' logs - ['==js-evaluator==106546==WARNING: AddressSanitizer failed to allocate 0x33fee0e6d6a800 bytes\n', "==js-evaluator==106546==AddressSanitizer's allocator is terminating the process instead of returning 0\n", "==js-evaluator==106546==If you don't like this behavior set allocator_may_return_null=1\n", '==js-evaluator==106546==AddressSanitizer CHECK failed: /tmp/deploy/objdir/../gcc-7.3.0/libsanitizer/sanitizer_common/sanitizer_allocator.cc:218 "((0)) != (0)" (0x0, 0x0)\n', ' #0 0x7f80587a9fe5 (/opt/couchbase/bin/../lib/libasan.so.4+0xe3fe5)\n', ' #1 0x7f80587c6b25 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/opt/couchbase/bin/../lib/libasan.so.4+0x100b25)\n', ' #2 0x7f80587ae8c2 (/opt/couchbase/bin/../lib/libasan.so.4+0xe88c2)\n', ' #3 0x7f80586ed896 (/opt/couchbase/bin/../lib/libasan.so.4+0x27896)\n', ' #4 0x7f805879f8bb in __interceptor_malloc (/opt/couchbase/bin/../lib/libasan.so.4+0xd98bb)\n', ' #5 0x61a20d (/opt/couchbase/bin/js-evaluator+0x61a20d)\n', ' #6 0x4539e2 (/opt/couchbase/bin/js-evaluator+0x4539e2)\n', ' #7 0x4543de (/opt/couchbase/bin/js-evaluator+0x4543de)\n', ' #8 0x45736e (/opt/couchbase/bin/js-evaluator+0x45736e)\n', ' #9 0x4a7589 (/opt/couchbase/bin/js-evaluator+0x4a7589)\n', ' #10 0x43b22c (/opt/couchbase/bin/js-evaluator+0x43b22c)\n', ' #11 0x50407d (/opt/couchbase/bin/js-evaluator+0x50407d)\n', ' #12 0x4f3417 (/opt/couchbase/bin/js-evaluator+0x4f3417)\n', ' #13 0x40d2ed (/opt/couchbase/bin/js-evaluator+0x40d2ed)\n', ' #14 0x7f8055cf3554 in __libc_start_main (/lib64/libc.so.6+0x22554)\n', ' #15 0x40ec20 (/opt/couchbase/bin/js-evaluator+0x40ec20)\n', '\n'] 2021-02-02 11:16:20,217 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1073] Running cbcollect on node ns_1@172.23.107.94,ns_1@172.23.107.95,ns_1@172.23.107.97,ns_1@172.23.107.98,ns_1@172.23.107.99 2021-02-02 11:16:30,236 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1076] ns_1@172.23.107.94,ns_1@172.23.107.95,ns_1@172.23.107.97,ns_1@172.23.107.98,ns_1@172.23.107.99 - cbcollect status: True 2021-02-02 11:16:30,236 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1081] Polling active_tasks to check cbcollect status 2021-02-02 11:21:20,753 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1104] ns_1@172.23.107.94: Copying cbcollect ZIP file to Client 2021-02-02 11:21:24,474 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] 172.23.107.94 node cb collect zip copied on client : True 2021-02-02 11:21:24,474 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1104] ns_1@172.23.107.95: Copying cbcollect ZIP file to Client 2021-02-02 11:21:27,826 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] 172.23.107.95 node cb collect zip copied on client : True 2021-02-02 11:21:27,826 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1104] ns_1@172.23.107.97: Copying cbcollect ZIP file to Client 2021-02-02 11:21:31,252 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] 172.23.107.97 node cb collect zip copied on client : True 2021-02-02 11:21:31,253 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1104] ns_1@172.23.107.98: Copying cbcollect ZIP file to Client 2021-02-02 11:21:34,634 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] 172.23.107.98 node cb collect zip copied on client : True 2021-02-02 11:21:34,635 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1104] ns_1@172.23.107.99: Copying cbcollect ZIP file to Client 2021-02-02 11:21:38,476 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] 172.23.107.99 node cb collect zip copied on client : True 2021-02-02 11:21:38,490 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:497] Skipping get_trace !! 2021-02-02 11:21:42,864 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:747] GET http://172.23.107.94:8091/pools/default/remoteClusters/ body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password 2021-02-02 11:21:53,994 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1438] Rebalance done. Taken 11.0529999733 seconds to complete 2021-02-02 11:21:54,002 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-02-02 11:22:28,535 | test | WARNING | MainProcess | MainThread | [basetestcase:tearDown:459] CRASH | CRITICAL | WARN messages found in cb_logs