2021-04-07 02:16:52,273 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionDurabilityTests setup started for test #1 test_sync_write_in_progress ========= 2021-04-07 02:16:53,336 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #1 test_sync_write_in_progress ========= 2021-04-07 02:17:07,203 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.7690000534 seconds to complete 2021-04-07 02:17:07,224 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-04-07 02:17:12,336 | test | WARNING | MainProcess | MainThread | [basetestcase:setUp:303] Cluster operation in setup will be skipped 2021-04-07 02:17:12,338 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #1 test_sync_write_in_progress ========= 2021-04-07 02:17:12,338 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #1 test_sync_write_in_progress ========= 2021-04-07 02:17:15,694 | test | INFO | MainProcess | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.105.155 | kv | 7.0.0-4854-enterprise | 2.7763755679 | Cluster node | | 172.23.105.211 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-04-07 02:17:15,717 | test | INFO | MainProcess | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-07 02:17:25,805 | test | INFO | MainProcess | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-07 02:17:25,835 | test | INFO | MainProcess | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 10.1419999599 sec 2021-04-07 02:17:26,178 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-04-07 02:17:26,795 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2261] Waiting for all collections to be created 2021-04-07 02:17:26,917 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | 69JcYKDRtbqS-26-225000 | couchbase | 1 | none | 0 | 0 | 5859442688 | 0 | 0 | +------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-04-07 02:17:26,918 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:146] Creating required SDK clients for client_pool 2021-04-07 02:17:37,805 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-04-07 02:17:37,823 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-04-07 02:17:39,599 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.105.155 | kv | 15.958259099 | 4201840640 | 3362824192 | 57933824 / 3758092288 | 0 / 0 | 7.0.0-4854-enterprise | | 172.23.105.211 | kv | 19.7070914697 | 4201840640 | 3494928384 | 0 / 3758092288 | 0 / 0 | 7.0.0-4854-enterprise | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2021-04-07 02:17:42,823 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-04-07 02:17:51,987 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | 69JcYKDRtbqS-26-225000 | couchbase | 1 | none | 0 | 600 | 5859442688 | 62126816 | 20188361 | +------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-04-07 02:17:51,989 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #1 test_sync_write_in_progress ========= 2021-04-07 02:17:51,990 | test | INFO | MainProcess | MainThread | [collection_crud_negative:_CollectionDurabilityTests__get_d_level_and_error_to_simulate:54] Testing with durability_level=MAJORITY_AND_PERSIST_TO_ACTIVE, simulate_error=stop_persistence 2021-04-07 02:17:54,157 | test | INFO | MainProcess | MainThread | [cb_error:create:39] Simulating 'stop_persistence' in 172.23.105.211 2021-04-07 02:17:54,813 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 3 seconds. Reason: Wait for error simulation to take effect 2021-04-07 02:17:57,815 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-04-07 02:17:57,823 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-04-07 02:17:57,918 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait for doc ops to reach server 2021-04-07 02:18:04,953 | test | ERROR | MainProcess | MainThread | [basetestcase:log_failure:626] Invalid retry reason for test_collections-100: com.couchbase.client.core.error.RequestCanceledException: InsertRequest, Reason: NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS) {"cancelled":true,"completed":true,"coreId":"0xd87dcbeb0000001f","idempotent":false,"lastChannelId":"D87DCBEB0000001F/00000000B329CA6D","lastDispatchedFrom":"172.23.106.205:60494","lastDispatchedTo":"172.23.105.211:11210","reason":"NO_MORE_RETRIES (COLLECTION_MAP_REFRESH_IN_PROGRESS)","requestId":3262,"requestType":"InsertRequest","retried":0,"service":{"bucket":"69JcYKDRtbqS-26-225000","collection":"WeFlJcuaiaTcYqgh1KC-26-226000","documentId":"test_collections-100","opaque":"0x4f57","scope":"_default","syncDurability":"MAJORITY_AND_PERSIST_TO_ACTIVE","type":"kv"},"timeoutMs":2000,"timings":{"encodingMicros":50,"totalMicros":551}} 2021-04-07 02:18:13,023 | test | INFO | MainProcess | MainThread | [cb_error:revert:68] Reverting 'stop_persistence' in 172.23.105.211 2021-04-07 02:18:13,371 | test | INFO | MainProcess | pool-2-thread-16 | [table_view:display:72] Ops trend for bucket '69JcYKDRtbqS-26-225000' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 30.000 | ...................................................................................................X | 30.000 | | 0.000 | *******X | 0.000 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2021-04-07 02:18:14,553 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:92] Bucket: 69JcYKDRtbqS-26-225000, Active Resident ratio(DGM): 100% 2021-04-07 02:18:14,555 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:95] Bucket: 69JcYKDRtbqS-26-225000, Replica Resident ratio(DGM): 100% 2021-04-07 02:18:21,842 | test | WARNING | MainProcess | MainThread | [basetestcase:tearDownEverything:466] CLEANUP WAS SKIPPED