guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/debian-p0-durability-vset00-00-tunable_rebalance_in_majority_6.5_P0/testexec.95893.ini process_concurrency=2,sdk_timeout=60,num_items=100000,GROUP=P0;durability,EXCLUDE_GROUP=non_dgm;not_for_majority,durability=MAJORITY,active_resident_threshold=70,upgrade_version=7.6.0-2032,sirius_url=http://172.23.120.103:4000 -t rebalance_new.rebalance_in.RebalanceInTests.test_rebalance_in_with_ops_sync_async,nodes_init=2,nodes_in=2,replicas=2,num_items=10000,GROUP=P0;MHbeta;durability;luks' Test Input params: {'conf_file': 'conf/rebalance/rebalance_in.conf', 'process_concurrency': '2', 'upgrade_version': '7.6.0-2032', 'nodes_in': '2', 'spec': 'rebalance_in', 'num_nodes': 7, 'rerun': False, 'GROUP': 'P0;durability', 'EXCLUDE_GROUP': 'non_dgm;not_for_majority', 'sdk_timeout': '60', 'case_number': 12, 'cluster_name': 'testexec.95893', 'ini': '/data/workspace/debian-p0-durability-vset00-00-tunable_rebalance_in_majority_6.5_P0/testexec.95893.ini', 'replicas': '2', 'durability': 'MAJORITY', 'active_resident_threshold': '70', 'logs_folder': '/data/workspace/debian-p0-durability-vset00-00-tunable_rebalance_in_majority_6.5_P0/logs/testrunner-24-Jan-20_20-53-08/test_12', 'nodes_init': '2', 'num_items': '100000', 'sirius_url': 'http://172.23.120.103:4000'} test_rebalance_in_with_ops_sync_async (rebalance_new.rebalance_in.RebalanceInTests) ... 2024-01-20 22:15:33,609 | test | INFO | MainThread | [cb_basetest:log_setup_status:214] ========= RebalanceInTests setup started for test #12 test_rebalance_in_with_ops_sync_async ========= 2024-01-20 22:15:36,483 | test | INFO | MainThread | [cb_basetest:log_setup_status:214] ========= OnPremBaseTest setup started for test #12 test_rebalance_in_with_ops_sync_async ========= 2024-01-20 22:15:36,694 | test | INFO | MainThread | [onPrem_basetestcase:setUp:251] Delete all buckets and rebalance out other nodes from 'C1' 2024-01-20 22:15:53,908 | test | INFO | MainThread | [rest_client:monitorRebalance:1403] Rebalance done. Taken 11.513999939 seconds to complete 2024-01-20 22:15:53,910 | test | INFO | MainThread | [common_lib:sleep:20] Sleep 10 seconds. Reason: Wait after rebalance complete 2024-01-20 22:16:48,653 | test | INFO | MainThread | [common_lib:sleep:20] Sleep 5 seconds. Reason: Wait for nodes to become ready after reset 2024-01-20 22:16:53,654 | test | INFO | MainThread | [onPrem_basetestcase:initialize_cluster:413] Initializing cluster : C1 2024-01-20 22:16:55,032 | infra | CRITICAL | pool-19-thread-29 | [task:call:7482] mcdMemoryReserved reported in nodes/self is: 3115 2024-01-20 22:16:55,069 | test | CRITICAL | pool-19-thread-29 | [table_view:display:72] Memory quota allocated: +---------------------+---------+ | Service | RAM MiB | +---------------------+---------+ | eventingMemoryQuota | 256 | | cbasMemoryQuota | 1024 | | indexMemoryQuota | 256 | | ftsMemoryQuota | 256 | | memoryQuota | 3015 | +---------------------+---------+ 2024-01-20 22:16:57,213 | test | WARNING | MainThread | [onPrem_basetestcase:_initialize_nodes:683] RAM quota was defined less than 100 MB: 2024-01-20 22:17:08,332 | test | INFO | MainThread | [onPrem_basetestcase:initialize_cluster:438] Cluster C1 initialized 2024-01-20 22:17:10,427 | test | INFO | MainThread | [onPrem_basetestcase:enable_tls_on_nodes:497] Validating if services obey tls only on servers [ip:172.23.217.74 port:8091 ssh_username:root, ip:172.23.217.73 port:8091 ssh_username:root, ip:172.23.108.36 port:8091 ssh_username:root, ip:172.23.217.248 port:8091 ssh_username:root, ip:172.23.108.54 port:8091 ssh_username:root, ip:172.23.108.57 port:8091 ssh_username:root, ip:172.23.120.126 port:8091 ssh_username:root] 2024-01-20 22:17:11,032 | test | INFO | MainThread | [cb_basetest:log_setup_status:214] ========= OnPremBaseTest setup finished for test #12 test_rebalance_in_with_ops_sync_async ========= 2024-01-20 22:17:11,253 | test | INFO | MainThread | [cb_basetest:log_setup_status:214] ========= OnPremBaseTest setup finished for test #12 test_rebalance_in_with_ops_sync_async ========= 2024-01-20 22:17:19,868 | test | CRITICAL | pool-19-thread-7 | [cluster_ready_functions:validate_orchestrator_selection:472] Orchestrator: 172.23.217.74 2024-01-20 22:17:20,138 | test | INFO | pool-19-thread-7 | [table_view:display:72] Rebalance Overview +---------------+---------+----------+---------------------------------+---------------+--------------+-----------------------+ | Nodes | Zone | Services | Version / Config | CPU | Status | Membership / Recovery | +---------------+---------+----------+---------------------------------+---------------+--------------+-----------------------+ | 172.23.217.74 | Group 1 | kv | 7.6.0-2032-enterprise / default | 3.34999999963 | Cluster node | active / none | | 172.23.217.73 | None | kv | | | <--- IN --- | | +---------------+---------+----------+---------------------------------+---------------+--------------+-----------------------+ 2024-01-20 22:17:20,348 | test | INFO | pool-19-thread-7 | [task:check:862] Rebalance - status: running, progress: 0.0 2024-01-20 22:17:35,665 | test | INFO | pool-19-thread-7 | [task:check:862] Rebalance - status: none, progress: 100 2024-01-20 22:17:35,772 | test | INFO | pool-19-thread-7 | [task:check:929] Rebalance completed with progress: 100% in 15.6460001469 sec 2024-01-20 22:17:36,415 | test | CRITICAL | pool-19-thread-7 | [cluster_ready_functions:validate_orchestrator_selection:472] Orchestrator: 172.23.217.74 2024-01-20 22:17:37,029 | infra | CRITICAL | pool-19-thread-7 | [task:print_nodes:627] Cluster nodes..: ['172.23.217.74:18091', '172.23.217.73:18091'] 2024-01-20 22:17:37,029 | infra | CRITICAL | pool-19-thread-7 | [task:print_nodes:627] KV............: ['172.23.217.74:18091', '172.23.217.73:18091'] 2024-01-20 22:17:40,691 | test | INFO | MainThread | [common_lib:sleep:20] Sleep 2 seconds. Reason: Bucket default is not ready for persistence on 172.23.217.73 2024-01-20 22:17:43,051 | test | INFO | MainThread | [common_lib:sleep:20] Sleep 2 seconds. Reason: Bucket default is not ready for persistence on 172.23.217.73 2024-01-20 22:17:46,967 | test | INFO | pool-19-thread-4 | [task:call:3024] Starting DGM doc loading task 2024-01-20 22:17:47,256 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3008] DGM doc loading for 'default' to atleast 70% 2024-01-20 22:17:47,256 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:17:47,256 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:17:48,826 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:17:48,828 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:17:50,361 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:17:50,361 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:17:51,762 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:17:51,763 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:17:53,443 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:17:53,444 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:17:54,980 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:17:54,980 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:17:56,404 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:17:56,404 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:17:57,835 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:17:57,835 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:17:59,315 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:17:59,315 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:00,729 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:00,730 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:02,318 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:02,318 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:03,663 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:03,664 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:05,107 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:05,107 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:06,453 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:06,453 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:08,101 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:08,101 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:09,627 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:09,627 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:11,118 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:11,118 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:12,634 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:12,635 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:14,167 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:14,167 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:15,749 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:15,749 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:17,267 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3012] Active_resident_items_ratio for default is 100 2024-01-20 22:18:17,269 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3014] Replica_resident_items_ratio for default is 100 2024-01-20 22:18:18,710 | test | INFO | pool-19-thread-4 | [task:_load_bucket_into_dgm:3018] Active DGM 97.4763177512% Replica DGM 60.7437512948% achieved for 'default'. Loaded docs: 210000 2024-01-20 22:18:18,822 | test | INFO | pool-19-thread-4 | [task:call:3041] Done loading docs for DGM 2024-01-20 22:18:28,934 | test | INFO | MainThread | [rebalance_base:setUp:154] Verifying num_items counts after doc_ops 2024-01-20 22:18:31,634 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:5720] Validating collection stats and item counts 2024-01-20 22:18:35,154 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +---------------+---------+----------+--------+-----------+----------+-------------------------+-------------------+---------------------------------+ | Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config | +---------------+---------+----------+--------+-----------+----------+-------------------------+-------------------+---------------------------------+ | 172.23.217.74 | Group 1 | kv | 61.025 | 3.80 GiB | 2.43 GiB | 1.00 MiB / 976.00 MiB | 105000 / 105000 | 7.6.0-2032-enterprise / default | | 172.23.217.73 | Group 1 | kv | 60.625 | 3.80 GiB | 2.08 GiB | 133.04 MiB / 976.00 MiB | 105000 / 105000 | 7.6.0-2032-enterprise / default | +---------------+---------+----------+--------+-----------+----------+-------------------------+-------------------+---------------------------------+ 2024-01-20 22:18:35,686 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-------------------+----------+------+----------+------------+-----+--------+-------------------------+------------+--------------+ | Bucket | Type / Storage | Replicas | Rank | Vbuckets | Durability | TTL | Items | RAM Quota / Used | Disk Used | ARR | +---------+-------------------+----------+------+----------+------------+-----+--------+-------------------------+------------+--------------+ | default | couchbase / magma | 2 | None | 1024 | none | 0 | 210000 | 512.00 MiB / 298.10 MiB | 564.22 MiB | 96.239047619 | +---------+-------------------+----------+------+----------+------------+-----+--------+-------------------------+------------+--------------+ 2024-01-20 22:18:35,687 | test | INFO | MainThread | [cb_basetest:log_setup_status:214] ========= RebalanceBase setup complete for test #12 test_rebalance_in_with_ops_sync_async ========= 2024-01-20 22:18:36,177 | test | INFO | MainThread | [common_lib:sleep:20] Sleep 10 seconds. Reason: wait for rebalance to start 2024-01-20 22:18:52,240 | test | CRITICAL | pool-19-thread-20 | [cluster_ready_functions:validate_orchestrator_selection:472] Orchestrator: 172.23.217.74 2024-01-20 22:18:52,598 | test | INFO | pool-19-thread-20 | [table_view:display:72] Rebalance Overview +----------------+---------+----------+---------------------------------+---------------+--------------+-----------------------+ | Nodes | Zone | Services | Version / Config | CPU | Status | Membership / Recovery | +----------------+---------+----------+---------------------------------+---------------+--------------+-----------------------+ | 172.23.217.74 | Group 1 | kv | 7.6.0-2032-enterprise / default | 61.0250000004 | Cluster node | active / none | | 172.23.217.73 | Group 1 | kv | 7.6.0-2032-enterprise / default | 60.625 | Cluster node | active / none | | 172.23.108.36 | None | kv | | | <--- IN --- | | | 172.23.217.248 | None | kv | | | <--- IN --- | | +----------------+---------+----------+---------------------------------+---------------+--------------+-----------------------+ 2024-01-20 22:18:52,844 | test | INFO | pool-19-thread-20 | [task:check:862] Rebalance - status: running, progress: 0.0 2024-01-20 22:18:57,434 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-253895 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/00000000929D7A23","lastDispatchedFrom":"172.23.123.69:35852","lastDispatchedTo":"172.23.217.248:11207","requestId":6405118,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-253895","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x63077a","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":481},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":2287,"totalDispatchMicros":3349,"totalServerMicros":0,"totalMicros":9521,"serverMicros":0}}', 'cas': 0, 'value': {}} | | test_docs-253905 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/00000000929D7A23","lastDispatchedFrom":"172.23.123.69:35852","lastDispatchedTo":"172.23.217.248:11207","requestId":6405120,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-253905","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x63077c","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":481},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":2008,"totalDispatchMicros":3997,"totalServerMicros":0,"totalMicros":9298,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:18:57,434 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-253895 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/00000000929D7A23","lastDispatchedFrom":"172.23.123.69:35852","lastDispatchedTo":"172.23.217.248:11207","requestId":6404297,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":135,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-253895","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x630445","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":481},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":918,"encodingMicros":3,"totalDispatchMicros":1452,"totalServerMicros":0,"totalMicros":11005,"serverMicros":0}} | | test_docs-253905 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/00000000929D7A23","lastDispatchedFrom":"172.23.123.69:35852","lastDispatchedTo":"172.23.217.248:11207","requestId":6404318,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":32,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-253905","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x63045a","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":481},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":1280,"encodingMicros":4,"totalDispatchMicros":1698,"totalServerMicros":0,"totalMicros":5637,"serverMicros":0}} | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:18:57,992 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-255368 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/000000000AB6F463","lastDispatchedFrom":"172.23.123.69:35870","lastDispatchedTo":"172.23.217.248:11207","requestId":6409404,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-255368","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x631838","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":502},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":1077,"totalDispatchMicros":2679,"totalServerMicros":0,"totalMicros":7932,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:18:57,993 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-255368 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/000000000AB6F463","lastDispatchedFrom":"172.23.123.69:35870","lastDispatchedTo":"172.23.217.248:11207","requestId":6407814,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":100,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-255368","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x631202","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":502},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":770,"encodingMicros":2,"totalDispatchMicros":51351,"totalServerMicros":42,"totalMicros":59249,"serverMicros":0}} | +------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:18:58,424 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-361389 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000091","idempotent":true,"lastChannelId":"F5110A2600000091/00000000358A068F","lastDispatchedFrom":"172.23.123.69:45962","lastDispatchedTo":"172.23.108.36:11207","requestId":6412529,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-361389","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x63246d","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":1004},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":987,"totalDispatchMicros":3795,"totalServerMicros":0,"totalMicros":10149,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:18:58,424 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-361389 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000091","idempotent":false,"lastChannelId":"F5110A2600000091/00000000358A068F","lastDispatchedFrom":"172.23.123.69:45962","lastDispatchedTo":"172.23.108.36:11207","requestId":6408915,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":92,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-361389","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x63164f","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":1004},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":583,"encodingMicros":4,"totalDispatchMicros":24816,"totalServerMicros":16,"totalMicros":33098,"serverMicros":0}} | +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:18:59,681 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-263724 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/00000000DCC8B70B","lastDispatchedFrom":"172.23.123.69:45884","lastDispatchedTo":"172.23.108.36:11207","requestId":6425918,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-263724","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x6358ba","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":494},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":708,"totalDispatchMicros":6480,"totalServerMicros":0,"totalMicros":10989,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:18:59,683 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-263724 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/00000000DCC8B70B","lastDispatchedFrom":"172.23.123.69:45884","lastDispatchedTo":"172.23.108.36:11207","requestId":6424261,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":92,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-263724","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x635241","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":494},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":769,"encodingMicros":24,"totalDispatchMicros":6344,"totalServerMicros":0,"totalMicros":17323,"serverMicros":0}} | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:18:59,944 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-369164 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000091","idempotent":true,"lastChannelId":"F5110A2600000091/0000000092E2206B","lastDispatchedFrom":"172.23.123.69:45996","lastDispatchedTo":"172.23.108.36:11207","requestId":6428155,"requestType":"GetRequest","retried":0,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-369164","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x636177","scope":"_default","type":"kv","vbucket":494},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":1031,"totalDispatchMicros":1031,"totalServerMicros":0,"totalMicros":1218,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:18:59,944 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-369164 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000091","idempotent":false,"lastChannelId":"F5110A2600000091/0000000092E2206B","lastDispatchedFrom":"172.23.123.69:45996","lastDispatchedTo":"172.23.108.36:11207","requestId":6424701,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":76,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-369164","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x6353f9","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":494},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":630,"encodingMicros":2,"totalDispatchMicros":1557,"totalServerMicros":0,"totalMicros":8367,"serverMicros":0}} | +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:00,480 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-371842 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000091","idempotent":true,"lastChannelId":"F5110A2600000091/0000000063416465","lastDispatchedFrom":"172.23.123.69:36000","lastDispatchedTo":"172.23.217.248:11207","requestId":6432543,"requestType":"GetRequest","retried":0,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-371842","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x63729b","scope":"_default","type":"kv","vbucket":446},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":941,"totalDispatchMicros":941,"totalServerMicros":0,"totalMicros":1139,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:00,480 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-371842 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000091","idempotent":false,"lastChannelId":"F5110A2600000091/0000000063416465","lastDispatchedFrom":"172.23.123.69:36000","lastDispatchedTo":"172.23.217.248:11207","requestId":6429383,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":84,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-371842","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x636643","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":446},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":830,"encodingMicros":3,"totalDispatchMicros":1380,"totalServerMicros":0,"totalMicros":7010,"serverMicros":0}} | +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:00,558 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-266122 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/0000000002617F12","lastDispatchedFrom":"172.23.123.69:45874","lastDispatchedTo":"172.23.108.36:11207","requestId":6433335,"requestType":"GetRequest","retried":0,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-266122","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x6375b3","scope":"_default","type":"kv","vbucket":990},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":981,"totalDispatchMicros":981,"totalServerMicros":0,"totalMicros":1164,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:00,559 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-266122 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/0000000002617F12","lastDispatchedFrom":"172.23.123.69:45874","lastDispatchedTo":"172.23.108.36:11207","requestId":6429761,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":49,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-266122","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x6367bd","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":990},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":534,"encodingMicros":3,"totalDispatchMicros":5717,"totalServerMicros":0,"totalMicros":13669,"serverMicros":0}} | +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:01,111 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-269288 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/000000002ECD8942","lastDispatchedFrom":"172.23.123.69:35902","lastDispatchedTo":"172.23.217.248:11207","requestId":6437836,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-269288","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x638748","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":961},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":851,"totalDispatchMicros":2768,"totalServerMicros":0,"totalMicros":4722,"serverMicros":0}}', 'cas': 0, 'value': {}} | | test_docs-269318 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/000000002ECD8942","lastDispatchedFrom":"172.23.123.69:35902","lastDispatchedTo":"172.23.217.248:11207","requestId":6437837,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-269318","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x638749","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":961},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":783,"totalDispatchMicros":3708,"totalServerMicros":0,"totalMicros":14689,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:01,112 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-269288 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/000000002ECD8942","lastDispatchedFrom":"172.23.123.69:35902","lastDispatchedTo":"172.23.217.248:11207","requestId":6435835,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":56,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-269288","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x637f77","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":961},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":864,"encodingMicros":2,"totalDispatchMicros":1496,"totalServerMicros":0,"totalMicros":8358,"serverMicros":0}} | | test_docs-269318 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/000000002ECD8942","lastDispatchedFrom":"172.23.123.69:35902","lastDispatchedTo":"172.23.217.248:11207","requestId":6435865,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":69,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-269318","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x637f95","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":961},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":694,"encodingMicros":3,"totalDispatchMicros":1347,"totalServerMicros":0,"totalMicros":12725,"serverMicros":0}} | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:01,631 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-270147 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/00000000149085D3","lastDispatchedFrom":"172.23.123.69:45866","lastDispatchedTo":"172.23.108.36:11207","requestId":6441920,"requestType":"GetRequest","retried":0,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-270147","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x63973c","scope":"_default","type":"kv","vbucket":485},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":866,"totalDispatchMicros":866,"totalServerMicros":0,"totalMicros":972,"serverMicros":0}}', 'cas': 0, 'value': {}} | | test_docs-270787 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/0000000008ACC169","lastDispatchedFrom":"172.23.123.69:45851","lastDispatchedTo":"172.23.108.36:11207","requestId":6441922,"requestType":"GetRequest","retried":0,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-270787","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x63973e","scope":"_default","type":"kv","vbucket":477},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":624,"totalDispatchMicros":624,"totalServerMicros":0,"totalMicros":788,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:01,632 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-270147 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/00000000149085D3","lastDispatchedFrom":"172.23.123.69:45866","lastDispatchedTo":"172.23.108.36:11207","requestId":6438191,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":62,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-270147","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x6388ab","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":485},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":824,"encodingMicros":2,"totalDispatchMicros":2389,"totalServerMicros":0,"totalMicros":9037,"serverMicros":0}} | | test_docs-270787 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/0000000008ACC169","lastDispatchedFrom":"172.23.123.69:45851","lastDispatchedTo":"172.23.108.36:11207","requestId":6439340,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":49,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-270787","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x638d28","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":477},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":570,"encodingMicros":3,"totalDispatchMicros":1132,"totalServerMicros":0,"totalMicros":4672,"serverMicros":0}} | +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:01,924 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-378473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000091","idempotent":true,"lastChannelId":"F5110A2600000091/0000000087E52A18","lastDispatchedFrom":"172.23.123.69:35998","lastDispatchedTo":"172.23.217.248:11207","requestId":6444562,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-378473","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x63a18e","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":390},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":2095,"totalDispatchMicros":2782,"totalServerMicros":0,"totalMicros":6599,"serverMicros":0}}', 'cas': 0, 'value': {}} | | test_docs-378301 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000091","idempotent":true,"lastChannelId":"F5110A2600000091/0000000087E52A18","lastDispatchedFrom":"172.23.123.69:35998","lastDispatchedTo":"172.23.217.248:11207","requestId":6444563,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-378301","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x63a18f","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":390},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":2040,"totalDispatchMicros":3609,"totalServerMicros":0,"totalMicros":6431,"serverMicros":0}}', 'cas': 0, 'value': {}} | | test_docs-378291 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000091","idempotent":true,"lastChannelId":"F5110A2600000091/0000000087E52A18","lastDispatchedFrom":"172.23.123.69:35998","lastDispatchedTo":"172.23.217.248:11207","requestId":6444564,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-378291","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x63a190","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":390},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":1902,"totalDispatchMicros":4188,"totalServerMicros":0,"totalMicros":6342,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:01,926 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-378473 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000091","idempotent":false,"lastChannelId":"F5110A2600000091/0000000087E52A18","lastDispatchedFrom":"172.23.123.69:35998","lastDispatchedTo":"172.23.217.248:11207","requestId":6442139,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":38,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-378473","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x639817","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":390},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":511,"encodingMicros":5,"totalDispatchMicros":1243,"totalServerMicros":0,"totalMicros":10092,"serverMicros":0}} | | test_docs-378301 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000091","idempotent":false,"lastChannelId":"F5110A2600000091/0000000087E52A18","lastDispatchedFrom":"172.23.123.69:35998","lastDispatchedTo":"172.23.217.248:11207","requestId":6441856,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":15,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-378301","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x6396fc","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":390},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":712,"encodingMicros":2,"totalDispatchMicros":1392,"totalServerMicros":0,"totalMicros":9900,"serverMicros":0}} | | test_docs-378291 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000091","idempotent":false,"lastChannelId":"F5110A2600000091/0000000087E52A18","lastDispatchedFrom":"172.23.123.69:35998","lastDispatchedTo":"172.23.217.248:11207","requestId":6441846,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":43,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-378291","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x6396f2","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":390},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":557,"encodingMicros":3,"totalDispatchMicros":1553,"totalServerMicros":0,"totalMicros":11497,"serverMicros":0}} | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:01,993 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-272233 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/000000007AAB8B65","lastDispatchedFrom":"172.23.123.69:35880","lastDispatchedTo":"172.23.217.248:11207","requestId":6445150,"requestType":"GetRequest","retried":0,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-272233","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x63a3da","scope":"_default","type":"kv","vbucket":390},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":679,"totalDispatchMicros":679,"totalServerMicros":0,"totalMicros":904,"serverMicros":0}}', 'cas': 0, 'value': {}} | | test_docs-273754 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/0000000042C5F615","lastDispatchedFrom":"172.23.123.69:45934","lastDispatchedTo":"172.23.108.36:11207","requestId":6445151,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-273754","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x63a3db","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":975},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":1447,"totalDispatchMicros":5244,"totalServerMicros":0,"totalMicros":14117,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:01,993 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-272233 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/000000007AAB8B65","lastDispatchedFrom":"172.23.123.69:35880","lastDispatchedTo":"172.23.217.248:11207","requestId":6442274,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":32,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-272233","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x63989e","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":390},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":504,"encodingMicros":1,"totalDispatchMicros":6580,"totalServerMicros":0,"totalMicros":16412,"serverMicros":0}} | | test_docs-273754 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/0000000042C5F615","lastDispatchedFrom":"172.23.123.69:45934","lastDispatchedTo":"172.23.108.36:11207","requestId":6444316,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":62,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-273754","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x63a098","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":975},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":571,"encodingMicros":5,"totalDispatchMicros":5894,"totalServerMicros":0,"totalMicros":17682,"serverMicros":0}} | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:03,112 | test | INFO | pool-19-thread-20 | [task:check:862] Rebalance - status: running, progress: 2.47 2024-01-20 22:19:04,845 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-391713 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000091","idempotent":true,"lastChannelId":"F5110A2600000091/0000000047D8B575","lastDispatchedFrom":"172.23.123.69:35956","lastDispatchedTo":"172.23.217.248:11207","requestId":6472581,"requestType":"GetRequest","retried":0,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-391713","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x640f01","scope":"_default","type":"kv","vbucket":930},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":1485,"totalDispatchMicros":1485,"totalServerMicros":0,"totalMicros":1611,"serverMicros":0}}', 'cas': 0, 'value': {}} | | test_docs-391683 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000091","idempotent":true,"lastChannelId":"F5110A2600000091/0000000047D8B575","lastDispatchedFrom":"172.23.123.69:35956","lastDispatchedTo":"172.23.217.248:11207","requestId":6472582,"requestType":"GetRequest","retried":0,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-391683","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x640f02","scope":"_default","type":"kv","vbucket":930},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":1311,"totalDispatchMicros":1311,"totalServerMicros":0,"totalMicros":1482,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:04,845 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-391713 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000091","idempotent":false,"lastChannelId":"F5110A2600000091/0000000047D8B575","lastDispatchedFrom":"172.23.123.69:35956","lastDispatchedTo":"172.23.217.248:11207","requestId":6469802,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":32,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-391713","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x640426","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":930},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":444,"encodingMicros":4,"totalDispatchMicros":905,"totalServerMicros":0,"totalMicros":7663,"serverMicros":0}} | | test_docs-391683 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000091","idempotent":false,"lastChannelId":"F5110A2600000091/0000000047D8B575","lastDispatchedFrom":"172.23.123.69:35956","lastDispatchedTo":"172.23.217.248:11207","requestId":6469763,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":38,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-391683","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x6403ff","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":930},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":1128,"encodingMicros":2,"totalDispatchMicros":1559,"totalServerMicros":0,"totalMicros":3310,"serverMicros":0}} | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:04,872 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-286665 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/0000000074D1834C","lastDispatchedFrom":"172.23.123.69:35858","lastDispatchedTo":"172.23.217.248:11207","requestId":6472839,"requestType":"GetRequest","retried":0,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-286665","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x641003","scope":"_default","type":"kv","vbucket":937},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":1510,"totalDispatchMicros":1510,"totalServerMicros":0,"totalMicros":1713,"serverMicros":0}}', 'cas': 0, 'value': {}} | | test_docs-287752 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000090","idempotent":true,"lastChannelId":"F5110A2600000090/000000003EBFEE30","lastDispatchedFrom":"172.23.123.69:35862","lastDispatchedTo":"172.23.217.248:11207","requestId":6472840,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-287752","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x641004","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":926},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":1482,"totalDispatchMicros":4952,"totalServerMicros":0,"totalMicros":14695,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:04,874 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:_default +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-286665 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/0000000074D1834C","lastDispatchedFrom":"172.23.123.69:35858","lastDispatchedTo":"172.23.217.248:11207","requestId":6470117,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":27,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-286665","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x640561","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":937},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":435,"encodingMicros":2,"totalDispatchMicros":4172,"totalServerMicros":0,"totalMicros":12476,"serverMicros":0}} | | test_docs-287752 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000090","idempotent":false,"lastChannelId":"F5110A2600000090/000000003EBFEE30","lastDispatchedFrom":"172.23.123.69:35862","lastDispatchedTo":"172.23.217.248:11207","requestId":6472328,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":49,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-287752","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x640e04","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":926},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":686,"encodingMicros":2,"totalDispatchMicros":1847,"totalServerMicros":0,"totalMicros":6779,"serverMicros":0}} | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:05,823 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-396862 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000091","idempotent":true,"lastChannelId":"F5110A2600000091/0000000030488DA6","lastDispatchedFrom":"172.23.123.69:35984","lastDispatchedTo":"172.23.217.248:11207","requestId":6481097,"requestType":"GetRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-396862","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x643045","rejectedWithNotMyVbucket":1,"scope":"_default","type":"kv","vbucket":367},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":685,"totalDispatchMicros":1225,"totalServerMicros":0,"totalMicros":7577,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:05,823 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-396862 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000091","idempotent":false,"lastChannelId":"F5110A2600000091/0000000030488DA6","lastDispatchedFrom":"172.23.123.69:35984","lastDispatchedTo":"172.23.217.248:11207","requestId":6479920,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":92,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-396862","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x642bac","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":367},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":878,"encodingMicros":3,"totalDispatchMicros":1440,"totalServerMicros":0,"totalMicros":11542,"serverMicros":0}} | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:13,348 | test | INFO | pool-19-thread-20 | [task:check:862] Rebalance - status: running, progress: 4.17 2024-01-20 22:19:22,766 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-409769 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found {"completed":true,"coreId":"0xf5110a2600000091","idempotent":true,"lastChannelId":"F5110A2600000091/00000000140DCA26","lastDispatchedFrom":"172.23.123.69:45976","lastDispatchedTo":"172.23.108.36:11207","requestId":6718669,"requestType":"GetRequest","retried":0,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-409769","errorCode":{"description":"Not Found","name":"KEY_ENOENT"},"opaque":"0x67d049","scope":"_default","type":"kv","vbucket":407},"status":"NOT_FOUND","timeoutMs":60000,"timings":{"dispatchMicros":1413,"totalDispatchMicros":1413,"totalServerMicros":0,"totalMicros":1647,"serverMicros":0}}', 'cas': 0, 'value': {}} | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:22,766 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_docs-409769 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible {"completed":true,"coreId":"0xf5110a2600000091","idempotent":false,"lastChannelId":"F5110A2600000091/00000000140DCA26","lastDispatchedFrom":"172.23.123.69:45976","lastDispatchedTo":"172.23.108.36:11207","requestId":6580515,"requestType":"InsertRequest","retried":1,"retryReasons":["KV_NOT_MY_VBUCKET"],"serverDuration":32,"service":{"bucket":"default","collection":"_default","documentId":"test_docs-409769","errorCode":{"description":"Durability requirements are impossible to achieve","name":"DurabilityImpossible"},"opaque":"0x65b49f","rejectedWithNotMyVbucket":1,"scope":"_default","syncDurability":"MAJORITY","type":"kv","vbucket":407},"status":"DURABILITY_IMPOSSIBLE","timeoutMs":60000,"timings":{"dispatchMicros":439,"encodingMicros":6,"totalDispatchMicros":4834,"totalServerMicros":0,"totalMicros":16062,"serverMicros":0}} | +------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2024-01-20 22:19:23,602 | test | INFO | pool-19-thread-20 | [task:check:862] Rebalance - status: running, progress: 6.77 2024-01-20 22:19:33,821 | test | ERROR | pool-19-thread-20 | [rest_client:_rebalance_status_and_progress:1470] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'05fde7ee06070213fa0ff74e2f228c61', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=4800a583c71105ff4f643f96656fafe6', u'status': u'notRunning'} - rebalance failed 2024-01-20 22:19:33,961 | test | INFO | pool-19-thread-20 | [rest_client:print_UI_logs:2666] Latest logs from UI on 172.23.217.74: 2024-01-20 22:19:33,963 | test | ERROR | pool-19-thread-20 | [rest_client:print_UI_logs:2668] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.217.74', u'tstamp': 1705817971616L, u'shortText': u'message', u'serverTime': u'2024-01-20T22:19:31.616Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.16391.31>,\n {{dcp_wait_for_data_move_failed,"default",\n 122,\'ns_1@172.23.217.73\',\n [\'ns_1@172.23.108.36\',\'ns_1@172.23.217.74\'],\n {error,\n {unexpected_status,\n <<"connection_does_not_exist">>},\n "Error getting dcp stats on \'ns_1@172.23.217.73\' for bucket \\"default\\", partition 122, connection \\"replication:ns_1@172.23.217.73->ns_1@172.23.217.74:default\\": {unexpected_status,\\n <<\\"connection_does_not_exist\\">>}"}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,453}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,225}]}]}}}}.\nRebalance Operation Id = d2e83b3a39f1566c3c7f16445d4da65e'} 2024-01-20 22:19:33,963 | test | ERROR | pool-19-thread-20 | [rest_client:print_UI_logs:2668] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.217.74', u'tstamp': 1705817971591L, u'shortText': u'message', u'serverTime': u'2024-01-20T22:19:31.591Z', u'text': u'Worker <0.16324.31> (for action {move,{122,\n [\'ns_1@172.23.217.73\',\n \'ns_1@172.23.217.74\',undefined],\n [\'ns_1@172.23.108.36\',\n \'ns_1@172.23.217.74\',\n \'ns_1@172.23.217.73\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.16391.31>,\n {{dcp_wait_for_data_move_failed,\n "default",\n 122,\n \'ns_1@172.23.217.73\',\n [\'ns_1@172.23.108.36\',\n \'ns_1@172.23.217.74\'],\n {error,\n {unexpected_status,\n <<"connection_does_not_exist">>},\n "Error getting dcp stats on \'ns_1@172.23.217.73\' for bucket \\"default\\", partition 122, connection \\"replication:ns_1@172.23.217.73->ns_1@172.23.217.74:default\\": {unexpected_status,\\n <<\\"connection_does_not_exist\\">>}"}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 453}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 225}]}]}}}'} 2024-01-20 22:19:33,963 | test | ERROR | pool-19-thread-20 | [rest_client:print_UI_logs:2668] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.217.73', u'tstamp': 1705817958527L, u'shortText': u'node up', u'serverTime': u'2024-01-20T22:19:18.527Z', u'text': u"Node 'ns_1@172.23.217.73' saw that node 'ns_1@172.23.108.36' came up. Tags: [] (repeated 1 times, last seen 39.206581 secs ago)"} 2024-01-20 22:19:33,963 | test | ERROR | pool-19-thread-20 | [rest_client:print_UI_logs:2668] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.217.74', u'tstamp': 1705817949306L, u'shortText': u'node up', u'serverTime': u'2024-01-20T22:19:09.306Z', u'text': u"Node 'ns_1@172.23.217.74' saw that node 'ns_1@172.23.217.248' came up. Tags: [] (repeated 1 times, last seen 22.072395 secs ago)"} 2024-01-20 22:19:33,963 | test | ERROR | pool-19-thread-20 | [rest_client:print_UI_logs:2668] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.217.74', u'tstamp': 1705817949306L, u'shortText': u'node up', u'serverTime': u'2024-01-20T22:19:09.306Z', u'text': u"Node 'ns_1@172.23.217.74' saw that node 'ns_1@172.23.108.36' came up. Tags: [] (repeated 1 times, last seen 29.764997 secs ago)"} 2024-01-20 22:19:33,964 | test | ERROR | pool-19-thread-20 | [rest_client:print_UI_logs:2668] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.217.74', u'tstamp': 1705817936474L, u'shortText': u'message', u'serverTime': u'2024-01-20T22:18:56.474Z', u'text': u'Bucket "default" rebalance does not seem to be swap rebalance'} 2024-01-20 22:19:33,964 | test | ERROR | pool-19-thread-20 | [rest_client:print_UI_logs:2668] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.217.248', u'tstamp': 1705817933409L, u'shortText': u'message', u'serverTime': u'2024-01-20T22:18:53.409Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.217.248\' in 0 seconds.'} 2024-01-20 22:19:33,964 | test | ERROR | pool-19-thread-20 | [rest_client:print_UI_logs:2668] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.108.36', u'tstamp': 1705817933402L, u'shortText': u'message', u'serverTime': u'2024-01-20T22:18:53.402Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.108.36\' in 0 seconds.'} 2024-01-20 22:19:33,964 | test | ERROR | pool-19-thread-20 | [rest_client:print_UI_logs:2668] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.217.74', u'tstamp': 1705817932930L, u'shortText': u'message', u'serverTime': u'2024-01-20T22:18:52.930Z', u'text': u'Started rebalancing bucket default'} 2024-01-20 22:19:33,964 | test | ERROR | pool-19-thread-20 | [rest_client:print_UI_logs:2668] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.217.74', u'tstamp': 1705817932485L, u'shortText': u'message', u'serverTime': u'2024-01-20T22:18:52.485Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.217.74','ns_1@172.23.217.248',\n 'ns_1@172.23.108.36','ns_1@172.23.217.73'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = d2e83b3a39f1566c3c7f16445d4da65e"} 2024-01-20 22:19:33,966 | test | ERROR | pool-19-thread-20 | [task:call:705] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'05fde7ee06070213fa0ff74e2f228c61', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=4800a583c71105ff4f643f96656fafe6', u'status': u'notRunning'} - rebalance failed 2024-01-20 22:19:33,967 | test | ERROR | pool-19-thread-22 | [task:call:2535] Failed to load 13 docs from 210000 to 315000 FAIL 2024-01-20 22:19:34,380 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+---------+----------+--------+-----------+----------+-------------------------+-------------------+---------------------------------+ | Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config | +----------------+---------+----------+--------+-----------+----------+-------------------------+-------------------+---------------------------------+ | 172.23.217.74 | Group 1 | kv | 88.75 | 3.80 GiB | 2.21 GiB | 1.00 MiB / 976.00 MiB | 270631 / 299515 | 7.6.0-2032-enterprise / default | | 172.23.217.248 | Group 1 | kv | 22.740 | 3.83 GiB | 2.75 GiB | 38.00 MiB / 3.99 GiB | 15123 / 0 | 7.6.0-2032-enterprise / default | | 172.23.108.36 | Group 1 | kv | 95.029 | 3.80 GiB | 2.61 GiB | 71.50 MiB / 976.00 MiB | 21150 / 573 | 7.6.0-2032-enterprise / default | | 172.23.217.73 | Group 1 | kv | 66.950 | 3.80 GiB | 1.97 GiB | 133.04 MiB / 976.00 MiB | 287796 / 324655 | 7.6.0-2032-enterprise / default | +----------------+---------+----------+--------+-----------+----------+-------------------------+-------------------+---------------------------------+ 2024-01-20 22:19:37,961 | test | INFO | MainThread | [onPrem_basetestcase:check_logs:782] Comparing timestamps: Log's latest timestamp: 2024-01-20 22:04:41, Test's start timestamp is 2024-01-20 22:15:33.280000 2024-01-20 22:19:37,963 | test | INFO | MainThread | [onPrem_basetestcase:check_logs:782] Comparing timestamps: Log's latest timestamp: 2024-01-20 22:04:41, Test's start timestamp is 2024-01-20 22:15:33.280000 2024-01-20 22:19:38,269 | test | INFO | MainThread | [onPrem_basetestcase:check_logs:782] Comparing timestamps: Log's latest timestamp: 2024-01-20 22:19:30, Test's start timestamp is 2024-01-20 22:15:33.280000 2024-01-20 22:19:38,269 | test | INFO | MainThread | [onPrem_basetestcase:check_logs:782] Comparing timestamps: Log's latest timestamp: 2024-01-20 22:19:30, Test's start timestamp is 2024-01-20 22:15:33.280000 2024-01-20 22:19:38,269 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:897] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2024-01-20 22:19:38,269 | test | CRITICAL | MainThread | [onPrem_basetestcase:check_coredump_exist:899] 172.23.217.73: Found ' ERROR ' logs - ['2024-01-20T22:19:30.415852-08:00 ERROR 193: Exception occurred during packet execution. Closing connection [ {"ip":"172.23.217.74","port":55540} - {"ip":"172.23.217.73","port":11206} (System, @ns_server) ]: ThrowExceptionUnderflowPolicy current:270108 arg:279424. Cookies: [{"aiostat":"success","ewouldblock":false,"packet":{"bodylen":4,"cas":0,"datatype":"raw","extlen":4,"extras":{"buffer_bytes":279424},"keylen":0,"magic":"ClientRequest","opaque":627,"opcode":"DCP_BUFFER_ACKNOWLEDGEMENT","vbucket":0},"refcount":1,"started":"4612621023688670 (5119 us ago)","throttled":false}] Exception thrown from: ["#0 /opt/couchbase/bin/memcached() [0x400000+0x1443b9]","#1 /opt/couchbase/bin/memcached() [0x400000+0x4b5ee7]","#2 /opt/couchbase/bin/memcached() [0x400000+0x4b684d]","#3 /opt/couchbase/bin/memcached() [0x400000+0x4b6a35]","#4 /opt/couchbase/bin/memcached() [0x400000+0x4b6d25]","#5 /opt/couchbase/bin/memcached() [0x400000+0x343584]","#6 /opt/couchbase/bin/memcached() [0x400000+0x1d333d]","#7 /opt/couchbase/bin/memcached() [0x400000+0x2a6e30]","#8 /opt/couchbase/bin/memcached() [0x400000+0x22243b]","#9 /opt/couchbase/bin/memcached() [0x400000+0x208b98]","#10 /opt/couchbase/bin/memcached() [0x400000+0x20d9f5]","#11 /opt/couchbase/bin/memcached() [0x400000+0x217f27]","#12 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7feecbe64000+0xf84e]","#13 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7feecbe64000+0x18b99]","#14 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x357) [0x7feecbe64000+0x19287]","#15 /opt/couchbase/bin/memcached() [0x400000+0x981a96]","#16 /opt/couchbase/bin/memcached() [0x400000+0x981f9e]","#17 /opt/couchbase/bin/memcached() [0x400000+0x984468]","#18 /opt/couchbase/bin/memcached() [0x400000+0x231a29]","#19 /opt/couchbase/bin/memcached() [0x400000+0x8d156c]","#20 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7feecb0af000+0xe4aa3]","#21 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7feecb80c000+0x7ea7]","#22 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7feecad70000+0xfba2f]"]\n'] 2024-01-20 22:19:41,153 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1635] Running cbcollect on node ns_1@172.23.108.36,ns_1@172.23.217.248,ns_1@172.23.217.73,ns_1@172.23.217.74 2024-01-20 22:19:51,178 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1638] ns_1@172.23.108.36,ns_1@172.23.217.248,ns_1@172.23.217.73,ns_1@172.23.217.74 - cbcollect status: True 2024-01-20 22:19:51,178 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1643] Polling active_tasks to check cbcollect status 2024-01-20 22:23:47,463 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1677] ns_1@172.23.108.36: Copying cbcollect ZIP file to Client 2024-01-20 22:23:47,976 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1677] ns_1@172.23.217.248: Copying cbcollect ZIP file to Client 2024-01-20 22:23:48,367 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1677] ns_1@172.23.217.73: Copying cbcollect ZIP file to Client 2024-01-20 22:23:49,029 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1677] ns_1@172.23.217.74: Copying cbcollect ZIP file to Client 2024-01-20 22:23:49,760 | test | CRITICAL | MainThread | [onPrem_basetestcase:tearDownEverything:601] Skipping get_trace !! ERROR ====================================================================== ERROR: test_rebalance_in_with_ops_sync_async (rebalance_new.rebalance_in.RebalanceInTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/rebalance_new/rebalance_in.py", line 22, in tearDown super(RebalanceInTests, self).tearDown() File "pytests/rebalance_new/rebalance_base.py", line 249, in tearDown super(RebalanceBaseTest, self).tearDown() File "pytests/onPrem_basetestcase.py", line 571, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ====================================================================== FAIL: test_rebalance_in_with_ops_sync_async (rebalance_new.rebalance_in.RebalanceInTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/rebalance_new/rebalance_in.py", line 184, in test_rebalance_in_with_ops_sync_async self.fail("Rebalance Failed") During the test, AssertionError: Rebalance Failed Remote Connections: 64, Disconnections: 50 SDK Connections: 7, Disconnections: 11 !!!!!! CRITICAL :: Shell disconnection mismatch !!!!! !!!!!! CRITICAL :: SDK disconnection mismatch !!!!! ---------------------------------------------------------------------- Ran 1 test in 496.535s FAILED (failures=1, errors=1) summary so far suite rebalance_new.rebalance_in.RebalanceInTests , pass 10 , fail 2