./testrunner -i node.ini threads_to_use=8,rerun=False,skip_cleanup=True,get-cbcollect-info=False,skip_collections_cleanup=True -t Atomicity.doc_isolation.IsolationDocTest.test_staged_doc_read,nodes_init=2,num_items=1,replicas=1,transaction_timeout=30,transaction_commit=True,doc_op=create,operation=beforeAtrPending,GROUP=P0 Test Input params: {'cluster_name': 'node', 'ini': 'node.ini', 'get-cbcollect-info': 'False', 'replicas': '1', 'transaction_timeout': '30', 'doc_op': 'create', 'conf_file': 'debug.conf', 'threads_to_use': '8', 'spec': 'debug', 'rerun': 'False', 'num_nodes': 2, 'logs_folder': '/Users/ashwin/gitRepos/couchbaselabs/TAF/logs/testrunner-20-Jun-25_01-21-47/test_1', 'skip_cleanup': 'True', 'nodes_init': '2', 'GROUP': 'P0', 'num_items': '1', 'skip_collections_cleanup': 'True', 'transaction_commit': 'True', 'case_number': 1, 'operation': 'beforeAtrPending'} 2020-06-25 01:21:47,944 | root | WARNING | MainThread | [data_helper::31] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially 2020-06-25 01:21:48,454 | test | INFO | MainThread | [basetestcase:_BaseTestCase__log_setup_status:412] ========= basetestcase setup started for test #1 test_staged_doc_read ========= 2020-06-25 01:21:49,447 | test | INFO | MainThread | [basetestcase:_BaseTestCase__log_setup_status:412] ========= basetestcase setup started for test #1 test_staged_doc_read ========= 2020-06-25 01:22:15,980 | test | INFO | MainThread | [rest_client:monitorRebalance:1423] Rebalance done. Taken 25.1029999256 seconds to complete 2020-06-25 01:22:15,982 | test | INFO | MainThread | [rest_client:monitorRebalance:1425] Sleep for 5 seconds after rebalance 2020-06-25 01:22:21,101 | test | WARNING | MainThread | [basetestcase:tearDownEverything:351] CLEANUP WAS SKIPPED 2020-06-25 01:22:21,101 | test | INFO | MainThread | [basetestcase:setUp:266] Initializing cluster 2020-06-25 01:22:21,319 | test | ERROR | pool-2-thread-1 | [rest_client:_http_request:737] POST http://10.112.191.105:8091//node/controller/setupServices body: services=kv&hostname=10.112.191.105&password=password&user=Administrator headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown ["cannot change node services after cluster is provisioned"] auth: Administrator:password 2020-06-25 01:22:21,345 | test | WARNING | pool-2-thread-1 | [rest_client:init_node_services:833] This node is already provisioned with services, we do not consider this as failure for test case 2020-06-25 01:22:21,459 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:443] RAM quota was defined less than 100 MB: 2020-06-25 01:22:22,029 | test | INFO | MainThread | [basetestcase:setUp:288] Couchbase Cluster: default, Nodes: 10.112.191.105, 10.112.191.106 initialized 2020-06-25 01:22:22,049 | test | INFO | MainThread | [basetestcase:_BaseTestCase__log_setup_status:412] ========= basetestcase setup finished for test #1 test_staged_doc_read ========= 2020-06-25 01:22:22,394 | test | INFO | MainThread | [basetestcase:sleep:514] Reason: None. Sleep for 5 secs ... 2020-06-25 01:22:28,894 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+----------+--------------+ | Nodes | Services | Status | +----------------+----------+--------------+ | 10.112.191.105 | kv | Cluster node | | 10.112.191.106 | None | <--- IN --- | +----------------+----------+--------------+ 2020-06-25 01:22:28,917 | test | INFO | pool-2-thread-3 | [task:check:304] Rebalance - status: running, progress: 0 2020-06-25 01:22:43,967 | test | INFO | pool-2-thread-3 | [task:check:304] Rebalance - status: none, progress: 100 2020-06-25 01:22:43,986 | test | INFO | pool-2-thread-3 | [task:check:363] Rebalance completed with progress: 100% in 15.0920000076 sec 2020-06-25 01:22:54,858 | test | INFO | MainThread | [doc_isolation:_IsolationDocTest__run_mock_test:148] Starting Mock_Transaction Transaction failed from runTransaction 01:22:54.933 1/MainThread/3f65d starting attempt 0/3f65d0fa-5057-4a8c-966d-58518d5910b0/61e788d7-7a01-4f36-8727-c863fb212e7a 01:22:54.933 1/MainThread/3f65d config: atrs=1024, expiry=30000msecs durability=MAJORITY per-txn config: durability=Optional.empty 01:22:54.953 1/MainThread/3f65d/61e78 First mutated doc in txn is 'test_docs-0' on vbucket 971, so using atr _txn:atr-971-#4b 01:22:54.957 1/MainThread/3f65d/61e78 about to set ATR default/_default/_txn:atr-971-#4b to Pending 01:22:54.958 1/MainThread/3f65d/61e78 error while setting ATR default/_default/_txn:atr-971-#4b to Pending: {err=TemporaryFailureException,ec=FAIL_TRANSIENT,msg='The document is temporarily not accessible on the server'} 01:22:54.958 1/MainThread/3f65d/61e78 transient error likely to be solved by retry 01:22:54.966 1/MainThread/3f65d/61e78 caught exception 'ErrorWrapper {ec:FAIL_TRANSIENT, cause:TemporaryFailureException, retry:true, autoRollback:true}' in runBlocking, rethrowing 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.transactions.error.internal.ErrorBuilder.build(ErrorBuilder.java:58) 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.transactions.AttemptContextReactive.lambda$atrPending$36(AttemptContextReactive.java:841) 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.transactions.AttemptContext.insert(AttemptContext.java:133) 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.transactions.AttemptContext.insert(AttemptContext.java:137) 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.test.transactions.SimpleTransaction.lambda$MockRunTransaction$9(SimpleTransaction.java:204) 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.transactions.TransactionsReactive.lambda$null$32(TransactionsReactive.java:567) 01:22:54.966 1/MainThread/3f65d/<> finishing attempt off after error 'ErrorWrapper {ec:FAIL_TRANSIENT, cause:TemporaryFailureException, retry:true, autoRollback:true}' 01:22:54.967 1/MainThread/3f65d/61e78 told to auto-rollback but in NOT_STARTED state, so nothing to do - skipping rollback 01:22:54.968 1/MainThread/3f65d/61e78 Skipping addition of cleanup request, as NOT_STARTED 01:22:54.970 1/MainThread/3f65d/61e78 added attempt TransactionAttempt{id=61e78,state=NOT_STARTED,atrColl=com.couchbase.client.java.ReactiveCollection@1515cece/_txn:atr-971-#4b} after error 01:22:54.970 1/MainThread/3f65d/<> reraising original exception 'ErrorWrapper' com.couchbase.client.core.error.TemporaryFailureException: The document is temporarily not accessible on the server 01:22:54.971 1/MainThread/3f65d ErrorWrapper retryTransaction=true 01:22:54.971 1/MainThread/3f65d/<> retrying transaction after backoff 1millis 01:22:54.973 1/MainThread/3f65d starting attempt 1/3f65d0fa-5057-4a8c-966d-58518d5910b0/e1515799-915e-4059-815f-1c268903752e 01:22:54.973 1/MainThread/3f65d config: atrs=1024, expiry=30000msecs durability=MAJORITY per-txn config: durability=Optional.empty 01:22:54.973 1/MainThread/3f65d/e1515 First mutated doc in txn is 'test_docs-0' on vbucket 971, so using atr _txn:atr-971-#4b 01:22:54.974 1/MainThread/3f65d/e1515 about to set ATR default/_default/_txn:atr-971-#4b to Pending 01:22:54.974 1/MainThread/3f65d/e1515 error while setting ATR default/_default/_txn:atr-971-#4b to Pending: {err=TemporaryFailureException,ec=FAIL_TRANSIENT,msg='The document is temporarily not accessible on the server'} 01:22:54.974 1/MainThread/3f65d/e1515 transient error likely to be solved by retry 01:22:54.974 1/MainThread/3f65d/e1515 caught exception 'ErrorWrapper {ec:FAIL_TRANSIENT, cause:TemporaryFailureException, retry:true, autoRollback:true}' in runBlocking, rethrowing 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.transactions.error.internal.ErrorBuilder.build(ErrorBuilder.java:58) 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.transactions.AttemptContextReactive.lambda$atrPending$36(AttemptContextReactive.java:841) 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.transactions.AttemptContext.insert(AttemptContext.java:133) 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.transactions.AttemptContext.insert(AttemptContext.java:137) 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.test.transactions.SimpleTransaction.lambda$MockRunTransaction$9(SimpleTransaction.java:204) 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.transactions.TransactionsReactive.lambda$null$32(TransactionsReactive.java:567) 01:22:54.975 1/MainThread/3f65d/<> finishing attempt off after error 'ErrorWrapper {ec:FAIL_TRANSIENT, cause:TemporaryFailureException, retry:true, autoRollback:true}' 01:22:54.975 1/MainThread/3f65d/e1515 told to auto-rollback but in NOT_STARTED state, so nothing to do - skipping rollback 01:22:54.975 1/MainThread/3f65d/e1515 Skipping addition of cleanup request, as NOT_STARTED 01:22:54.975 1/MainThread/3f65d/e1515 added attempt TransactionAttempt{id=e1515,state=NOT_STARTED,atrColl=com.couchbase.client.java.ReactiveCollection@1515cece/_txn:atr-971-#4b} after error 01:22:54.975 1/MainThread/3f65d/<> reraising original exception 'ErrorWrapper' com.couchbase.client.core.error.TemporaryFailureException: The document is temporarily not accessible on the server 01:22:54.976 1/MainThread/3f65d ErrorWrapper retryTransaction=true 01:22:54.976 1/MainThread/3f65d/<> retrying transaction after backoff 2millis 01:22:54.979 1/MainThread/3f65d starting attempt 2/3f65d0fa-5057-4a8c-966d-58518d5910b0/b4e0d389-e5ae-416b-854f-41a7c954309b 01:22:54.979 1/MainThread/3f65d config: atrs=1024, expiry=30000msecs durability=MAJORITY per-txn config: durability=Optional.empty 01:22:54.979 1/MainThread/3f65d/b4e0d First mutated doc in txn is 'test_docs-0' on vbucket 971, so using atr _txn:atr-971-#4b 01:22:54.979 1/MainThread/3f65d/b4e0d about to set ATR default/_default/_txn:atr-971-#4b to Pending 01:22:54.984 1/MainThread/3f65d/b4e0d set ATR default/_default/_txn:atr-971-#4b to Pending in 4ms, got CAS (start time) cas=1593028374962307072,seqno=1,vbucket=971 01:22:54.988 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:22:57.264 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=DurabilityAmbiguousException,ec=FAIL_AMBIGUOUS,msg='The server returned with a durability ambiguous response on this request'} 01:22:57.267 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:22:59.523 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=DurabilityAmbiguousException,ec=FAIL_AMBIGUOUS,msg='The server returned with a durability ambiguous response on this request'} 01:22:59.526 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:01.856 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=RequestCanceledException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'} 01:23:01.860 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:04.364 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'} 01:23:04.368 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:06.874 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'} 01:23:06.878 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:09.385 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'} 01:23:09.389 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:11.895 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'} 01:23:11.900 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:14.404 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'} 01:23:14.408 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:16.915 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'} 01:23:16.919 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:19.423 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'} 01:23:19.427 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:21.935 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'} 01:23:21.938 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:24.445 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'} 01:23:24.449 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:26.955 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'} 01:23:26.958 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty 01:23:26.958 1/MainThread/3f65d Has expired client side (now=62728749523173ns start=62696696945068ns deferredElapsed=0ns expired=32052578105ns (32052ms) config=30000ms) 01:23:26.958 1/MainThread/3f65d/b4e0d expired in createdStagedInsert 01:23:26.958 1/MainThread/3f65d/b4e0d has expired in stage createdStagedInsert 01:23:26.959 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AttemptExpired,ec=FAIL_EXPIRY,msg='Attempt has expired in stage createdStagedInsert'} 01:23:26.959 1/MainThread/3f65d/b4e0d moving to expiry-overtime-mode in stage createdStagedInsert, and raising error 01:23:26.960 1/MainThread/3f65d/b4e0d caught exception 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}' in runBlocking, rethrowing 01:23:26.960 1/MainThread/3f65d/b4e0d com.couchbase.transactions.error.internal.ErrorBuilder.build(ErrorBuilder.java:58) 01:23:26.960 1/MainThread/3f65d/b4e0d com.couchbase.transactions.AttemptContextReactive.setExpiryOvertimeModeAndFail(AttemptContextReactive.java:1934) 01:23:26.960 1/MainThread/3f65d/b4e0d com.couchbase.transactions.AttemptContextReactive.lambda$null$63(AttemptContextReactive.java:1228) 01:23:26.960 1/MainThread/3f65d/<> finishing attempt off after error 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}' 01:23:26.960 1/MainThread/3f65d/b4e0d auto-rolling-back on error 01:23:26.961 1/MainThread/3f65d/b4e0d rollback AttemptContextReactive{id=b4e0d,state=PENDING,atr=_default/_txn:atr-971-#4b,staged=[]} expiryOvertimeMode=true isAppRollback=false 01:23:26.968 1/MainThread/3f65d/b4e0d aborting ATR default/_default/_txn:atr-971-#4b isAppRollback=false ambiguityResolutionMode=false 01:23:26.968 1/MainThread/3f65d/b4e0d not doing expiry check in atrAbort as already in expiry-overtime-mode 01:23:29.475 1/MainThread/3f65d/b4e0d error 'AmbiguousTimeoutException' 'SubdocMutateRequest' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-971-#4b 01:23:29.475 1/MainThread/3f65d/b4e0d in expiry-overtime mode so changing error 'com.couchbase.client.core.error.AmbiguousTimeoutException: SubdocMutateRequest {"cancelled":true,"clientContext":{"txn.op":"atrAbort"},"completed":true,"coreId":"0x70a3e32800000002","idempotent":false,"lastChannelId":"70A3E32800000002/000000000D25B294","lastDispatchedFrom":"10.112.191.1:65166","lastDispatchedTo":"10.112.191.106:11210","reason":"TIMEOUT","requestId":685,"requestType":"SubdocMutateRequest","retried":14,"retryReasons":["KV_TEMPORARY_FAILURE"],"service":{"bucket":"default","collection":"_default","documentId":"_txn:atr-971-#4b","opaque":"0x31e","scope":"_default","syncDurability":{"present":true},"type":"kv"},"timeoutMs":2500,"timings":{"dispatchMicros":763,"encodingMicros":248,"totalMicros":2506522,"serverMicros":0}}' to raise TRANSACTION_EXPIRED in stage 'atrAbort'; no rollback will be tried 01:23:29.483 1/MainThread/3f65d/<> rollback failed with 'ErrorWrapper' com.couchbase.transactions.error.internal.AttemptExpired: com.couchbase.client.core.error.AmbiguousTimeoutException: SubdocMutateRequest {"cancelled":true,"clientContext":{"txn.op":"atrAbort"},"completed":true,"coreId":"0x70a3e32800000002","idempotent":false,"lastChannelId":"70A3E32800000002/000000000D25B294","lastDispatchedFrom":"10.112.191.1:65166","lastDispatchedTo":"10.112.191.106:11210","reason":"TIMEOUT","requestId":685,"requestType":"SubdocMutateRequest","retried":14,"retryReasons":["KV_TEMPORARY_FAILURE"],"service":{"bucket":"default","collection":"_default","documentId":"_txn:atr-971-#4b","opaque":"0x31e","scope":"_default","syncDurability":{"present":true},"type":"kv"},"timeoutMs":2500,"timings":{"dispatchMicros":763,"encodingMicros":248,"totalMicros":2506522,"serverMicros":0}}, raising original error but with retryTransaction turned off 01:23:29.483 1/MainThread/3f65d/b4e0d Adding cleanup request for _default/_txn:atr-971-#4b to run in 30000 msecs 01:23:29.484 1/MainThread/3f65d/b4e0d added attempt TransactionAttempt{id=b4e0d,state=PENDING,atrColl=com.couchbase.client.java.ReactiveCollection@1515cece/_txn:atr-971-#4b} after error 01:23:29.484 1/MainThread/3f65d ErrorWrapper retryTransaction=false 01:23:29.484 1/MainThread/3f65d/b4e0d converted ErrorWrapper TRANSACTION_EXPIRED to final error com.couchbase.transactions.error.TransactionExpired: Transaction has expired configured timeout of 30000msecs. The transaction is not committed. 2020-06-25 01:23:40,010 | test | INFO | MainThread | [doc_isolation:test_staged_doc_read:217] [01:22:54.933 1/MainThread/3f65d starting attempt 0/3f65d0fa-5057-4a8c-966d-58518d5910b0/61e788d7-7a01-4f36-8727-c863fb212e7a, 01:22:54.933 1/MainThread/3f65d config: atrs=1024, expiry=30000msecs durability=MAJORITY per-txn config: durability=Optional.empty, 01:22:54.953 1/MainThread/3f65d/61e78 First mutated doc in txn is 'test_docs-0' on vbucket 971, so using atr _txn:atr-971-#4b, 01:22:54.957 1/MainThread/3f65d/61e78 about to set ATR default/_default/_txn:atr-971-#4b to Pending, 01:22:54.958 1/MainThread/3f65d/61e78 error while setting ATR default/_default/_txn:atr-971-#4b to Pending: {err=TemporaryFailureException,ec=FAIL_TRANSIENT,msg='The document is temporarily not accessible on the server'}, 01:22:54.958 1/MainThread/3f65d/61e78 transient error likely to be solved by retry, 01:22:54.966 1/MainThread/3f65d/61e78 caught exception 'ErrorWrapper {ec:FAIL_TRANSIENT, cause:TemporaryFailureException, retry:true, autoRollback:true}' in runBlocking, rethrowing, 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.transactions.error.internal.ErrorBuilder.build(ErrorBuilder.java:58), 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.transactions.AttemptContextReactive.lambda$atrPending$36(AttemptContextReactive.java:841), 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.transactions.AttemptContext.insert(AttemptContext.java:133), 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.transactions.AttemptContext.insert(AttemptContext.java:137), 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.test.transactions.SimpleTransaction.lambda$MockRunTransaction$9(SimpleTransaction.java:204), 01:22:54.966 1/MainThread/3f65d/61e78 com.couchbase.transactions.TransactionsReactive.lambda$null$32(TransactionsReactive.java:567), 01:22:54.966 1/MainThread/3f65d/<> finishing attempt off after error 'ErrorWrapper {ec:FAIL_TRANSIENT, cause:TemporaryFailureException, retry:true, autoRollback:true}', 01:22:54.967 1/MainThread/3f65d/61e78 told to auto-rollback but in NOT_STARTED state, so nothing to do - skipping rollback, 01:22:54.968 1/MainThread/3f65d/61e78 Skipping addition of cleanup request, as NOT_STARTED, 01:22:54.970 1/MainThread/3f65d/61e78 added attempt TransactionAttempt{id=61e78,state=NOT_STARTED,atrColl=com.couchbase.client.java.ReactiveCollection@1515cece/_txn:atr-971-#4b} after error, 01:22:54.970 1/MainThread/3f65d/<> reraising original exception 'ErrorWrapper' com.couchbase.client.core.error.TemporaryFailureException: The document is temporarily not accessible on the server, 01:22:54.971 1/MainThread/3f65d ErrorWrapper retryTransaction=true, 01:22:54.971 1/MainThread/3f65d/<> retrying transaction after backoff 1millis, 01:22:54.973 1/MainThread/3f65d starting attempt 1/3f65d0fa-5057-4a8c-966d-58518d5910b0/e1515799-915e-4059-815f-1c268903752e, 01:22:54.973 1/MainThread/3f65d config: atrs=1024, expiry=30000msecs durability=MAJORITY per-txn config: durability=Optional.empty, 01:22:54.973 1/MainThread/3f65d/e1515 First mutated doc in txn is 'test_docs-0' on vbucket 971, so using atr _txn:atr-971-#4b, 01:22:54.974 1/MainThread/3f65d/e1515 about to set ATR default/_default/_txn:atr-971-#4b to Pending, 01:22:54.974 1/MainThread/3f65d/e1515 error while setting ATR default/_default/_txn:atr-971-#4b to Pending: {err=TemporaryFailureException,ec=FAIL_TRANSIENT,msg='The document is temporarily not accessible on the server'}, 01:22:54.974 1/MainThread/3f65d/e1515 transient error likely to be solved by retry, 01:22:54.974 1/MainThread/3f65d/e1515 caught exception 'ErrorWrapper {ec:FAIL_TRANSIENT, cause:TemporaryFailureException, retry:true, autoRollback:true}' in runBlocking, rethrowing, 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.transactions.error.internal.ErrorBuilder.build(ErrorBuilder.java:58), 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.transactions.AttemptContextReactive.lambda$atrPending$36(AttemptContextReactive.java:841), 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.transactions.AttemptContext.insert(AttemptContext.java:133), 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.transactions.AttemptContext.insert(AttemptContext.java:137), 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.test.transactions.SimpleTransaction.lambda$MockRunTransaction$9(SimpleTransaction.java:204), 01:22:54.974 1/MainThread/3f65d/e1515 com.couchbase.transactions.TransactionsReactive.lambda$null$32(TransactionsReactive.java:567), 01:22:54.975 1/MainThread/3f65d/<> finishing attempt off after error 'ErrorWrapper {ec:FAIL_TRANSIENT, cause:TemporaryFailureException, retry:true, autoRollback:true}', 01:22:54.975 1/MainThread/3f65d/e1515 told to auto-rollback but in NOT_STARTED state, so nothing to do - skipping rollback, 01:22:54.975 1/MainThread/3f65d/e1515 Skipping addition of cleanup request, as NOT_STARTED, 01:22:54.975 1/MainThread/3f65d/e1515 added attempt TransactionAttempt{id=e1515,state=NOT_STARTED,atrColl=com.couchbase.client.java.ReactiveCollection@1515cece/_txn:atr-971-#4b} after error, 01:22:54.975 1/MainThread/3f65d/<> reraising original exception 'ErrorWrapper' com.couchbase.client.core.error.TemporaryFailureException: The document is temporarily not accessible on the server, 01:22:54.976 1/MainThread/3f65d ErrorWrapper retryTransaction=true, 01:22:54.976 1/MainThread/3f65d/<> retrying transaction after backoff 2millis, 01:22:54.979 1/MainThread/3f65d starting attempt 2/3f65d0fa-5057-4a8c-966d-58518d5910b0/b4e0d389-e5ae-416b-854f-41a7c954309b, 01:22:54.979 1/MainThread/3f65d config: atrs=1024, expiry=30000msecs durability=MAJORITY per-txn config: durability=Optional.empty, 01:22:54.979 1/MainThread/3f65d/b4e0d First mutated doc in txn is 'test_docs-0' on vbucket 971, so using atr _txn:atr-971-#4b, 01:22:54.979 1/MainThread/3f65d/b4e0d about to set ATR default/_default/_txn:atr-971-#4b to Pending, 01:22:54.984 1/MainThread/3f65d/b4e0d set ATR default/_default/_txn:atr-971-#4b to Pending in 4ms, got CAS (start time) cas=1593028374962307072,seqno=1,vbucket=971, 01:22:54.988 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:22:57.264 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=DurabilityAmbiguousException,ec=FAIL_AMBIGUOUS,msg='The server returned with a durability ambiguous response on this request'}, 01:22:57.267 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:22:59.523 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=DurabilityAmbiguousException,ec=FAIL_AMBIGUOUS,msg='The server returned with a durability ambiguous response on this request'}, 01:22:59.526 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:01.856 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=RequestCanceledException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'}, 01:23:01.860 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:04.364 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'}, 01:23:04.368 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:06.874 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'}, 01:23:06.878 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:09.385 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'}, 01:23:09.389 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:11.895 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'}, 01:23:11.900 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:14.404 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'}, 01:23:14.408 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:16.915 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'}, 01:23:16.919 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:19.423 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'}, 01:23:19.427 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:21.935 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'}, 01:23:21.938 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:24.445 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'}, 01:23:24.449 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:26.955 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest'}, 01:23:26.958 1/MainThread/3f65d/b4e0d about to insert staged doc test_docs-0 as shadow document, cas=Optional.empty, 01:23:26.958 1/MainThread/3f65d Has expired client side (now=62728749523173ns start=62696696945068ns deferredElapsed=0ns expired=32052578105ns (32052ms) config=30000ms), 01:23:26.958 1/MainThread/3f65d/b4e0d expired in createdStagedInsert, 01:23:26.958 1/MainThread/3f65d/b4e0d has expired in stage createdStagedInsert, 01:23:26.959 1/MainThread/3f65d/b4e0d got err while staging insert of test_docs-0: {err=AttemptExpired,ec=FAIL_EXPIRY,msg='Attempt has expired in stage createdStagedInsert'}, 01:23:26.959 1/MainThread/3f65d/b4e0d moving to expiry-overtime-mode in stage createdStagedInsert, and raising error, 01:23:26.960 1/MainThread/3f65d/b4e0d caught exception 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}' in runBlocking, rethrowing, 01:23:26.960 1/MainThread/3f65d/b4e0d com.couchbase.transactions.error.internal.ErrorBuilder.build(ErrorBuilder.java:58), 01:23:26.960 1/MainThread/3f65d/b4e0d com.couchbase.transactions.AttemptContextReactive.setExpiryOvertimeModeAndFail(AttemptContextReactive.java:1934), 01:23:26.960 1/MainThread/3f65d/b4e0d com.couchbase.transactions.AttemptContextReactive.lambda$null$63(AttemptContextReactive.java:1228), 01:23:26.960 1/MainThread/3f65d/<> finishing attempt off after error 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}', 01:23:26.960 1/MainThread/3f65d/b4e0d auto-rolling-back on error, 01:23:26.961 1/MainThread/3f65d/b4e0d rollback AttemptContextReactive{id=b4e0d,state=PENDING,atr=_default/_txn:atr-971-#4b,staged=[]} expiryOvertimeMode=true isAppRollback=false, 01:23:26.968 1/MainThread/3f65d/b4e0d aborting ATR default/_default/_txn:atr-971-#4b isAppRollback=false ambiguityResolutionMode=false, 01:23:26.968 1/MainThread/3f65d/b4e0d not doing expiry check in atrAbort as already in expiry-overtime-mode, 01:23:29.475 1/MainThread/3f65d/b4e0d error 'AmbiguousTimeoutException' 'SubdocMutateRequest' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-971-#4b, 01:23:29.475 1/MainThread/3f65d/b4e0d in expiry-overtime mode so changing error 'com.couchbase.client.core.error.AmbiguousTimeoutException: SubdocMutateRequest {"cancelled":true,"clientContext":{"txn.op":"atrAbort"},"completed":true,"coreId":"0x70a3e32800000002","idempotent":false,"lastChannelId":"70A3E32800000002/000000000D25B294","lastDispatchedFrom":"10.112.191.1:65166","lastDispatchedTo":"10.112.191.106:11210","reason":"TIMEOUT","requestId":685,"requestType":"SubdocMutateRequest","retried":14,"retryReasons":["KV_TEMPORARY_FAILURE"],"service":{"bucket":"default","collection":"_default","documentId":"_txn:atr-971-#4b","opaque":"0x31e","scope":"_default","syncDurability":{"present":true},"type":"kv"},"timeoutMs":2500,"timings":{"dispatchMicros":763,"encodingMicros":248,"totalMicros":2506522,"serverMicros":0}}' to raise TRANSACTION_EXPIRED in stage 'atrAbort'; no rollback will be tried, 01:23:29.483 1/MainThread/3f65d/<> rollback failed with 'ErrorWrapper' com.couchbase.transactions.error.internal.AttemptExpired: com.couchbase.client.core.error.AmbiguousTimeoutException: SubdocMutateRequest {"cancelled":true,"clientContext":{"txn.op":"atrAbort"},"completed":true,"coreId":"0x70a3e32800000002","idempotent":false,"lastChannelId":"70A3E32800000002/000000000D25B294","lastDispatchedFrom":"10.112.191.1:65166","lastDispatchedTo":"10.112.191.106:11210","reason":"TIMEOUT","requestId":685,"requestType":"SubdocMutateRequest","retried":14,"retryReasons":["KV_TEMPORARY_FAILURE"],"service":{"bucket":"default","collection":"_default","documentId":"_txn:atr-971-#4b","opaque":"0x31e","scope":"_default","syncDurability":{"present":true},"type":"kv"},"timeoutMs":2500,"timings":{"dispatchMicros":763,"encodingMicros":248,"totalMicros":2506522,"serverMicros":0}}, raising original error but with retryTransaction turned off, 01:23:29.483 1/MainThread/3f65d/b4e0d Adding cleanup request for _default/_txn:atr-971-#4b to run in 30000 msecs, 01:23:29.484 1/MainThread/3f65d/b4e0d added attempt TransactionAttempt{id=b4e0d,state=PENDING,atrColl=com.couchbase.client.java.ReactiveCollection@1515cece/_txn:atr-971-#4b} after error, 01:23:29.484 1/MainThread/3f65d ErrorWrapper retryTransaction=false, 01:23:29.484 1/MainThread/3f65d/b4e0d converted ErrorWrapper TRANSACTION_EXPIRED to final error com.couchbase.transactions.error.TransactionExpired: Transaction has expired configured timeout of 30000msecs. The transaction is not committed.] 2020-06-25 01:23:45,038 | test | WARNING | MainThread | [sdk_client3:read:613] Request cancelled/timed-out: com.couchbase.client.core.error.UnambiguousTimeoutException: GetRequest {"cancelled":true,"completed":true,"coreId":"0x70a3e32800000002","idempotent":true,"lastChannelId":"70A3E32800000002/000000000D25B294","lastDispatchedFrom":"10.112.191.1:65166","lastDispatchedTo":"10.112.191.106:11210","reason":"TIMEOUT","requestId":797,"requestType":"GetRequest","retried":19,"retryReasons":["KV_SYNC_WRITE_RE_COMMIT_IN_PROGRESS"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-0","opaque":"0x38e","scope":"_default","type":"kv"},"timeoutMs":5000,"timings":{"dispatchMicros":552,"totalMicros":5010872,"serverMicros":0}} 2020-06-25 01:23:45,051 | test | INFO | MainThread | [doc_isolation:test_staged_doc_read:242] Key test_docs-0 Actual: {}, Expected: {"value":"value1"} 2020-06-25 01:23:45,052 | test | ERROR | MainThread | [basetestcase:log_failure:518] Mismatch in doc content