./testrunner -i /tmp/testexec.26923.ini durability=PERSIST_TO_MAJORITY,services_init=kv;n1ql;index,GROUP=P1,rerun=False,upgrade_version=6.6.0-7880 -t Atomicity.doc_isolation.IsolationDocTest.test_transaction_with_rebalance,nodes_init=4,replicas=2,num_items=20000,rebalance_type=out,nodes_out=1,doc_op=create,GROUP=P1 Test Input params: {'cluster_name': 'testexec.26923', 'ini': '/tmp/testexec.26923.ini', 'replicas': '2', 'doc_op': 'create', 'durability': 'PERSIST_TO_MAJORITY', 'conf_file': 'conf/Atomicity/doc_isolation.conf', 'upgrade_version': '6.6.0-7880', 'rebalance_type': 'out', 'spec': 'doc_isolation', 'rerun': 'False', 'num_nodes': 4, 'logs_folder': '/data/workspace/centos-p0-transaction-vset00-00-doc_isolation_durability_persist_to_majority_6.6_P1/logs/testrunner-20-Jul-14_22-58-18/test_8', 'nodes_init': '4', 'GROUP': 'P1', 'nodes_out': '1', 'num_items': '20000', 'services_init': 'kv;n1ql;index', 'case_number': 8} test_transaction_with_rebalance (Atomicity.doc_isolation.IsolationDocTest) ... 2020-07-14 23:27:54,516 | test | INFO | MainThread | [basetestcase:_BaseTestCase__log_setup_status:412] ========= basetestcase setup started for test #8 test_transaction_with_rebalance ========= 2020-07-14 23:27:55,221 | test | INFO | MainThread | [basetestcase:_BaseTestCase__log_setup_status:412] ========= basetestcase setup started for test #8 test_transaction_with_rebalance ========= 2020-07-14 23:27:55,752 | test | INFO | MainThread | [basetestcase:setUp:266] Initializing cluster 2020-07-14 23:27:56,069 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:443] RAM quota was defined less than 100 MB: 2020-07-14 23:27:58,203 | test | INFO | MainThread | [basetestcase:setUp:288] Couchbase Cluster: default, Nodes: 172.23.107.52, 172.23.123.102, 172.23.123.100, 172.23.123.101 initialized 2020-07-14 23:27:58,223 | test | INFO | MainThread | [basetestcase:_BaseTestCase__log_setup_status:412] ========= basetestcase setup finished for test #8 test_transaction_with_rebalance ========= 2020-07-14 23:27:58,270 | test | INFO | MainThread | [basetestcase:sleep:514] Reason: None. Sleep for 5 secs ... 2020-07-14 23:28:08,604 | test | INFO | pool-11-thread-5 | [table_view:display:72] Rebalance Overview +----------------+-----------------+--------------+ | Nodes | Services | Status | +----------------+-----------------+--------------+ | 172.23.107.52 | index, kv, n1ql | Cluster node | | 172.23.123.102 | None | <--- IN --- | | 172.23.123.100 | None | <--- IN --- | | 172.23.123.101 | None | <--- IN --- | +----------------+-----------------+--------------+ 2020-07-14 23:28:08,624 | test | INFO | pool-11-thread-5 | [task:check:304] Rebalance - status: running, progress: 0 2020-07-14 23:28:23,686 | test | INFO | pool-11-thread-5 | [task:check:304] Rebalance - status: none, progress: 100 2020-07-14 23:28:23,710 | test | INFO | pool-11-thread-5 | [task:check:363] Rebalance completed with progress: 100% in 15.1070001125 sec 2020-07-14 23:28:28,989 | test | INFO | pool-11-thread-7 | [task:call:3695] transaction timeout is 100 2020-07-14 23:28:28,990 | test | INFO | pool-11-thread-7 | [task:call:3699] transaction is com.couchbase.transactions.Transactions@215e0d54 2020-07-14 23:28:29,000 | test | INFO | pool-11-thread-8 | [task:call:3803] Starting Atomicity load generation thread 2020-07-14 23:28:29,000 | test | INFO | pool-11-thread-9 | [task:call:3803] Starting Atomicity load generation thread 2020-07-14 23:28:29,000 | test | INFO | pool-11-thread-10 | [task:call:3803] Starting Atomicity load generation thread 2020-07-14 23:28:29,000 | test | INFO | pool-11-thread-4 | [task:call:3803] Starting Atomicity load generation thread TransactionsStarted: Transactions successfully started, regular cleanup enabled=true, lost cleanup enabled=true 2020-07-14 23:28:29,306 | test | INFO | pool-11-thread-2 | [task:call:3695] transaction timeout is 100 2020-07-14 23:28:29,309 | test | INFO | pool-11-thread-2 | [task:call:3699] transaction is com.couchbase.transactions.Transactions@7c29507f 2020-07-14 23:28:29,315 | test | INFO | pool-11-thread-1 | [task:call:3803] Starting Atomicity load generation thread 2020-07-14 23:28:29,315 | test | INFO | pool-11-thread-3 | [task:call:3803] Starting Atomicity load generation thread 2020-07-14 23:28:29,315 | test | INFO | pool-11-thread-5 | [task:call:3803] Starting Atomicity load generation thread 2020-07-14 23:28:29,315 | test | INFO | pool-11-thread-6 | [task:call:3803] Starting Atomicity load generation thread 2020-07-14 23:28:29,322 | test | INFO | MainThread | [basetestcase:sleep:514] Reason: Wait for transactions to start. Sleep for 3 secs ... TransactionsStarted: Transactions successfully started, regular cleanup enabled=true, lost cleanup enabled=true 2020-07-14 23:28:30,871 | test | INFO | pool-11-thread-4 | [task:call:3931] Atomicity Load generation thread completed 2020-07-14 23:28:30,871 | test | INFO | pool-11-thread-4 | [task:call:3937] Starting Atomicity Verification thread 2020-07-14 23:28:30,874 | test | INFO | pool-11-thread-4 | [task:call:3957] Completed Atomicity Verification thread 2020-07-14 23:28:32,437 | test | INFO | pool-11-thread-4 | [table_view:display:72] Rebalance Overview +----------------+-----------------+--------------+ | Nodes | Services | Status | +----------------+-----------------+--------------+ | 172.23.107.52 | index, kv, n1ql | Cluster node | | 172.23.123.101 | [u'kv'] | --- OUT ---> | | 172.23.123.102 | kv | Cluster node | | 172.23.123.100 | kv | Cluster node | +----------------+-----------------+--------------+ 2020-07-14 23:28:32,457 | test | INFO | pool-11-thread-4 | [task:check:304] Rebalance - status: running, progress: 0 2020-07-14 23:28:35,292 | test | INFO | pool-11-thread-6 | [task:call:3931] Atomicity Load generation thread completed 2020-07-14 23:28:35,292 | test | INFO | pool-11-thread-6 | [task:call:3937] Starting Atomicity Verification thread 2020-07-14 23:28:35,295 | test | INFO | pool-11-thread-6 | [task:call:3957] Completed Atomicity Verification thread 2020-07-14 23:28:42,503 | test | ERROR | pool-11-thread-4 | [rest_client:_rebalance_status_and_progress:1470] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2020-07-14 23:28:42,576 | test | INFO | pool-11-thread-4 | [rest_client:print_UI_logs:2526] Latest logs from UI on 172.23.107.52: 2020-07-14 23:28:42,578 | test | ERROR | pool-11-thread-4 | [rest_client:print_UI_logs:2528] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.107.52', u'tstamp': 1594794514072L, u'shortText': u'message', u'serverTime': u'2020-07-14T23:28:34.072Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.6670.0>,\n {{{{{child_interrupted,\n {\'EXIT\',<17502.2478.0>,socket_closed}},\n [{dcp_replicator,spawn_and_wait,1,\n [{file,"src/dcp_replicator.erl"},\n {line,266}]},\n {dcp_replicator,handle_call,3,\n [{file,"src/dcp_replicator.erl"},\n {line,121}]},\n {gen_server,try_handle_call,4,\n [{file,"gen_server.erl"},{line,636}]},\n {gen_server,handle_msg,6,\n [{file,"gen_server.erl"},{line,665}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,247}]}]},\n {gen_server,call,\n [<17502.2476.0>,get_partitions,infinity]}},\n {gen_server,call,\n [\'dcp_replication_manager-default\',\n {get_replicator_pid,543},\n infinity]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.123.102\'},\n {if_rebalance,<0.3620.0>,\n {update_vbucket_state,979,active,paused,\n undefined,\n [[\'ns_1@172.23.123.102\',\n \'ns_1@172.23.123.101\',\n \'ns_1@172.23.107.52\']]}},\n infinity]}}}}}.\nRebalance Operation Id = 322d92a2335598e144eb0bb97f14f1a3'} 2020-07-14 23:28:42,578 | test | ERROR | pool-11-thread-4 | [rest_client:print_UI_logs:2528] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.107.52', u'tstamp': 1594794514063L, u'shortText': u'message', u'serverTime': u'2020-07-14T23:28:34.063Z', u'text': u'Worker <0.6325.0> (for action {move,{979,\n [\'ns_1@172.23.123.102\',\n \'ns_1@172.23.123.101\',\n \'ns_1@172.23.107.52\'],\n [\'ns_1@172.23.107.52\',\n \'ns_1@172.23.123.100\',\n \'ns_1@172.23.123.102\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.6670.0>,\n {{{{{child_interrupted,\n {\'EXIT\',\n <17502.2478.0>,\n socket_closed}},\n [{dcp_replicator,\n spawn_and_wait,\n 1,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 266}]},\n {dcp_replicator,\n handle_call,\n 3,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 121}]},\n {gen_server,\n try_handle_call,\n 4,\n [{file,\n "gen_server.erl"},\n {line,\n 636}]},\n {gen_server,\n handle_msg,\n 6,\n [{file,\n "gen_server.erl"},\n {line,\n 665}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 247}]}]},\n {gen_server,\n call,\n [<17502.2476.0>,\n get_partitions,\n infinity]}},\n {gen_server,\n call,\n [\'dcp_replication_manager-default\',\n {get_replicator_pid,\n 543},\n infinity]}},\n {gen_server,\n call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.123.102\'},\n {if_rebalance,\n <0.3620.0>,\n {update_vbucket_state,\n 979,\n active,\n paused,\n undefined,\n [[\'ns_1@172.23.123.102\',\n \'ns_1@172.23.123.101\',\n \'ns_1@172.23.107.52\']]}},\n infinity]}}}}'} 2020-07-14 23:28:42,579 | test | ERROR | pool-11-thread-4 | [rest_client:print_UI_logs:2528] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.107.52', u'tstamp': 1594794512784L, u'shortText': u'message', u'serverTime': u'2020-07-14T23:28:32.784Z', u'text': u'Bucket "default" rebalance does not seem to be swap rebalance'} 2020-07-14 23:28:42,581 | test | ERROR | pool-11-thread-4 | [rest_client:print_UI_logs:2528] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.107.52', u'tstamp': 1594794512549L, u'shortText': u'message', u'serverTime': u'2020-07-14T23:28:32.549Z', u'text': u'Started rebalancing bucket default'} 2020-07-14 23:28:42,581 | test | ERROR | pool-11-thread-4 | [rest_client:print_UI_logs:2528] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.107.52', u'tstamp': 1594794512422L, u'shortText': u'message', u'serverTime': u'2020-07-14T23:28:32.422Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.107.52','ns_1@172.23.123.102',\n 'ns_1@172.23.123.100'], EjectNodes = ['ns_1@172.23.123.101'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 322d92a2335598e144eb0bb97f14f1a3"} 2020-07-14 23:28:42,582 | test | ERROR | pool-11-thread-4 | [rest_client:print_UI_logs:2528] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.123.102', u'tstamp': 1594794504543L, u'shortText': u'message', u'serverTime': u'2020-07-14T23:28:24.543Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.123.102\' in 0 seconds.'} 2020-07-14 23:28:42,582 | test | ERROR | pool-11-thread-4 | [rest_client:print_UI_logs:2528] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.107.52', u'tstamp': 1594794504087L, u'shortText': u'message', u'serverTime': u'2020-07-14T23:28:24.087Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.107.52\' in 0 seconds.'} 2020-07-14 23:28:42,582 | test | ERROR | pool-11-thread-4 | [rest_client:print_UI_logs:2528] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.123.100', u'tstamp': 1594794504063L, u'shortText': u'message', u'serverTime': u'2020-07-14T23:28:24.063Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.123.100\' in 0 seconds.'} 2020-07-14 23:28:42,582 | test | ERROR | pool-11-thread-4 | [rest_client:print_UI_logs:2528] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.123.101', u'tstamp': 1594794504057L, u'shortText': u'message', u'serverTime': u'2020-07-14T23:28:24.057Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.123.101\' in 0 seconds.'} 2020-07-14 23:28:42,582 | test | ERROR | pool-11-thread-4 | [rest_client:print_UI_logs:2528] {u'code': 12, u'module': u'menelaus_web', u'type': u'info', u'node': u'ns_1@172.23.107.52', u'tstamp': 1594794503966L, u'shortText': u'message', u'serverTime': u'2020-07-14T23:28:23.966Z', u'text': u'Created bucket "default" of type: couchbase\n[{num_replicas,2},\n {replica_index,true},\n {ram_quota,104857600},\n {durability_min_level,none},\n {flush_enabled,true},\n {num_threads,3},\n {eviction_policy,value_only},\n {conflict_resolution_type,seqno},\n {storage_mode,couchstore},\n {max_ttl,0},\n {compression_mode,passive}]'} 2020-07-14 23:28:42,584 | test | ERROR | pool-11-thread-4 | [task:call:231] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=0,numClients=1,ATRs={checking=1024,total=1024},runLength=60000millis},end={taken=15371millis,ATRs={read=214,errored=0,present=3},entries={expired=0,failed=0,total=4},maxEntries=2} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=15127millis,ATRs={read=71,errored=0,present=2},entries={expired=0,failed=0,total=3},maxEntries=2} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=15127millis,ATRs={read=71,errored=0,present=7},entries={expired=0,failed=0,total=10},maxEntries=3} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=15128millis,ATRs={read=71,errored=0,present=9},entries={expired=0,failed=0,total=12},maxEntries=3} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=1,numClients=2,ATRs={checking=512,total=1024},runLength=60000millis},end={taken=60403millis,ATRs={read=512,errored=0,present=51},entries={expired=0,failed=0,total=57},maxEntries=2} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=60151millis,ATRs={read=342,errored=0,present=43},entries={expired=0,failed=0,total=44},maxEntries=2} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=4089millis,ATRs={read=8,errored=0,present=1},entries={expired=0,failed=0,total=1},maxEntries=1} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=15128millis,ATRs={read=71,errored=0,present=17},entries={expired=0,failed=0,total=20},maxEntries=3} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=15127millis,ATRs={read=71,errored=0,present=20},entries={expired=0,failed=0,total=24},maxEntries=3} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=15127millis,ATRs={read=71,errored=0,present=26},entries={expired=0,failed=0,total=36},maxEntries=3} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=15129millis,ATRs={read=71,errored=0,present=36},entries={expired=0,failed=0,total=51},maxEntries=3} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=59952millis,ATRs={read=342,errored=0,present=147},entries={expired=1,failed=0,total=183},maxEntries=4,oldest=102690millis,id=_txn:atr-750-#1294/950e8}} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=59767millis,ATRs={read=341,errored=0,present=129},entries={expired=1,failed=0,total=171},maxEntries=5,oldest=104362millis,id=_txn:atr-974-#22b/a415d}} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=15127millis,ATRs={read=71,errored=0,present=43},entries={expired=0,failed=0,total=64},maxEntries=3} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=13027millis,ATRs={read=59,errored=0,present=37},entries={expired=0,failed=0,total=56},maxEntries=3} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=34582millis,ATRs={read=182,errored=0,present=128},entries={expired=1,failed=0,total=172},maxEntries=4,oldest=102224millis,id=_txn:atr-303-#b0a/cf36c}} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=34410millis,ATRs={read=181,errored=0,present=116},entries={expired=0,failed=0,total=157},maxEntries=4} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2682millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2680millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2681millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2685millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2679millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2686millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2680millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2685millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2685millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2683millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2679millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} 23:30:45.053 4342/pool-11-thread-5/7b69a starting attempt 0/7b69a7d2-6d73-412c-93a6-da9fa1c91315/2fd313ed-ee22-479b-992c-360c065e5f12 23:30:45.053 4342/pool-11-thread-5/7b69a config: atrs=1024, expiry=100000msecs durability=PERSIST_TO_MAJORITY per-txn config: durability=Optional.empty 23:30:45.053 4342/pool-11-thread-5/7b69a/2fd31 First mutated doc in txn is 'test_docs-00034207' on vbucket 755, so using atr _txn:atr-755-#71 23:30:45.053 4342/pool-11-thread-5/7b69a/2fd31 expiration time remaining is 100000 millis (capped from 100000) 23:30:45.053 4342/pool-11-thread-5/7b69a/2fd31 about to set ATR default/_default/_txn:atr-755-#71 to Pending 23:30:45.058 4342/pool-11-thread-5/7b69a/2fd31 set ATR default/_default/_txn:atr-755-#71 to Pending in 5ms, got CAS (start time) cas=1594794645053898752,seqno=20,vbucket=755 23:30:45.058 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034207 as shadow document, cas=Optional.empty 23:30:45.066 4342/pool-11-thread-5/7b69a/2fd31 inserted doc test_docs-00034207 got cas=1594794645059010560,seqno=22,vbucket=755, in 7ms 23:30:45.066 4342/pool-11-thread-5/7b69a/2fd31 getting doc test_docs-00034207, resolvingMissingATREntry= 23:30:45.066 4342/pool-11-thread-5/7b69a/2fd31 found own-write of mutated doc test_docs-00034207 23:30:45.066 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034204 as shadow document, cas=Optional.empty 23:30:45.071 4342/pool-11-thread-5/7b69a/2fd31 inserted doc test_docs-00034204 got cas=1594794645067137024,seqno=19,vbucket=1018, in 4ms 23:30:45.071 4342/pool-11-thread-5/7b69a/2fd31 getting doc test_docs-00034204, resolvingMissingATREntry= 23:30:45.071 4342/pool-11-thread-5/7b69a/2fd31 found own-write of mutated doc test_docs-00034204 23:30:45.071 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional.empty 23:30:47.329 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=DurabilityAmbiguousException,ec=FAIL_AMBIGUOUS,msg='The server returned with a durability ambiguous response on this request'} 23:30:47.332 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional.empty 23:30:49.817 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=DurabilityAmbiguousException,ec=FAIL_AMBIGUOUS,msg='The server returned with a durability ambiguous response on this request'} 23:30:49.820 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional.empty 23:30:52.328 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:30:52.331 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional.empty 23:30:54.417 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=RequestCanceledException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT)'} 23:30:54.420 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional.empty 23:30:56.928 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:30:56.931 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional.empty 23:30:56.932 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=DocumentExistsException,ec=FAIL_DOC_ALREADY_EXISTS,msg='Document with the given id already exists'} 23:30:56.932 4342/pool-11-thread-5/7b69a/2fd31 DocExists on test_docs-00034205: getting doc 23:30:56.934 4342/pool-11-thread-5/7b69a/2fd31 DocExists on test_docs-00034205: doc test_docs-00034205 exists inTransaction=true isDeleted=true 23:30:56.935 4342/pool-11-thread-5/7b69a/2fd31 doc test_docs-00034205 has been written by this transaction, ok to continue 23:30:56.935 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:30:59.438 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:30:59.441 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:01.948 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:01.952 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:04.458 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:04.461 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:06.968 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:06.972 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:09.478 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:09.481 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:11.988 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:11.991 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:14.498 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:14.501 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:17.008 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:17.011 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:19.518 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:19.521 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:22.028 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:22.031 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:24.538 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:24.541 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:27.049 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:27.052 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:29.558 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:29.561 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:32.068 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:32.072 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:34.578 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:34.581 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:37.088 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:37.091 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:39.598 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:39.602 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:42.108 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:42.111 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:44.618 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:44.621 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:47.129 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:47.132 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:49.638 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:49.642 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:52.148 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:52.151 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:54.658 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:54.661 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:57.169 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:57.172 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:31:59.678 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:59.682 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:02.188 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:02.191 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:04.698 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:04.701 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:07.208 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:07.211 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:09.719 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:09.722 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:12.228 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:12.232 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:14.738 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:14.741 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:17.248 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:17.252 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:19.759 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:19.762 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:22.268 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:22.271 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:24.778 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:24.781 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:27.288 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:27.291 4342/pool-11-thread-5/7b69a/2fd31 about to insert staged doc test_docs-00034205 as shadow document, cas=Optional[1594794652077326336] 23:32:27.291 4342/pool-11-thread-5/7b69a Has expired client side (now=7187692079254157ns start=7187589840634388ns deferredElapsed=0ns expired=102238millis config=100000ms) 23:32:27.291 4342/pool-11-thread-5/7b69a/2fd31 expired in createdStagedInsert 23:32:27.291 4342/pool-11-thread-5/7b69a/2fd31 has expired in stage createdStagedInsert 23:32:27.291 4342/pool-11-thread-5/7b69a/2fd31 got err while staging insert of test_docs-00034205: {err=AttemptExpired,ec=FAIL_EXPIRY,msg='Attempt has expired in stage createdStagedInsert'} 23:32:27.292 4342/pool-11-thread-5/7b69a/2fd31 moving to expiry-overtime-mode in stage createdStagedInsert, and raising error 23:32:27.293 4342/pool-11-thread-5/7b69a/2fd31 caught exception 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}' in runBlocking, rethrowing 23:32:27.293 4342/pool-11-thread-5/7b69a/2fd31 com.couchbase.transactions.error.internal.ErrorBuilder.build(ErrorBuilder.java:58) 23:32:27.293 4342/pool-11-thread-5/7b69a/2fd31 com.couchbase.transactions.AttemptContextReactive.setExpiryOvertimeModeAndFail(AttemptContextReactive.java:2045) 23:32:27.293 4342/pool-11-thread-5/7b69a/2fd31 com.couchbase.transactions.AttemptContextReactive.lambda$null$63(AttemptContextReactive.java:1232) 23:32:27.293 4342/pool-11-thread-5/7b69a/<> finishing attempt off after error 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}' 23:32:27.293 4342/pool-11-thread-5/7b69a/2fd31 auto-rolling-back on error 23:32:27.293 4342/pool-11-thread-5/7b69a/2fd31 rollback AttemptContextReactive{id=2fd31,state=ROLLED_BACK,atr=_default/_txn:atr-755-#71,staged=[INSERT test_docs-00034207, INSERT test_docs-00034204]} expiryOvertimeMode=true isAppRollback=false 23:32:27.293 4342/pool-11-thread-5/7b69a/2fd31 aborting ATR default/_default/_txn:atr-755-#71 isAppRollback=false ambiguityResolutionMode=false 23:32:27.293 4342/pool-11-thread-5/7b69a/2fd31 not doing expiry check in atrAbort as already in expiry-overtime-mode 23:32:27.300 4342/pool-11-thread-5/7b69a/2fd31 aborted ATR default/_default/_txn:atr-755-#71 23:32:27.300 4342/pool-11-thread-5/7b69a/2fd31 deleting inserted doc test_docs-00034207 with cas 1594794645059010560 23:32:27.300 4342/pool-11-thread-5/7b69a/2fd31 not doing expiry check in deleteInserted as already in expiry-overtime-mode 23:32:27.304 4342/pool-11-thread-5/7b69a/2fd31 deleted inserted doc test_docs-00034207, mt Optional[mt{vbID=755, vbUUID=260443685903230, seqno=28, bucket=default}] 23:32:27.304 4342/pool-11-thread-5/7b69a/2fd31 deleting inserted doc test_docs-00034204 with cas 1594794645067137024 23:32:27.304 4342/pool-11-thread-5/7b69a/2fd31 not doing expiry check in deleteInserted as already in expiry-overtime-mode 23:32:27.307 4342/pool-11-thread-5/7b69a/2fd31 deleted inserted doc test_docs-00034204, mt Optional[mt{vbID=1018, vbUUID=143217333424859, seqno=23, bucket=default}] 23:32:27.307 4342/pool-11-thread-5/7b69a/2fd31 marking ATR default/_default/_txn:atr-755-#71 as rollback complete 23:32:27.307 4342/pool-11-thread-5/7b69a/2fd31 ignoring expiry in stage atrRollbackComplete, as in expiry-overtime mode 23:32:27.312 4342/pool-11-thread-5/7b69a/2fd31 rollback - atr rolled back 23:32:27.312 4342/pool-11-thread-5/7b69a/2fd31 Adding cleanup request for _default/_txn:atr-755-#71 to run in 100000 msecs 23:32:27.312 4342/pool-11-thread-5/7b69a/2fd31 added attempt TransactionAttempt{id=2fd31,state=ROLLED_BACK,atrColl=com.couchbase.client.java.ReactiveCollection@53617190/_txn:atr-755-#71} after error 23:32:27.312 4342/pool-11-thread-5/7b69a/<> reraising original exception 'ErrorWrapper' com.couchbase.transactions.error.internal.AttemptExpired: com.couchbase.transactions.error.internal.AttemptExpired: Attempt has expired in stage createdStagedInsert 23:32:27.312 4342/pool-11-thread-5/7b69a ErrorWrapper retryTransaction=false 23:32:27.312 4342/pool-11-thread-5/7b69a/2fd31 converted ErrorWrapper TRANSACTION_EXPIRED to final error com.couchbase.transactions.error.TransactionExpired: Transaction has expired configured timeout of 100000msecs. The transaction is not committed. 2020-07-14 23:32:27,319 | test | INFO | pool-11-thread-5 | [task:transaction_load:3987] Sleep for 60 seconds so that the txn gets cleaned up TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2688millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} 23:30:45.036 4336/pool-11-thread-3/6ac27 starting attempt 0/6ac27c57-08ae-4c91-85f9-189947e3dd25/55dcc89e-1df9-46cf-a7dd-65c168f45667 23:30:45.036 4336/pool-11-thread-3/6ac27 config: atrs=1024, expiry=100000msecs durability=PERSIST_TO_MAJORITY per-txn config: durability=Optional.empty 23:30:45.036 4336/pool-11-thread-3/6ac27/55dcc First mutated doc in txn is 'test_docs-00027579' on vbucket 144, so using atr _txn:atr-144-#171 23:30:45.036 4336/pool-11-thread-3/6ac27/55dcc expiration time remaining is 100000 millis (capped from 100000) 23:30:45.036 4336/pool-11-thread-3/6ac27/55dcc about to set ATR default/_default/_txn:atr-144-#171 to Pending 23:30:45.043 4336/pool-11-thread-3/6ac27/55dcc set ATR default/_default/_txn:atr-144-#171 to Pending in 6ms, got CAS (start time) cas=1594794645036531712,seqno=23,vbucket=144 23:30:45.043 4336/pool-11-thread-3/6ac27/55dcc about to insert staged doc test_docs-00027579 as shadow document, cas=Optional.empty 23:30:45.051 4336/pool-11-thread-3/6ac27/55dcc inserted doc test_docs-00027579 got cas=1594794645042757632,seqno=25,vbucket=144, in 8ms 23:30:45.051 4336/pool-11-thread-3/6ac27/55dcc getting doc test_docs-00027579, resolvingMissingATREntry= 23:30:45.051 4336/pool-11-thread-3/6ac27/55dcc found own-write of mutated doc test_docs-00027579 23:30:45.051 4336/pool-11-thread-3/6ac27/55dcc about to insert staged doc test_docs-00027612 as shadow document, cas=Optional.empty 23:30:45.054 4336/pool-11-thread-3/6ac27/55dcc inserted doc test_docs-00027612 got cas=1594794645051867136,seqno=37,vbucket=862, in 3ms 23:30:45.054 4336/pool-11-thread-3/6ac27/55dcc getting doc test_docs-00027612, resolvingMissingATREntry= 23:30:45.054 4336/pool-11-thread-3/6ac27/55dcc found own-write of mutated doc test_docs-00027612 23:30:45.054 4336/pool-11-thread-3/6ac27/55dcc about to insert staged doc test_docs-00027578 as shadow document, cas=Optional.empty 23:30:45.058 4336/pool-11-thread-3/6ac27/55dcc inserted doc test_docs-00027578 got cas=1594794645055012864,seqno=21,vbucket=919, in 3ms 23:30:45.058 4336/pool-11-thread-3/6ac27/55dcc getting doc test_docs-00027578, resolvingMissingATREntry= 23:30:45.058 4336/pool-11-thread-3/6ac27/55dcc found own-write of mutated doc test_docs-00027578 23:30:45.058 4336/pool-11-thread-3/6ac27/55dcc about to insert staged doc test_docs-00027611 as shadow document, cas=Optional.empty 23:30:45.064 4336/pool-11-thread-3/6ac27/55dcc inserted doc test_docs-00027611 got cas=1594794645059272704,seqno=33,vbucket=599, in 5ms 23:30:45.064 4336/pool-11-thread-3/6ac27/55dcc getting doc test_docs-00027611, resolvingMissingATREntry= 23:30:45.064 4336/pool-11-thread-3/6ac27/55dcc found own-write of mutated doc test_docs-00027611 23:30:45.064 4336/pool-11-thread-3/6ac27/55dcc about to insert staged doc test_docs-00027614 as shadow document, cas=Optional.empty 23:30:47.332 4336/pool-11-thread-3/6ac27/55dcc got err while staging insert of test_docs-00027614: {err=DurabilityAmbiguousException,ec=FAIL_AMBIGUOUS,msg='The server returned with a durability ambiguous response on this request'} 23:30:47.336 4336/pool-11-thread-3/6ac27/55dcc about to insert staged doc test_docs-00027614 as shadow document, cas=Optional.empty 23:30:47.546 4336/pool-11-thread-3/6ac27/55dcc inserted doc test_docs-00027614 got cas=1594794647336583168,seqno=19,vbucket=573, in 210ms 23:30:47.546 4336/pool-11-thread-3/6ac27/55dcc getting doc test_docs-00027614, resolvingMissingATREntry= 23:30:47.546 4336/pool-11-thread-3/6ac27/55dcc found own-write of mutated doc test_docs-00027614 23:30:47.546 4336/pool-11-thread-3/6ac27/55dcc rollback AttemptContextReactive{id=55dcc,state=PENDING,atr=_default/_txn:atr-144-#171,staged=[INSERT test_docs-00027579, INSERT test_docs-00027612, INSERT test_docs-00027578, INSERT test_docs-00027611, INSERT test_docs-00027614]} expiryOvertimeMode=false isAppRollback=true 23:30:47.546 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:30:49.816 4336/pool-11-thread-3/6ac27/55dcc error 'DurabilityAmbiguousException' 'The server returned with a durability ambiguous response on this request' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:30:49.818 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:30:52.078 4336/pool-11-thread-3/6ac27/55dcc error 'DurabilityAmbiguousException' 'The server returned with a durability ambiguous response on this request' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:30:52.081 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:30:54.338 4336/pool-11-thread-3/6ac27/55dcc error 'DurabilityAmbiguousException' 'The server returned with a durability ambiguous response on this request' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:30:54.343 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:30:54.418 4336/pool-11-thread-3/6ac27/55dcc error 'RequestCanceledException' 'SubdocMutateRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT)' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:30:54.427 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:30:56.935 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:30:56.953 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:30:59.455 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:30:59.474 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:01.975 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:02.021 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:04.525 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:04.580 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:07.085 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:07.164 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:09.665 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:09.727 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:12.235 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:12.292 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:14.795 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:14.852 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:17.355 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:17.410 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:19.915 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:20.007 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:22.515 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:22.567 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:25.075 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:25.149 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:27.655 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:27.755 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:30.265 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:30.351 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:32.856 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:32.930 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:35.435 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:35.501 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:38.005 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:38.079 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:40.585 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:40.641 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:43.145 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:43.226 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:45.735 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:45.788 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:48.295 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:48.348 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:50.855 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:50.919 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:53.424 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:53.518 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:56.025 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:56.096 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:31:58.605 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:31:58.661 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:01.165 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:01.245 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:03.755 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:03.831 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:06.335 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:06.428 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:08.935 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:08.995 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:11.505 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:11.596 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:14.105 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:14.204 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:16.705 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:16.802 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:19.305 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:19.381 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:21.885 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:21.939 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:24.445 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:24.506 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:27.015 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:27.114 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:27.114 4336/pool-11-thread-3/6ac27 Has expired client side (now=7187691901697564ns start=7187589824351947ns deferredElapsed=0ns expired=102077millis config=100000ms) 23:32:27.114 4336/pool-11-thread-3/6ac27/55dcc expired in atrAbort 23:32:27.114 4336/pool-11-thread-3/6ac27/55dcc has expired in stage atrAbort 23:32:27.114 4336/pool-11-thread-3/6ac27/55dcc error 'AttemptExpired' 'Attempt has expired in stage atrAbort' FAIL_EXPIRY while aborting ATR default/_default/_txn:atr-144-#171 23:32:27.115 4336/pool-11-thread-3/6ac27/55dcc moving to expiry-overtime-mode in stage atrAbort 23:32:27.169 4336/pool-11-thread-3/6ac27/55dcc aborting ATR default/_default/_txn:atr-144-#171 isAppRollback=true ambiguityResolutionMode=false 23:32:27.169 4336/pool-11-thread-3/6ac27/55dcc not doing expiry check in atrAbort as already in expiry-overtime-mode 23:32:29.675 4336/pool-11-thread-3/6ac27/55dcc error 'AmbiguousTimeoutException' 'SubdocMutateRequest, Reason: TIMEOUT' FAIL_AMBIGUOUS while aborting ATR default/_default/_txn:atr-144-#171 23:32:29.675 4336/pool-11-thread-3/6ac27/55dcc in expiry-overtime mode so changing error 'com.couchbase.client.core.error.AmbiguousTimeoutException: SubdocMutateRequest, Reason: TIMEOUT {"cancelled":true,"clientContext":{"txn.op":"atrAbort"},"completed":true,"coreId":"0xdc25be61000000c3","idempotent":false,"lastChannelId":"DC25BE61000000C3/0000000059CD0069","lastDispatchedFrom":"172.23.106.162:40394","lastDispatchedTo":"172.23.123.102:11210","reason":"TIMEOUT","requestId":2574778,"requestType":"SubdocMutateRequest","retried":8,"retryReasons":["KV_NOT_MY_VBUCKET","KV_TEMPORARY_FAILURE"],"service":{"bucket":"default","collection":"_default","documentId":"_txn:atr-144-#171","opaque":"0x276f99","scope":"_default","syncDurability":{"present":true},"type":"kv"},"timeoutMs":2500,"timings":{"dispatchMicros":739,"encodingMicros":34,"totalMicros":2505883}}' to raise TRANSACTION_EXPIRED in stage 'atrAbort'; no rollback will be tried 23:32:29.677 4336/pool-11-thread-3/6ac27/55dcc caught exception 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:false, raise:TRANSACTION_EXPIRED}' in runBlocking, rethrowing 23:32:29.677 4336/pool-11-thread-3/6ac27/55dcc com.couchbase.transactions.error.internal.ErrorBuilder.build(ErrorBuilder.java:58) 23:32:29.677 4336/pool-11-thread-3/6ac27/55dcc com.couchbase.transactions.AttemptContextReactive.mapErrorInOvertimeToExpired(AttemptContextReactive.java:1616) 23:32:29.677 4336/pool-11-thread-3/6ac27/55dcc com.couchbase.transactions.AttemptContextReactive.lambda$atrAbort$136(AttemptContextReactive.java:2347) 23:32:29.677 4336/pool-11-thread-3/6ac27/<> finishing attempt off after error 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:false, raise:TRANSACTION_EXPIRED}' 23:32:29.677 4336/pool-11-thread-3/6ac27/55dcc has been told to skip auto-rollback 23:32:29.677 4336/pool-11-thread-3/6ac27/55dcc Adding cleanup request for _default/_txn:atr-144-#171 to run in 100000 msecs 23:32:29.677 4336/pool-11-thread-3/6ac27/55dcc added attempt TransactionAttempt{id=55dcc,state=PENDING,atrColl=com.couchbase.client.java.ReactiveCollection@7b57560b/_txn:atr-144-#171} after error 23:32:29.677 4336/pool-11-thread-3/6ac27/<> reraising original exception 'ErrorWrapper' com.couchbase.transactions.error.internal.AttemptExpired: com.couchbase.client.core.error.AmbiguousTimeoutException: SubdocMutateRequest, Reason: TIMEOUT {"cancelled":true,"clientContext":{"txn.op":"atrAbort"},"completed":true,"coreId":"0xdc25be61000000c3","idempotent":false,"lastChannelId":"DC25BE61000000C3/0000000059CD0069","lastDispatchedFrom":"172.23.106.162:40394","lastDispatchedTo":"172.23.123.102:11210","reason":"TIMEOUT","requestId":2574778,"requestType":"SubdocMutateRequest","retried":8,"retryReasons":["KV_NOT_MY_VBUCKET","KV_TEMPORARY_FAILURE"],"service":{"bucket":"default","collection":"_default","documentId":"_txn:atr-144-#171","opaque":"0x276f99","scope":"_default","syncDurability":{"present":true},"type":"kv"},"timeoutMs":2500,"timings":{"dispatchMicros":739,"encodingMicros":34,"totalMicros":2505883}} 23:32:29.677 4336/pool-11-thread-3/6ac27 ErrorWrapper retryTransaction=false 23:32:29.677 4336/pool-11-thread-3/6ac27/55dcc converted ErrorWrapper TRANSACTION_EXPIRED to final error com.couchbase.transactions.error.TransactionExpired: Transaction has expired configured timeout of 100000msecs. The transaction is not committed. 2020-07-14 23:32:29,681 | test | INFO | pool-11-thread-3 | [task:transaction_load:3987] Sleep for 60 seconds so that the txn gets cleaned up 23:30:47.564 4447/pool-11-thread-10/4758d starting attempt 0/4758d07e-d8cd-4aec-aa95-e5571d6a105a/cf419d0b-7587-435f-8cf1-4fc2843e8a3f 23:30:47.564 4447/pool-11-thread-10/4758d config: atrs=1024, expiry=100000msecs durability=PERSIST_TO_MAJORITY per-txn config: durability=Optional.empty 23:30:47.564 4447/pool-11-thread-10/4758d/cf419 First mutated doc in txn is 'test_docs-00014294' on vbucket 248, so using atr _txn:atr-248-#106 23:30:47.564 4447/pool-11-thread-10/4758d/cf419 expiration time remaining is 100000 millis (capped from 100000) 23:30:47.564 4447/pool-11-thread-10/4758d/cf419 about to set ATR default/_default/_txn:atr-248-#106 to Pending 23:30:49.816 4447/pool-11-thread-10/4758d/cf419 error while setting ATR default/_default/_txn:atr-248-#106 to Pending: {err=DurabilityAmbiguousException,ec=FAIL_AMBIGUOUS,msg='The server returned with a durability ambiguous response on this request'} 23:30:49.816 4447/pool-11-thread-10/4758d/cf419 retrying the op on FAIL_AMBIGUOUS to resolve ambiguity 23:30:49.816 4447/pool-11-thread-10/4758d/cf419 expiration time remaining is 97748 millis (capped from 97748) 23:30:49.819 4447/pool-11-thread-10/4758d/cf419 about to set ATR default/_default/_txn:atr-248-#106 to Pending 23:30:52.081 4447/pool-11-thread-10/4758d/cf419 error while setting ATR default/_default/_txn:atr-248-#106 to Pending: {err=DurabilityAmbiguousException,ec=FAIL_AMBIGUOUS,msg='The server returned with a durability ambiguous response on this request'} 23:30:52.081 4447/pool-11-thread-10/4758d/cf419 retrying the op on FAIL_AMBIGUOUS to resolve ambiguity 23:30:52.081 4447/pool-11-thread-10/4758d/cf419 expiration time remaining is 95484 millis (capped from 95484) 23:30:52.084 4447/pool-11-thread-10/4758d/cf419 about to set ATR default/_default/_txn:atr-248-#106 to Pending 23:30:54.418 4447/pool-11-thread-10/4758d/cf419 error while setting ATR default/_default/_txn:atr-248-#106 to Pending: {err=RequestCanceledException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT)'} 23:30:54.418 4447/pool-11-thread-10/4758d/cf419 retrying the op on FAIL_AMBIGUOUS to resolve ambiguity 23:30:54.418 4447/pool-11-thread-10/4758d/cf419 expiration time remaining is 93146 millis (capped from 93146) 23:30:54.422 4447/pool-11-thread-10/4758d/cf419 about to set ATR default/_default/_txn:atr-248-#106 to Pending 23:30:56.932 4447/pool-11-thread-10/4758d/cf419 error while setting ATR default/_default/_txn:atr-248-#106 to Pending: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:30:56.932 4447/pool-11-thread-10/4758d/cf419 retrying the op on FAIL_AMBIGUOUS to resolve ambiguity 23:30:56.932 4447/pool-11-thread-10/4758d/cf419 expiration time remaining is 90632 millis (capped from 90632) 23:30:56.936 4447/pool-11-thread-10/4758d/cf419 about to set ATR default/_default/_txn:atr-248-#106 to Pending 23:30:56.938 4447/pool-11-thread-10/4758d/cf419 error while setting ATR default/_default/_txn:atr-248-#106 to Pending: {err=PathExistsException,ec=FAIL_PATH_ALREADY_EXISTS,msg='Path already exists in document'} 23:30:56.938 4447/pool-11-thread-10/4758d/cf419 assuming this is caused by resolved ambiguity, and proceeding as though successful 23:30:56.938 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:30:59.442 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:30:59.446 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:01.953 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:01.956 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:04.462 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:04.465 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:06.973 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:06.976 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:09.482 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:09.485 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:11.992 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:11.995 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:14.502 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:14.505 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:17.012 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:17.015 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:19.522 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:19.525 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:22.032 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:22.035 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:24.543 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:24.547 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:27.053 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:27.057 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:29.562 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:29.565 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:32.073 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:32.076 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:34.583 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:34.586 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:37.093 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:37.096 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:39.602 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:39.605 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:42.113 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:42.116 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:44.622 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:44.626 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:47.132 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:47.135 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:49.642 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:49.646 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:52.153 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:52.156 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:54.662 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:54.665 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:57.173 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:57.176 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:31:59.683 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:59.686 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:02.192 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:02.195 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:04.702 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:04.706 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:07.212 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:07.215 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:09.723 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:09.726 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:12.232 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:12.235 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:14.742 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:14.746 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:17.252 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:17.255 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:19.762 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:19.765 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:22.272 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:22.275 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:24.783 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:24.786 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:27.293 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:27.296 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:29.803 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:29.807 4447/pool-11-thread-10/4758d/cf419 about to insert staged doc test_docs-00014294 as shadow document, cas=Optional.empty 23:32:29.807 4447/pool-11-thread-10/4758d Has expired client side (now=7187694594828493ns start=7187592352287690ns deferredElapsed=0ns expired=102242millis config=100000ms) 23:32:29.807 4447/pool-11-thread-10/4758d/cf419 expired in createdStagedInsert 23:32:29.807 4447/pool-11-thread-10/4758d/cf419 has expired in stage createdStagedInsert 23:32:29.807 4447/pool-11-thread-10/4758d/cf419 got err while staging insert of test_docs-00014294: {err=AttemptExpired,ec=FAIL_EXPIRY,msg='Attempt has expired in stage createdStagedInsert'} 23:32:29.807 4447/pool-11-thread-10/4758d/cf419 moving to expiry-overtime-mode in stage createdStagedInsert, and raising error 23:32:29.808 4447/pool-11-thread-10/4758d/cf419 caught exception 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}' in runBlocking, rethrowing 23:32:29.808 4447/pool-11-thread-10/4758d/cf419 com.couchbase.transactions.error.internal.ErrorBuilder.build(ErrorBuilder.java:58) 23:32:29.808 4447/pool-11-thread-10/4758d/cf419 com.couchbase.transactions.AttemptContextReactive.setExpiryOvertimeModeAndFail(AttemptContextReactive.java:2045) 23:32:29.808 4447/pool-11-thread-10/4758d/cf419 com.couchbase.transactions.AttemptContextReactive.lambda$null$63(AttemptContextReactive.java:1232) 23:32:29.808 4447/pool-11-thread-10/4758d/<> finishing attempt off after error 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}' 23:32:29.808 4447/pool-11-thread-10/4758d/cf419 told to auto-rollback but in NOT_STARTED state, so nothing to do - skipping rollback 23:32:29.808 4447/pool-11-thread-10/4758d/cf419 Skipping addition of cleanup request, as NOT_STARTED 23:32:29.808 4447/pool-11-thread-10/4758d/cf419 added attempt TransactionAttempt{id=cf419,state=NOT_STARTED,atrColl=com.couchbase.client.java.ReactiveCollection@530797e9/_txn:atr-248-#106} after error 23:32:29.808 4447/pool-11-thread-10/4758d/<> reraising original exception 'ErrorWrapper' com.couchbase.transactions.error.internal.AttemptExpired: com.couchbase.transactions.error.internal.AttemptExpired: Attempt has expired in stage createdStagedInsert 23:32:29.808 4447/pool-11-thread-10/4758d ErrorWrapper retryTransaction=false 23:32:29.808 4447/pool-11-thread-10/4758d/cf419 converted ErrorWrapper TRANSACTION_EXPIRED to final error com.couchbase.transactions.error.TransactionExpired: Transaction has expired configured timeout of 100000msecs. The transaction is not committed. 2020-07-14 23:32:29,812 | test | INFO | pool-11-thread-10 | [task:transaction_load:3987] Sleep for 60 seconds so that the txn gets cleaned up 23:30:47.330 4446/pool-11-thread-9/ca71d starting attempt 0/ca71dc90-2c0b-4ec1-a3e2-02a18dceb1da/6983fe7f-7a65-46c4-a24c-67147a15ef44 23:30:47.330 4446/pool-11-thread-9/ca71d config: atrs=1024, expiry=100000msecs durability=PERSIST_TO_MAJORITY per-txn config: durability=Optional.empty 23:30:47.330 4446/pool-11-thread-9/ca71d/6983f First mutated doc in txn is 'test_docs-00007601' on vbucket 396, so using atr _txn:atr-396-#b78 23:30:47.330 4446/pool-11-thread-9/ca71d/6983f expiration time remaining is 100000 millis (capped from 100000) 23:30:47.330 4446/pool-11-thread-9/ca71d/6983f about to set ATR default/_default/_txn:atr-396-#b78 to Pending 23:30:47.333 4446/pool-11-thread-9/ca71d/6983f set ATR default/_default/_txn:atr-396-#b78 to Pending in 2ms, got CAS (start time) cas=1594794647330619392,seqno=40,vbucket=396 23:30:47.333 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007601 as shadow document, cas=Optional.empty 23:30:47.337 4446/pool-11-thread-9/ca71d/6983f inserted doc test_docs-00007601 got cas=1594794647333634048,seqno=42,vbucket=396, in 3ms 23:30:47.337 4446/pool-11-thread-9/ca71d/6983f getting doc test_docs-00007601, resolvingMissingATREntry= 23:30:47.337 4446/pool-11-thread-9/ca71d/6983f found own-write of mutated doc test_docs-00007601 23:30:47.337 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional.empty 23:30:49.816 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=DurabilityAmbiguousException,ec=FAIL_AMBIGUOUS,msg='The server returned with a durability ambiguous response on this request'} 23:30:49.820 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional.empty 23:30:52.329 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:30:52.332 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional.empty 23:30:54.417 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=RequestCanceledException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT)'} 23:30:54.420 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional.empty 23:30:56.928 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:30:56.931 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional.empty 23:30:56.932 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=DocumentExistsException,ec=FAIL_DOC_ALREADY_EXISTS,msg='Document with the given id already exists'} 23:30:56.932 4446/pool-11-thread-9/ca71d/6983f DocExists on test_docs-00007567: getting doc 23:30:56.934 4446/pool-11-thread-9/ca71d/6983f DocExists on test_docs-00007567: doc test_docs-00007567 exists inTransaction=true isDeleted=true 23:30:56.935 4446/pool-11-thread-9/ca71d/6983f doc test_docs-00007567 has been written by this transaction, ok to continue 23:30:56.935 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:30:59.439 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:30:59.442 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:01.949 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:01.952 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:04.459 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:04.462 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:06.969 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:06.972 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:09.479 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:09.482 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:11.988 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:11.991 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:14.499 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:14.502 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:17.008 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:17.012 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:19.518 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:19.521 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:22.029 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:22.032 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:24.538 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:24.542 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:27.048 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:27.052 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:29.558 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:29.562 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:32.069 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:32.072 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:34.578 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:34.582 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:37.089 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:37.093 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:39.599 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:39.602 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:42.108 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:42.111 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:44.619 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:44.622 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:47.129 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:47.132 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:49.639 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:49.642 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:52.149 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:52.152 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:54.659 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:54.662 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:57.168 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:57.172 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:31:59.678 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:59.682 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:02.188 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:02.191 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:04.699 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:04.702 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:07.208 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:07.212 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:09.718 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:09.722 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:12.228 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:12.231 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:14.738 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:14.741 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:17.248 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:17.251 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:19.758 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:19.761 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:22.269 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:22.272 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:24.779 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:24.782 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:27.289 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:27.293 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:29.798 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:29.801 4446/pool-11-thread-9/ca71d/6983f about to insert staged doc test_docs-00007567 as shadow document, cas=Optional[1594794652077260800] 23:32:29.802 4446/pool-11-thread-9/ca71d Has expired client side (now=7187694589580806ns start=7187592117851825ns deferredElapsed=0ns expired=102471millis config=100000ms) 23:32:29.802 4446/pool-11-thread-9/ca71d/6983f expired in createdStagedInsert 23:32:29.802 4446/pool-11-thread-9/ca71d/6983f has expired in stage createdStagedInsert 23:32:29.802 4446/pool-11-thread-9/ca71d/6983f got err while staging insert of test_docs-00007567: {err=AttemptExpired,ec=FAIL_EXPIRY,msg='Attempt has expired in stage createdStagedInsert'} 23:32:29.802 4446/pool-11-thread-9/ca71d/6983f moving to expiry-overtime-mode in stage createdStagedInsert, and raising error 23:32:29.803 4446/pool-11-thread-9/ca71d/6983f caught exception 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}' in runBlocking, rethrowing 23:32:29.803 4446/pool-11-thread-9/ca71d/6983f com.couchbase.transactions.error.internal.ErrorBuilder.build(ErrorBuilder.java:58) 23:32:29.803 4446/pool-11-thread-9/ca71d/6983f com.couchbase.transactions.AttemptContextReactive.setExpiryOvertimeModeAndFail(AttemptContextReactive.java:2045) 23:32:29.803 4446/pool-11-thread-9/ca71d/6983f com.couchbase.transactions.AttemptContextReactive.lambda$null$63(AttemptContextReactive.java:1232) 23:32:29.803 4446/pool-11-thread-9/ca71d/<> finishing attempt off after error 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}' 23:32:29.803 4446/pool-11-thread-9/ca71d/6983f auto-rolling-back on error 23:32:29.804 4446/pool-11-thread-9/ca71d/6983f rollback AttemptContextReactive{id=6983f,state=ROLLED_BACK,atr=_default/_txn:atr-396-#b78,staged=[INSERT test_docs-00007601]} expiryOvertimeMode=true isAppRollback=false 23:32:29.804 4446/pool-11-thread-9/ca71d/6983f aborting ATR default/_default/_txn:atr-396-#b78 isAppRollback=false ambiguityResolutionMode=false 23:32:29.804 4446/pool-11-thread-9/ca71d/6983f not doing expiry check in atrAbort as already in expiry-overtime-mode 23:32:29.811 4446/pool-11-thread-9/ca71d/6983f aborted ATR default/_default/_txn:atr-396-#b78 23:32:29.811 4446/pool-11-thread-9/ca71d/6983f deleting inserted doc test_docs-00007601 with cas 1594794647333634048 23:32:29.811 4446/pool-11-thread-9/ca71d/6983f not doing expiry check in deleteInserted as already in expiry-overtime-mode 23:32:29.816 4446/pool-11-thread-9/ca71d/6983f deleted inserted doc test_docs-00007601, mt Optional[mt{vbID=396, vbUUID=203027522871119, seqno=49, bucket=default}] 23:32:29.816 4446/pool-11-thread-9/ca71d/6983f marking ATR default/_default/_txn:atr-396-#b78 as rollback complete 23:32:29.816 4446/pool-11-thread-9/ca71d/6983f ignoring expiry in stage atrRollbackComplete, as in expiry-overtime mode 23:32:29.818 4446/pool-11-thread-9/ca71d/6983f rollback - atr rolled back 23:32:29.818 4446/pool-11-thread-9/ca71d/6983f Adding cleanup request for _default/_txn:atr-396-#b78 to run in 100000 msecs 23:32:29.818 4446/pool-11-thread-9/ca71d/6983f added attempt TransactionAttempt{id=6983f,state=ROLLED_BACK,atrColl=com.couchbase.client.java.ReactiveCollection@f1c7955/_txn:atr-396-#b78} after error 23:32:29.818 4446/pool-11-thread-9/ca71d/<> reraising original exception 'ErrorWrapper' com.couchbase.transactions.error.internal.AttemptExpired: com.couchbase.transactions.error.internal.AttemptExpired: Attempt has expired in stage createdStagedInsert 23:32:29.819 4446/pool-11-thread-9/ca71d ErrorWrapper retryTransaction=false 23:32:29.819 4446/pool-11-thread-9/ca71d/6983f converted ErrorWrapper TRANSACTION_EXPIRED to final error com.couchbase.transactions.error.TransactionExpired: Transaction has expired configured timeout of 100000msecs. The transaction is not committed. 2020-07-14 23:32:29,822 | test | INFO | pool-11-thread-9 | [task:transaction_load:3987] Sleep for 60 seconds so that the txn gets cleaned up TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2686millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} 23:30:47.694 4334/pool-11-thread-1/f020c starting attempt 0/f020c906-7c2a-49c0-a78f-8eb30ac48fb4/45b541c8-af2a-4e4f-9aae-e3af0fccd874 23:30:47.694 4334/pool-11-thread-1/f020c config: atrs=1024, expiry=100000msecs durability=PERSIST_TO_MAJORITY per-txn config: durability=Optional.empty 23:30:47.694 4334/pool-11-thread-1/f020c/45b54 First mutated doc in txn is 'test_docs-00020938' on vbucket 822, so using atr _txn:atr-822-#a78 23:30:47.694 4334/pool-11-thread-1/f020c/45b54 expiration time remaining is 100000 millis (capped from 100000) 23:30:47.694 4334/pool-11-thread-1/f020c/45b54 about to set ATR default/_default/_txn:atr-822-#a78 to Pending 23:30:47.697 4334/pool-11-thread-1/f020c/45b54 set ATR default/_default/_txn:atr-822-#a78 to Pending in 2ms, got CAS (start time) cas=1594794647695196160,seqno=28,vbucket=822 23:30:47.697 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020938 as shadow document, cas=Optional.empty 23:30:47.699 4334/pool-11-thread-1/f020c/45b54 inserted doc test_docs-00020938 got cas=1594794647697948672,seqno=30,vbucket=822, in 2ms 23:30:47.699 4334/pool-11-thread-1/f020c/45b54 getting doc test_docs-00020938, resolvingMissingATREntry= 23:30:47.699 4334/pool-11-thread-1/f020c/45b54 found own-write of mutated doc test_docs-00020938 23:30:47.699 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020935 as shadow document, cas=Optional.empty 23:30:47.702 4334/pool-11-thread-1/f020c/45b54 inserted doc test_docs-00020935 got cas=1594794647700111360,seqno=41,vbucket=391, in 2ms 23:30:47.702 4334/pool-11-thread-1/f020c/45b54 getting doc test_docs-00020935, resolvingMissingATREntry= 23:30:47.702 4334/pool-11-thread-1/f020c/45b54 found own-write of mutated doc test_docs-00020935 23:30:47.702 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020936 as shadow document, cas=Optional.empty 23:30:49.915 4334/pool-11-thread-1/f020c/45b54 inserted doc test_docs-00020936 got cas=1594794647702142976,seqno=43,vbucket=142, in 2212ms 23:30:49.915 4334/pool-11-thread-1/f020c/45b54 getting doc test_docs-00020936, resolvingMissingATREntry= 23:30:49.915 4334/pool-11-thread-1/f020c/45b54 found own-write of mutated doc test_docs-00020936 23:30:49.915 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional.empty 23:30:52.421 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:30:52.424 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional.empty 23:30:54.418 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=RequestCanceledException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT)'} 23:30:54.422 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional.empty 23:30:56.931 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:30:56.934 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional.empty 23:30:56.935 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=DocumentExistsException,ec=FAIL_DOC_ALREADY_EXISTS,msg='Document with the given id already exists'} 23:30:56.935 4334/pool-11-thread-1/f020c/45b54 DocExists on test_docs-00020933: getting doc 23:30:56.936 4334/pool-11-thread-1/f020c/45b54 DocExists on test_docs-00020933: doc test_docs-00020933 exists inTransaction=true isDeleted=true 23:30:56.936 4334/pool-11-thread-1/f020c/45b54 doc test_docs-00020933 has been written by this transaction, ok to continue 23:30:56.936 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:30:59.441 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:30:59.444 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:01.951 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:01.954 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:04.461 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:04.464 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:06.971 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:06.974 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:09.481 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:09.485 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:11.991 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:11.994 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:14.501 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:14.504 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:17.011 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:17.014 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:19.521 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:19.524 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:22.031 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:22.034 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:24.541 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:24.544 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:27.051 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:27.054 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:29.561 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:29.564 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:32.071 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:32.074 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:34.581 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:34.585 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:37.091 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:37.094 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:39.601 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:39.604 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:42.110 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:42.114 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:44.621 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:44.624 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:47.131 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:47.134 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:49.641 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:49.644 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:52.150 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:52.154 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:54.661 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:54.664 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:57.170 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:57.174 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:31:59.681 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:31:59.684 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:02.191 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:02.194 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:04.701 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:04.704 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:07.211 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:07.214 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:09.721 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:09.724 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:12.231 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:12.234 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:14.741 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:14.745 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:17.251 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:17.254 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:19.760 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:19.764 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:22.271 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:22.274 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:24.781 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:24.784 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:27.291 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:27.294 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:29.801 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:29.804 4334/pool-11-thread-1/f020c/45b54 about to insert staged doc test_docs-00020933 as shadow document, cas=Optional[1594794652138602496] 23:32:29.804 4334/pool-11-thread-1/f020c Has expired client side (now=7187694592312609ns start=7187592482099839ns deferredElapsed=0ns expired=102110millis config=100000ms) 23:32:29.804 4334/pool-11-thread-1/f020c/45b54 expired in createdStagedInsert 23:32:29.804 4334/pool-11-thread-1/f020c/45b54 has expired in stage createdStagedInsert 23:32:29.804 4334/pool-11-thread-1/f020c/45b54 got err while staging insert of test_docs-00020933: {err=AttemptExpired,ec=FAIL_EXPIRY,msg='Attempt has expired in stage createdStagedInsert'} 23:32:29.804 4334/pool-11-thread-1/f020c/45b54 moving to expiry-overtime-mode in stage createdStagedInsert, and raising error 23:32:29.805 4334/pool-11-thread-1/f020c/45b54 caught exception 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}' in runBlocking, rethrowing 23:32:29.805 4334/pool-11-thread-1/f020c/45b54 com.couchbase.transactions.error.internal.ErrorBuilder.build(ErrorBuilder.java:58) 23:32:29.805 4334/pool-11-thread-1/f020c/45b54 com.couchbase.transactions.AttemptContextReactive.setExpiryOvertimeModeAndFail(AttemptContextReactive.java:2045) 23:32:29.805 4334/pool-11-thread-1/f020c/45b54 com.couchbase.transactions.AttemptContextReactive.lambda$null$63(AttemptContextReactive.java:1232) 23:32:29.805 4334/pool-11-thread-1/f020c/<> finishing attempt off after error 'ErrorWrapper {ec:FAIL_EXPIRY, cause:AttemptExpired, retry:false, autoRollback:true, raise:TRANSACTION_EXPIRED}' 23:32:29.805 4334/pool-11-thread-1/f020c/45b54 auto-rolling-back on error 23:32:29.805 4334/pool-11-thread-1/f020c/45b54 rollback AttemptContextReactive{id=45b54,state=ABORTED,atr=_default/_txn:atr-822-#a78,staged=[INSERT test_docs-00020938, INSERT test_docs-00020935, INSERT test_docs-00020936]} expiryOvertimeMode=true isAppRollback=false 23:32:29.806 4334/pool-11-thread-1/f020c/45b54 aborting ATR default/_default/_txn:atr-822-#a78 isAppRollback=false ambiguityResolutionMode=false 23:32:29.806 4334/pool-11-thread-1/f020c/45b54 not doing expiry check in atrAbort as already in expiry-overtime-mode 23:32:29.809 4334/pool-11-thread-1/f020c/45b54 aborted ATR default/_default/_txn:atr-822-#a78 23:32:29.809 4334/pool-11-thread-1/f020c/45b54 deleting inserted doc test_docs-00020938 with cas 1594794647697948672 23:32:29.809 4334/pool-11-thread-1/f020c/45b54 not doing expiry check in deleteInserted as already in expiry-overtime-mode 23:32:29.813 4334/pool-11-thread-1/f020c/45b54 deleted inserted doc test_docs-00020938, mt Optional[mt{vbID=822, vbUUID=104811645873981, seqno=36, bucket=default}] 23:32:29.813 4334/pool-11-thread-1/f020c/45b54 deleting inserted doc test_docs-00020935 with cas 1594794647700111360 23:32:29.813 4334/pool-11-thread-1/f020c/45b54 not doing expiry check in deleteInserted as already in expiry-overtime-mode 23:32:29.817 4334/pool-11-thread-1/f020c/45b54 deleted inserted doc test_docs-00020935, mt Optional[mt{vbID=391, vbUUID=119885558950940, seqno=47, bucket=default}] 23:32:29.818 4334/pool-11-thread-1/f020c/45b54 deleting inserted doc test_docs-00020936 with cas 1594794647702142976 23:32:29.818 4334/pool-11-thread-1/f020c/45b54 not doing expiry check in deleteInserted as already in expiry-overtime-mode 23:32:32.321 4334/pool-11-thread-1/f020c/45b54 error while rolling back inserted doc test_docs-00020936: {err=AmbiguousTimeoutException,ec=FAIL_AMBIGUOUS,msg='SubdocMutateRequest, Reason: TIMEOUT'} 23:32:32.321 4334/pool-11-thread-1/f020c/45b54 in expiry-overtime mode so changing error 'com.couchbase.client.core.error.AmbiguousTimeoutException: SubdocMutateRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xdc25be61000000c2","idempotent":false,"lastChannelId":"DC25BE61000000C2/000000008206A92F","lastDispatchedFrom":"172.23.106.162:40368","lastDispatchedTo":"172.23.123.102:11210","reason":"TIMEOUT","requestId":2577312,"requestType":"SubdocMutateRequest","retried":8,"retryReasons":["KV_NOT_MY_VBUCKET","KV_TEMPORARY_FAILURE"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-00020936","opaque":"0x27797f","scope":"_default","syncDurability":{"present":true},"type":"kv"},"timeoutMs":2500,"timings":{"dispatchMicros":869,"encodingMicros":1,"totalMicros":2503389}}' to raise TRANSACTION_EXPIRED in stage 'removeDoc'; no rollback will be tried 23:32:32.321 4334/pool-11-thread-1/f020c/<> rollback failed with 'ErrorWrapper' com.couchbase.transactions.error.internal.AttemptExpired: com.couchbase.client.core.error.AmbiguousTimeoutException: SubdocMutateRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xdc25be61000000c2","idempotent":false,"lastChannelId":"DC25BE61000000C2/000000008206A92F","lastDispatchedFrom":"172.23.106.162:40368","lastDispatchedTo":"172.23.123.102:11210","reason":"TIMEOUT","requestId":2577312,"requestType":"SubdocMutateRequest","retried":8,"retryReasons":["KV_NOT_MY_VBUCKET","KV_TEMPORARY_FAILURE"],"service":{"bucket":"default","collection":"_default","documentId":"test_docs-00020936","opaque":"0x27797f","scope":"_default","syncDurability":{"present":true},"type":"kv"},"timeoutMs":2500,"timings":{"dispatchMicros":869,"encodingMicros":1,"totalMicros":2503389}}, raising original error but with retryTransaction turned off 23:32:32.321 4334/pool-11-thread-1/f020c/45b54 Adding cleanup request for _default/_txn:atr-822-#a78 to run in 100000 msecs 23:32:32.321 4334/pool-11-thread-1/f020c/45b54 added attempt TransactionAttempt{id=45b54,state=ABORTED,atrColl=com.couchbase.client.java.ReactiveCollection@54fb9061/_txn:atr-822-#a78} after error 23:32:32.322 4334/pool-11-thread-1/f020c ErrorWrapper retryTransaction=false 23:32:32.322 4334/pool-11-thread-1/f020c/45b54 converted ErrorWrapper TRANSACTION_EXPIRED to final error com.couchbase.transactions.error.TransactionExpired: Transaction has expired configured timeout of 100000msecs. The transaction is not committed. 2020-07-14 23:32:32,325 | test | INFO | pool-11-thread-1 | [task:transaction_load:3987] Sleep for 60 seconds so that the txn gets cleaned up TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2678millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=7d86a,index=1,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=c81b4,index=2,numClients=3,ATRs={checking=341,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_default/clientId=5413a,index=0,numClients=3,ATRs={checking=342,total=1024},runLength=60000millis},end={taken=2687millis,ATRs={read=0,errored=0,present=0},entries={expired=0,failed=0,total=0},maxEntries=0} TransactionCleanupEndRunEvent: start={default/_defau