Subdoc DICT_UPSERT with durability intermittently stomping on previous DICT_UPSERT operation

Observing DurableWriteInProgressException exception while setting ATR to pending status which eventually ends up as MultiMutationException causes the transaction to fail when transactiondurability is set to MAJORITY .

 

Cluster Setup : 

Nodes : 3

Replica_number : 2

Durability : MAJOR

Items : 10M

transaction distribution : 3 READS & 1 UPDATE

requestDistribution : uniform

Test logs for analysis : http://perf.jenkins.couchbase.com/job/hebe-txn/31/artifact/celery/

transactoin-SNAPSHOT : last updated  Jul 29 09:53

 

 

Transaction log Snippet :

Transaction logger:16:26:53.104 31/Thread-10/8dfd1 starting attempt 0/8dfd1ae5-76d4-40fa-8017-311bc76a318b/ce714145-4aa0-4e3b-9215-7d94d15f70f6 Transaction logger:16:26:53.104 31/Thread-10/8dfd1 config: expiry=PT15S durability=MAJORITY per-txn config: durability=Optional.empty Transaction logger:16:26:53.105 31/Thread-10/8dfd1/ce714 getting doc usertable:user5924161920252886633 Transaction logger:16:26:53.105 31/Thread-10/8dfd1/ce714 completed get of TransactionJsonDocument{id=usertable:user5924161920252886633,cas=1564529069544767488,status=NORMAL,bucket=bucket-1,coll=_default,links=TransactionLinks{atr=none,atrBkt=none,atrColl=none,ver=none,op=none,restore=

{none,none,-1}}} in 0ms Transaction logger:16:26:53.105 31/Thread-10/8dfd1/ce714 replace doc TransactionJsonDocument{id=usertable:user5924161920252886633,cas=1564529069544767488,status=NORMAL,bucket=bucket-1,coll=_default,links=TransactionLinks{atr=none,atrBkt=none,atrColl=none,ver=none,op=none,restore={none,none,-1}

}} Transaction logger:16:26:53.105 31/Thread-10/8dfd1/ce714 First mutated doc in txn is 'usertable:user5924161920252886633' on vbucket 1013, so using atr atr-1013-#18e Transaction logger:16:26:53.105 31/Thread-10/8dfd1/ce714 about to set ATR bucket-1/_default/atr-1013-#18e to Pending Transaction logger:16:26:53.106 31/Thread-10/8dfd1/ce714 error com.couchbase.client.core.error.DurableWriteInProgressException while setting ATR bucket-1/_default/atr-1013-#18e to Pending Transaction logger:16:26:53.107 31/Thread-10/8dfd1/ce714 caught exception 'com.couchbase.client.core.error.DurableWriteInProgressException' in runBlocking, rethrowing Transaction logger:16:26:53.107 31/Thread-10/8dfd1/ce714 exception DurableWriteInProgressException needs to be wrapped as AttemptWrappedExceptionNeedsRetry Transaction logger:16:26:53.107 31/Thread-10/8dfd1/<> hit error TransactionWrappedExceptionNeedsRetry(com.couchbase.client.core.error.DurableWriteInProgressException) Transaction logger:16:26:53.107 31/Thread-10/8dfd1/ce714 rolling back on error Transaction logger:16:26:53.107 31/Thread-10/8dfd1/ce714 Adding cleanup request for _default/atr-1013-#18e to run in 15000 msecs Transaction logger:16:26:53.107 31/Thread-10/8dfd1/ce714 added attempt TransactionAttempt{id=ce714,state=PENDING,atrColl=com.couchbase.client.java.ReactiveCollection@7f1f8a8b/atr-1013-#18e} after error Transaction logger:16:26:53.107 31/Thread-10/8dfd1/ce714 rollback AttemptContextReactive{id=ce714,state=ROLLED_BACK,isDone=true,atr=_default/atr-1013-#18e,staged=[]} Transaction logger:16:26:53.107 31/Thread-10/8dfd1/ce714 aborting (retrying if needed) AttemptContextReactive{id=ce714,state=ROLLED_BACK,isDone=true,atr=_default/atr-1013-#18e,staged=[]} Transaction logger:16:26:53.109 31/Thread-10/8dfd1/ce714 could not find ATR entry in bucket-1/_default/atr-1013-#18e Transaction logger:16:26:53.109 31/Thread-10/8dfd1/ce714 aborting ATR bucket-1/_default/atr-1013-#18e Transaction logger:16:26:53.109 31/Thread-10/8dfd1/ce714 atrAbortRetryUntilExpiry: error com.couchbase.client.core.error.subdoc.MultiMutationException: Multiple mutation could not be applied. First problematic failure at 0 with status PATH_NOT_FOUND while doing something with ATR bucket-1/_default/atr-1013-#18e Transaction logger:16:26:53.109 31/Thread-10/8dfd1/ce714 atrAbortRetryUntilExpiry: ATR entry for bucket-1/_default/atr-1013-#18e does not exist, assuming we're rolling back as it could not be created and continuing Transaction logger:16:26:53.109 31/Thread-10/8dfd1/ce714 aborted ATR bucket-1/_default/atr-1013-#18e Transaction logger:16:26:53.109 31/Thread-10/8dfd1/ce714 marking ATR bucket-1/_default/atr-1013-#18e as rollback complete Transaction logger:16:26:53.110 31/Thread-10/8dfd1/ce714 atrRollbackComplete: error com.couchbase.client.core.error.subdoc.MultiMutationException: Multiple mutation could not be applied. First problematic failure at 0 with status PATH_NOT_FOUND while doing something with ATR bucket-1/_default/atr-1013-#18e Transaction logger:16:26:53.110 31/Thread-10/8dfd1/ce714 atrRollbackComplete: ATR entry for bucket-1/_default/atr-1013-#18e does not exist, assuming we're rolling back as it could not be created and continuing Transaction logger:16:26:53.110 31/Thread-10/8dfd1/ce714 rollback - atr rolled back Transaction logger:16:26:53.110 31/Thread-10/8dfd1/<> retrying on error 'TransactionWrappedExceptionNeedsRetry(com.couchbase.client.core.error.DurableWriteInProgressException)' Transaction logger:16:26:53.123 31/Thread-10/8dfd1 starting attempt 1/8dfd1ae5-76d4-40fa-8017-311bc76a318b/ef7d13c7-ae57-42bd-ba53-2d5e069cf014 Transaction logger:16:26:53.123 31/Thread-10/8dfd1 config: expiry=PT15S durability=MAJORITY per-txn config: durability=Optional.empty