RequestTimeoutException while setting ATR to Pending

Getting RequestTimeoutException exception while setting ATR to pending status while running load test with 120 concurrent threads/requests . Below is the error snippet for one of the transaction .

Please take a look at the log attached for more details as there few more exceptions related to  TemporaryFailureException & DocumentAlreadyInTransaction . 

(I was not able to comprehend if these these are inter related MB-34347) 

CB Build: 6.5.0-3274

SDK : 3.0.0-alpha.3

Couchbase-transactions : 1.0.0-alpha.4

 

Transaction error log :

Transaction logger:32/Thread-11/87853366-55a6-47e6-9491-a75418b3dabe starting attempt 0/dec1be32-3c3d-4115-9866-7dc74274ccd5
Transaction logger:32/Thread-11/87853366-55a6-47e6-9491-a75418b3dabe/dec1be32-3c3d-4115-9866-7dc74274ccd5 getting doc usertable:user707278906139368243
Transaction logger:32/Thread-11/87853366-55a6-47e6-9491-a75418b3dabe/dec1be32-3c3d-4115-9866-7dc74274ccd5 completed get of TransactionJsonDocument{id=usertable:user707278906139368243,cas=1558659019246469120,status=NORMAL,bucket=bucket-1,coll=_default,links=TransactionLinks{atr=none,atrBkt=none,atrColl=none,ver=none}} in 32ms
Transaction logger:32/Thread-11/87853366-55a6-47e6-9491-a75418b3dabe/dec1be32-3c3d-4115-9866-7dc74274ccd5 replace doc TransactionJsonDocument{id=usertable:user707278906139368243,cas=1558659019246469120,status=NORMAL,bucket=bucket-1,coll=_default,links=TransactionLinks{atr=none,atrBkt=none,atrColl=none,ver=none}}
Transaction logger:32/Thread-11/87853366-55a6-47e6-9491-a75418b3dabe/dec1be32-3c3d-4115-9866-7dc74274ccd5 First mutated doc in txn is 'usertable:user707278906139368243' on vbucket 628, so using atr atr-628-#21b
Transaction logger:32/Thread-11/87853366-55a6-47e6-9491-a75418b3dabe/dec1be32-3c3d-4115-9866-7dc74274ccd5 about to set ATR bucket-1/_default/atr-628-#21b to Pending
Transaction logger:32/Thread-11/87853366-55a6-47e6-9491-a75418b3dabe/dec1be32-3c3d-4115-9866-7dc74274ccd5 error com.couchbase.client.core.error.RequestTimeoutException: SubdocMutateRequest {"retried":0,"reason":"TIMEOUT","requestId":170,"timeoutMs":2500,"service":

{"bucket":"bucket-1","type":"kv","key":"atr-628-#21b","cid":0}

,"cancelled":true,"coreId":15,"completed":true} while setting ATR bucket-1/_default/atr-628-#21b to Pending
[DEBUG] (cb-timer-12-1) Stopping retries since predicate returned false, retry context: iteration=1 exception=com.couchbase.client.core.error.RequestTimeoutException: SubdocGetRequest {"retried":13,"reason":"TIMEOUT","requestId":429,"timeoutMs":2500,"service":

{"bucket":"bucket-1","type":"kv","key":"atr-681-#178","cid":0}

,"cancelled":true,"coreId":13,"completed":true} backoff={11ms/250ms}
Transaction logger:32/Thread-11/87853366-55a6-47e6-9491-a75418b3dabe/dec1be32-3c3d-4115-9866-7dc74274ccd5 caught exception 'com.couchbase.client.core.error.RequestTimeoutException: SubdocMutateRequest {"retried":0,"reason":"TIMEOUT","requestId":170,"timeoutMs":2500,"service":

{"bucket":"bucket-1","type":"kv","key":"atr-628-#21b","cid":0}

,"cancelled":true,"coreId":15,"completed":true}' in asyncInternal, rethrowing to rollback
Transaction logger:32/Thread-11/87853366-55a6-47e6-9491-a75418b3dabe/dec1be32-3c3d-4115-9866-7dc74274ccd5 com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:131)
com.couchbase.client.core.Timer.lambda$register$1(Timer.java:96)
com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:682)
com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:757)
com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:485)
com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
Transaction logger:32/Thread-11/87853366-55a6-47e6-9491-a75418b3dabe/dec1be32-3c3d-4115-9866-7dc74274ccd5 exception RequestTimeoutException needs to wrapped as AttemptWrappedExceptionNeedsRetry
Transaction logger:32/Thread-11/87853366-55a6-47e6-9491-a75418b3dabe/<> hit error TransactionWrappedExceptionNeedsRetry(com.couchbase.client.core.error.RequestTimeoutException: SubdocMutateRequest {"retried":0,"reason":"TIMEOUT","requestId":170,"timeoutMs":2500,"service":

{"bucket":"bucket-1","type":"kv","key":"atr-628-#21b","cid":0}

,"cancelled":true,"coreId":15,"completed":true})
[DEBUG] (elastic-42) Scheduling retry attempt, retry context: iteration=2 exception=TransactionWrappedExceptionNeedsRetry(com.couchbase.client.core.error.RequestTimeoutException: SubdocGetRequest {"retried":13,"reason":"TIMEOUT","requestId":433,"timeoutMs":2500,"service":

{"bucket":"bucket-1","type":"kv","key":"usertable:user3426306227801922038","cid":0}

,"cancelled":true,"coreId":7,"completed":true}) backoff={18ms/250ms}