Transaction throughput dropped from 1800 trans/sec to 300 trans/sec on build 7.6.0-1451

Avg Throughput (trans/sec), Workload T, 4 nodes, 1 bucket x 1B x 1KB, Persist_Majority, Uniform distribution, 2% Resident Ratio, Magma

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=rhea_760-1419_access_b8f8&snapshot=rhea_760-1483_access_e56f&snapshot=rhea_760-1451_access_af46

The runs using build 7.6.0-1451 and 7.6.0-1483 were able to do similar amount of ops in the first 5 minutes, but ops dropped to almost 0 later.

In the log, there are many transaction error messages.

http://perf.jenkins.couchbase.com/job/rhea-5node2/3914/artifact/YCSB/output.log

21:18:48.372 [Thread-106] WARN c.y.y.d.c.Couchbase3Client.bad - transaction failed with exception :21:16:48.320 130/Thread-106/c4233 library version: 1.2.1 config: atrs=20480, metadataCollection=Optional.empty, expiry=120000msecs durability=PERSIST_TO_MAJORITY per-txn config= durability=Optional.empty, supported=Supported {extensions=[EXT_TRANSACTION_ID, EXT_DEFERRED_COMMIT, EXT_TIME_OPT_UNSTAGING, EXT_BINARY_METADATA, EXT_CUSTOM_METADATA_COLLECTION, EXT_QUERY, EXT_STORE_DURABILITY, BF_CBD_3838, BF_CBD_3787, BF_CBD_3705, BF_CBD_3794, EXT_REMOVE_COMPLETED, EXT_ALL_KV_COMBINATIONS, EXT_UNKNOWN_ATR_STATES, BF_CBD_3791, EXT_SINGLE_QUERY], protocol=2.1}