Linearizability Failure during Failover Workload

Linearizability failure found during kv-engine-madhatter-jepsen-post-commit-82, while running against the failover workload where we perform a hard failover of a node and then ten seconds later we perform a delta recovery of the node.
Leinin command of the test:
lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --package /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install --workload=failover --failover-type=hard --recovery-type=delta --replicas=2 --no-autofailover --disrupt-count=1 --rate=0 --durability=0:100:0:0 --eviction-policy=value --cas --hashdump --enable-memcached-debug-log-level --enable-tcp-capture --autofailover-timeout=30

Logs can be found here for the test with PCAP and debug level logs.

First mad-hatter build containing the patches for this commit is couchbase-server-6.5.0-4880.(FYI note that Jepsen is statistical and that the failing patch may not the one that this post-commit job was for.)