HLC is persisted at 'arbitrary' points and is not linked to the data-stream

It's not quite persisted randomly, but at points which don't really do much for consistency...

Each VB owns a maxCas value, this is generally the largest time stamp that the VB has seen, either its own reads when creating new HLC timestamps (for use as the CAS in mutations) or the HLC from DCP/setWithMeta mutations (it is only updated when the mutation has a higher HLC than our own).

The maxCas value has a place on disk for it (in the _local document) and when a VB warmups up, the _local maxCas value is used to initialise the VB. This is an optimisation to cope with full-eviction mode where we won't read all keys from disk.

However the writing of _local maxCas happens at the following points:

1. When a replica VB accepts a stream
2. When warmup completes
3. Every 300 seconds

So the persisted maxCas is divorced from the data stream enabling failover scenarios to 'rollback' the CAS
forcing XDCR conflict resolution to reject writes.

Perhaps there are simpler examples but the following illustrates the issue.

Vbucket A DCP replicates to A'
Vbucket A also XDCR replicates to B, B is a pure backup and only receives data from A

• A receives 5 mutations and generates a stream [seqno, HLC], HLC generated from local time.
  • [0,10], [1,11], [2,22], [3,24], [4,150]
• A' receives the stream and updates the maxCas of A'
• B receives the stream and updates the maxCas of B
• A and A' have not yet written maxCas to disk, assume maxCas on disk is 5
• Something happens, A' is promoted to be active.
• B now follows to A'.
• A' restarts though, causing it to warmup and set maxCas from disk.
• Mutations occur against A', a new CAS will be generated and there's no guarantee that the new HLC will be > 150
  • Maybe the following occurs [5, 140], [6,145], [7,149], [8, 151]
• B receives the new mutations and will drop seqno 5, 6 and 7.

I've marked as a regression as this wouldn't happened with rev-id based XDCR