last_modified can be incorrect if bucket contains old documents

In summary last_modified cannot be wholly trusted in 5.0+ until we implement a solution (summary solution at end of description). If the data-set is brand new in 5.0, all is OK (although MB-27709 exists against last_modified), however old buckets which have been upgraded and have old documents, last_modified cannot be relied upon.

For now consider last_modified 'advisory', don't use it in a court of law...

last_modified is generated from the document's CAS and the CAS is a hybrid-logical clock timestamp if the document was updated by a version of couchbase which uses HLC CAS

If a document is old, it's CAS cannot be used as a last_modified timestamp.

In 5.0 we added some code which tracked the lowest seqno written (http://review.couchbase.org/#/c/79127/), we call this seqno the hlc_epoch. Any document with a seqno >= hlc_epoch must have a HLC CAS and can thus display last_modified. Documents with a seqno < hlc_epoch may or may not have a HLC CAS (as HLC was introduced in 4.6.2), so those documents do not display a last_modified vattr.

However the implementation of hlc_epoch is flawed, basically it is not replicated, only offline upgrade will correctly set hlc_epoch and the first rebalance will break it.

Consider that we have offline upgraded from 4.0 to 5.0, every document in the vbucket has an old format CAS, the first write to 5.0 sets the hlc_epoch say to 999. Documents below 999 are from 4.0, >= 999 are from 5.0.

Now we add a new node and rebalance.

On the new node, a new vbucket is built (a DCP consumer), the consumer receives the first seqno over DCP (1) and writes it, it blindly sets the hlc_epoch to be the lowest seqno, which is 1, but the CAS is not a HLC CAS.

Note XDCR presents a similar issue, if the bucket is populated by XDCR we don't know the pedigree of the CAS

Possible solution (although the XDCR case would need to replicate the epoch too ):

Effectively redo http://review.couchbase.org/#/c/79127/ however:

• _local will persist a new epoch named "hlc_epoch2", the old "hlc_epoch" field gets ignored
• DCP must transmit the hlc_epoch2 so the consumer can initialise its hlc_epoch2
• Some way for XDCR to do the same