last_modified can be incorrect due to CAS poisoning

Logging this issue for awareness, there isn't a planned fix.

The vattr 'last_modified' field is derived from the document's CAS (which for many releases has effectively been a timestamp, a Hybrid Logical Clock). KV-engine even tracks the first seqno generated by Spock (which introduced last_modified) so that it can safely exclude the last_modified field from old documents which may have an effectively random CAS.

However a known problem exists with the HLC which we term poisoning and this problem impacts on last_modified.

Poisoning:

The HLC to ensure it can guarantee ordering with its peer, monitors the peers clock values and if the peer drifts ahead, then the local HLC adjusts (forwards only) to be in keeping with the peer, once it has jumped forwards, it goes into logical mode, which means it ticks only when read. This works well for small amounts of drift and the local real clock would eventually catch-up with the HLC, to which the HLC switches back to using the real clock. However if the peers clock was badly adjusted (perhaps an accidental clock change) and 20 years were added to the real clock, then the HLC will effectively adjust by 20 years too, now if the peer fixes its clock and goes back by 20 years, it's too late, the local HLC never goes backwards and will sit in logical-mode until the real clock catches up, in-effect the CAS is poisoned and will now generate incorrect timestamps.

A workaround does exist when the CAS becomes poisoned, in that epctl allows the user to change the HLC value, unwinding the bad clock, however it may not be obvious without monitoring that they need to do this.