Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29809

last_modified can be incorrect if bucket contains old documents

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Is this a Regression?:
      No

      Description

      In summary last_modified cannot be wholly trusted in 5.0+ until we implement a solution (summary solution at end of description). If the data-set is brand new in 5.0, all is OK (although MB-27709 exists against last_modified), however old buckets which have been upgraded and have old documents, last_modified cannot be relied upon.

      For now consider last_modified 'advisory', don't use it in a court of law...

      last_modified is generated from the document's CAS and the CAS is a hybrid-logical clock timestamp if the document was updated by a version of couchbase which uses HLC CAS

      If a document is old, it's CAS cannot be used as a last_modified timestamp.

      In 5.0 we added some code which tracked the lowest seqno written (http://review.couchbase.org/#/c/79127/), we call this seqno the hlc_epoch. Any document with a seqno >= hlc_epoch must have a HLC CAS and can thus display last_modified. Documents with a seqno < hlc_epoch may or may not have a HLC CAS (as HLC was introduced in 4.6.2), so those documents do not display a last_modified vattr.

      However the implementation of hlc_epoch is flawed, basically it is not replicated, only offline upgrade will correctly set hlc_epoch and the first rebalance will break it.

      Consider that we have offline upgraded from 4.0 to 5.0, every document in the vbucket has an old format CAS, the first write to 5.0 sets the hlc_epoch say to 999. Documents below 999 are from 4.0, >= 999 are from 5.0.

      Now we add a new node and rebalance.

      On the new node, a new vbucket is built (a DCP consumer), the consumer receives the first seqno over DCP (1) and writes it, it blindly sets the hlc_epoch to be the lowest seqno, which is 1, but the CAS is not a HLC CAS.

      Note XDCR presents a similar issue, if the bucket is populated by XDCR we don't know the pedigree of the CAS

      Possible solution (although the XDCR case would need to replicate the epoch too ):

      Effectively redo http://review.couchbase.org/#/c/79127/ however:

      • _local will persist a new epoch named "hlc_epoch2", the old "hlc_epoch" field gets ignored
      • DCP must transmit the hlc_epoch2 so the consumer can initialise its hlc_epoch2
      • Some way for XDCR to do the same

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          dfinlay Dave Finlay added a comment - - edited

          Jim - Based on my reading of the bug I don't think there are any significant logic errors in things like conflict resolution that can occur as a result of this – and that this bug is limited to potential incorrectness of the last_modified virtual xattr. Can you confirm this?

          I'll assume this is the case and move this ticket to Mad Hatter. If the effect of this bug is potentially more serious let me know.

          Show
          dfinlay Dave Finlay added a comment - - edited Jim - Based on my reading of the bug I don't think there are any significant logic errors in things like conflict resolution that can occur as a result of this – and that this bug is limited to potential incorrectness of the last_modified virtual xattr. Can you confirm this? I'll assume this is the case and move this ticket to Mad Hatter. If the effect of this bug is potentially more serious let me know.
          Hide
          jwalker Jim Walker added a comment -

          Correct: only an issue for the last_modified virtual attr. LWW conflict resolution won't be enabled on buckets with pre-HLC documents

          Show
          jwalker Jim Walker added a comment - Correct: only an issue for the last_modified virtual attr. LWW conflict resolution won't be enabled on buckets with pre-HLC documents

            People

            • Assignee:
              jwalker Jim Walker
              Reporter:
              jwalker Jim Walker
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.