Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34262

Ephemeral tombstone's delete time can be incorrect

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      This MB is a follow on of MB-33919, forward merge of the fix has triggered failure in some tests that exist in master but not alice (where MB-33919 was committed).

      The issue is isolated to ephemeral buckets only.

      The code to set the tombstone exists in Vbucket::queueDirty and works by updating the queued_item with the tombstone time to use.

      The bug is visible in a test that does a DCP backfill.

      • In persistent buckets this works as the queued_item will be used for DCP and writing to disk, a later disk backfill will read items from disk and see our generated delete time.
      • In ephemeral the queueDirty queued_item is only consumed by in-memory DCP. If a DCP backfill occurs, DCP is fed items from the hash-table's seqno ordered linked-list, and the OrderedStoredValue representing the tombstone still has the original expiry time set by the client, and as demonstrated by MB-33919 could be junk.

      However in-terms of tombstone purging, MB-33919 does not fully occur because the OrderStoredValue has a separate tombstone field for the tombstone time, any replica built with backfill items though will use the wrong tombstone timestamp.

      That is

      • An item deleted/expired will on the active have the correct tombstone time
      • Any backfilled DCP delete though is not get given the proper delete-time, a delete-time of 0 for example is used for items with no expiry, otherwise the raw expiry is used. For KV replica's fortunatley when processing an incoming DCP deletion, the OrderedStoredValue has the deletion time set by the delete path, so an incorrectly on-the-wire value of 0 is corrected to something sane. However re-calculating the delete-time allows for MB-27457 to occur.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Please share the steps to verify

            anitha.kuberan Anitha Kuberan added a comment - Please share the steps to verify

            Anitha Kuberan do you have the required information now to validate this bug ?

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Anitha Kuberan do you have the required information now to validate this bug ?
            jwalker Jim Walker added a comment -

            Approximate steps to reproduce the bug e.g. using 6.0.0 and verify the fix using 6.0.2

            Setup

            1. Create an ephemeral bucket
            2. Populate the bucket with many items say 10k that have an expiry of 0.
            3. Delete all of the items

            Recreate, we need a DCP stream that performs a backfill

            1. Create a DCP client and for all stream-requests, start from seqno 0 and set the flags field as 0x02 (disk-only)
              • On persistent and ephemeral this has the affect of forcing a backfill
            2. DCP will now stream many DCP_DELETION packets to the client, if the bug is present the expiry field is 0, if fixed it will be a timestamp near to when the item was deleted.
            jwalker Jim Walker added a comment - Approximate steps to reproduce the bug e.g. using 6.0.0 and verify the fix using 6.0.2 Setup Create an ephemeral bucket Populate the bucket with many items say 10k that have an expiry of 0. Delete all of the items Recreate, we need a DCP stream that performs a backfill Create a DCP client and for all stream-requests, start from seqno 0 and set the flags field as 0x02 (disk-only) On persistent and ephemeral this has the affect of forcing a backfill DCP will now stream many DCP_DELETION packets to the client, if the bug is present the expiry field is 0, if fixed it will be a timestamp near to when the item was deleted.

            Verified on 6.0.2-2392

            anitha.kuberan Anitha Kuberan added a comment - Verified on 6.0.2-2392

            Verified on 6.5.0-3458

            anitha.kuberan Anitha Kuberan added a comment - Verified on 6.5.0-3458

            People

              anitha.kuberan Anitha Kuberan
              jwalker Jim Walker
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty