Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-26907

DCP: Differentiate Between Delete and Expiry Operations

    XMLWordPrintable

Details

    Description

      DCP doesn’t differentiate between delete/expiry – both come down the wire to dcp consumer (eventing) as opcode 0x58 (DCP_DELETION). Like differentiating b/w create & update (MB-26905) , we would need dcp producer to implement differentiation b/w expiry & delete as well. Delete caused due to Expiry and 'normal' Delete should also be captured separately.

      Sample dump from a test(run by Abhishek Singh ): 

      Created doc (DCP_MUTATION – opcode 0x57) with 10 sec expiry

      2017-10-11T10:16:20.781+05:30 [Debug] DCPT[secidx:rawupr/0] ##54 DCP_SNAPSHOT for vb 84
      2017-10-11T10:16:20.781+05:30 [Trace] dcp event dump &memcached.DcpEvent\{Opcode:0x57, Status:0x0, Datatype:0x1, VBucket:0x54, Opaque:0x54, VBuuid:0x1fe4b693c08b, Key:[]uint8\{0x74, 0x73, 0x5f, 0x30}, Value:[]uint8\{0x7b, 0x22, 0x74, 0x79, 0x70, 0x65, 0x22, 0x3a, 0x22, 0x74, 0x72, 0x61, 0x76, 0x65, 0x6c, 0x5f, 0x73, 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x22, 0x2c, 0x22, 0x64, 0x6f, 0x63, 0x5f, 0x69, 0x64, 0x22, 0x3a, 0x30, 0x2c, 0x22, 0x73, 0x72, 0x63, 0x22, 0x3a, 0x22, 0x42, 0x4c, 0x52, 0x22, 0x2c, 0x22, 0x64, 0x73, 0x74, 0x22, 0x3a, 0x22, 0x44, 0x45, 0x4c, 0x22, 0x7d}, OldValue:[]uint8(nil), Cas:0x14ec6a9909510000, Seqno:0xc, RevSeqno:0x1, Flags:0x0, Expiry:0x59dda227, LockTime:0x0, Nru:0x2, SnapstartSeq:0x0, SnapendSeq:0x0, SnapshotType:0x0, FailoverLog:(*memcached.FailoverLog)(nil), Error:error(nil), Ctime:1507697180781464929}
      2017-10-11T10:16:20.781+05:30 [Trace]      \{"type":"travel_sample","doc_id":0,"src":"BLR","dst":"DEL"}
      

       
      Around 10 sec, got DCP_DELETION opcode instead of expected DCP_EXPIRATION (for this I made expiry pager to run every 1s instead of default 3600s)

      2017-10-11T10:16:32.356+05:30 [Debug] DCPT[secidx:rawupr/0] ##54 DCP_SNAPSHOT for vb 84
      2017-10-11T10:16:32.356+05:30 [Trace] dcp event dump &memcached.DcpEvent\{Opcode:0x58, Status:0x0, Datatype:0x0, VBucket:0x54, Opaque:0x54, VBuuid:0x1fe4b693c08b, Key:[]uint8\{0x74, 0x73, 0x5f, 0x30}, Value:[]uint8\{}, OldValue:[]uint8(nil), Cas:0x14ec6a9bbb400000, Seqno:0xd, RevSeqno:0x12, Flags:0x0, Expiry:0x0, LockTime:0x0, Nru:0x0, SnapstartSeq:0x0, SnapendSeq:0x0, SnapshotType:0x0, FailoverLog:(*memcached.FailoverLog)(nil), Error:error(nil), Ctime:1507697192356903614}
      

       

      Eventing won’t be able to expose OnExpiry handler without DCP producer(KV-Engine) side changes. Because of above DCP behavior, if user creates doc timer to notify on expiry – they will also get call to their OnDelete handler as well when expiry pager runs. 

       

      This is an issue as Delete and Expiry are two very different operations with vary different business implications.

      From a consumer's(example: Eventing) perspective, it is very important that we provide a mechanism so that customers have OnDelete and OnExpiry handler. As of now, the only way possible is to create a Timer on the document few minutes before expiry. Though this might sound like a possible option, it has 2 consequences:

      1. Timer's execution is not guaranteed at the specified time if the load on the system is high & the nodes are under provisioned.

      2. Requires a metadata bucket(future collection) wherein the information is stored - this requires that Sizing be more considerate if there is non-trivial amount of documents and their respective timers.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-6.5.0-1857 contains kv_engine commit b791fa8 with commit message:
            MB-26907: Documentation additions for DCP Enable Expiry Output

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1857 contains kv_engine commit b791fa8 with commit message: MB-26907 : Documentation additions for DCP Enable Expiry Output

            Build couchbase-server-6.5.0-1913 contains kv_engine commit c76b02d with commit message:
            MB-26907: Remove unused LAST from DeleteSource

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1913 contains kv_engine commit c76b02d with commit message: MB-26907 : Remove unused LAST from DeleteSource

            Build couchbase-server-6.5.0-1914 contains kv_engine commit 67c6c66 with commit message:
            MB-26907: Fix deletionCause in Item operators and copy constructor

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1914 contains kv_engine commit 67c6c66 with commit message: MB-26907 : Fix deletionCause in Item operators and copy constructor

            christopher farman have marked this as resolved as I believe all of the work is in - please let me know if there's more to be done.

            dhaikney David Haikney added a comment - christopher farman have marked this as resolved as I believe all of the work is in - please let me know if there's more to be done.

            Verified on  6.5.0-4744.

            Opcode Dump - Receive: 0x57 CMD_MUTATION
            Opcode Dump - Receive: 0x58 CMD_DELETION
            Opcode Dump - Receive: 0x59 CMD_EXPIRATION

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Verified on  6.5.0-4744. Opcode Dump - Receive: 0x57 CMD_MUTATION Opcode Dump - Receive: 0x58 CMD_DELETION Opcode Dump - Receive: 0x59 CMD_EXPIRATION

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              venkatraman.subramanian Venkatraman Subramanian (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty