Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31832

DCP notification of durable persistence

    XMLWordPrintable

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • DCP
    • None

    Description

      This is a feature request to have Couchbase Server send notifications over DCP when document changes have been durably persisted.

      One use case is for the Kafka and Elasticsearch connectors, which don't act on changes until they have been persisted to all replicas. The current implementation polls Couchbase Server using the observeseqnos command, but this is complicated and inefficient.

      It would be great if instead of polling, clients could opt in to receive periodic notifications of the highest sequence number that has been durably persistence.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            drigby Dave Rigby added a comment -

            Could you perhaps give some more detail as to why the Kafka & ElasticSearch connectors wait for data to be persisted to all replicas? Note that's more paranoid than typical use-cases (and in fact the proposed Synchronous Replication will only support persistToMajority as the strictest durability level.

            Additionally; it's worth noting that _currently the Active vBuckets' node has no idea what the high/persisted seqno is on the replicas - it only knows the seqno for itself (the active node). The changes to support synchronous durability would be needed as a prerequisite for anything along these lines...

            drigby Dave Rigby added a comment - Could you perhaps give some more detail as to why the Kafka & ElasticSearch connectors wait for data to be persisted to all replicas? Note that's more paranoid than typical use-cases (and in fact the proposed Synchronous Replication will only support persistToMajority as the strictest durability level. Additionally; it's worth noting that _currently the Active vBuckets' node has no idea what the high/persisted seqno is on the replicas - it only knows the seqno for itself (the active node). The changes to support synchronous durability would be needed as a prerequisite for anything along these lines...
            david.nault David Nault added a comment -

            Could you perhaps give some more detail as to why the Kafka & ElasticSearch connectors wait for data to be persisted to all replicas?

            The connectors have no way of retracting or reverting changes when a rollback occurs, so they try to not propagate events until it's unlikely they will be rolled back.

            The changes to support synchronous durability would be needed as a prerequisite for anything along these lines...

            Yes... this is a long-term request. Just wanted to get it into the pipeline.

            david.nault David Nault added a comment - Could you perhaps give some more detail as to why the Kafka & ElasticSearch connectors wait for data to be persisted to all replicas? The connectors have no way of retracting or reverting changes when a rollback occurs, so they try to not propagate events until it's unlikely they will be rolled back. The changes to support synchronous durability would be needed as a prerequisite for anything along these lines... Yes... this is a long-term request. Just wanted to get it into the pipeline.

            This applies to eventing as well - once we trigger user's handler, there's no way to undo the action.

            siri Sriram Melkote (Inactive) added a comment - This applies to eventing as well - once we trigger user's handler, there's no way to undo the action.

            People

              drigby Dave Rigby
              david.nault David Nault
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty