Uploaded image for project: 'Couchbase Kafka Connector'
  1. Couchbase Kafka Connector
  2. KAFKAC-64

Specify start/end condition for streaming

    XMLWordPrintable

Details

    • New Feature
    • Resolution: Unresolved
    • Major
    • .future
    • None
    • None

    Description

      Certain use cases call for extracting a "snapshot" of data rather than streaming continuously.  It would be nice if there were a configuration option that allowed streaming to stop at a certain point, e.g. StreamFrom.BEGINNING, StreamTo.NOW

      Currently two approaches evaluated:

      1) setting like "action_on_restart", which would take several values like "continue" (current implementation relying on the state the connector tracks and serialize into kafka offsets), "from_now" (connectors will pull the current sequence numbers and just start listening new changes), "from_beginning" (rollback to zero).

      2) programmatic extension point, just like we have for custom filters for example[1]. The connector will call this when restart has been detected, give it the current state as input, the object, which allows to perform some state queries to the server and expect this hook to return new state. The implementation would be like this (pseudo code):

      pubic class ContinueOnRestart implements RestartHook {
          public SessionState onRestart(SessionState currentState, DcpConnection connection) {
              if (GlobalSettings.startFromNow()) {
                  return connection.fetchCurrentStateFromServer();
              } else if (GlobalSettings.continue()) {
                  return currentState;
              } else {
                  return SessionState.EMPTY;
              }
          }
      }
      

      [1]: https://github.com/couchbaselabs/kafka-example-filter/blob/master/src/main/java/example/FieldFilter.java

       

      Questions:

      In start from now mode, the connector should not rollback to zero?

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              david.nault David Nault
              dproctor Dean Proctor (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty