Uploaded image for project: 'Couchbase Kafka Connector'
  1. Couchbase Kafka Connector
  2. KAFKAC-64

Specify start/end condition for streaming

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: .future
    • Labels:
      None

      Description

      Certain use cases call for extracting a "snapshot" of data rather than streaming continuously.  It would be nice if there were a configuration option that allowed streaming to stop at a certain point, e.g. StreamFrom.BEGINNING, StreamTo.NOW

      Currently two approaches evaluated:

      1) setting like "action_on_restart", which would take several values like "continue" (current implementation relying on the state the connector tracks and serialize into kafka offsets), "from_now" (connectors will pull the current sequence numbers and just start listening new changes), "from_beginning" (rollback to zero).

      2) programmatic extension point, just like we have for custom filters for example[1]. The connector will call this when restart has been detected, give it the current state as input, the object, which allows to perform some state queries to the server and expect this hook to return new state. The implementation would be like this (pseudo code):

      pubic class ContinueOnRestart implements RestartHook {
          public SessionState onRestart(SessionState currentState, DcpConnection connection) {
              if (GlobalSettings.startFromNow()) {
                  return connection.fetchCurrentStateFromServer();
              } else if (GlobalSettings.continue()) {
                  return currentState;
              } else {
                  return SessionState.EMPTY;
              }
          }
      }
      

      [1]: https://github.com/couchbaselabs/kafka-example-filter/blob/master/src/main/java/example/FieldFilter.java

       

      Questions:

      In start from now mode, the connector should not rollback to zero?

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            david.nault David Nault added a comment - - edited

            Splitting this into two separate issues for start and end conditions. See KAFKAC-86 for the start condition.

            Show
            david.nault David Nault added a comment - - edited Splitting this into two separate issues for start and end conditions. See KAFKAC-86  for the start condition.

              People

              • Assignee:
                david.nault David Nault
                Reporter:
                dproctor Dean Proctor (Inactive)
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.