Details
-
New Feature
-
Resolution: Unresolved
-
Major
-
None
-
None
Description
Certain use cases call for extracting a "snapshot" of data rather than streaming continuously. It would be nice if there were a configuration option that allowed streaming to stop at a certain point, e.g. StreamFrom.BEGINNING, StreamTo.NOW
Currently two approaches evaluated:
1) setting like "action_on_restart", which would take several values like "continue" (current implementation relying on the state the connector tracks and serialize into kafka offsets), "from_now" (connectors will pull the current sequence numbers and just start listening new changes), "from_beginning" (rollback to zero).
2) programmatic extension point, just like we have for custom filters for example[1]. The connector will call this when restart has been detected, give it the current state as input, the object, which allows to perform some state queries to the server and expect this hook to return new state. The implementation would be like this (pseudo code):
pubic class ContinueOnRestart implements RestartHook { |
public SessionState onRestart(SessionState currentState, DcpConnection connection) { |
if (GlobalSettings.startFromNow()) { |
return connection.fetchCurrentStateFromServer(); |
} else if (GlobalSettings.continue()) { |
return currentState; |
} else { |
return SessionState.EMPTY; |
}
|
}
|
}
|
Questions:
In start from now mode, the connector should not rollback to zero?
Attachments
Issue Links
- relates to
-
KAFKAC-86 Source: Restart from a given state / offset
- Resolved