Details
-
Improvement
-
Resolution: Won't Fix
-
Major
-
None
-
None
-
1
Description
We don't have any documentation details how the Kafka connector handles crashes or other unexpected shutdowns. Answered from a recent discussion with David Nault
Q: If the Kafka connector unexpectedly shuts down, are the messages in-flight completely lost ? Does the connector perform a retry if we stop/reboot it ?
No, the connector will not lose messages if it terminates unexpectedly. Every message it sends to Kafka has the DCP sequence number (seqno) associated with it. The Kafka Connect framework periodically saves the seqno of the last message it committed to the topic. When the connector starts up, the framework passes this seqno back to the connector and the connector uses it to resume the DCP stream.
Attachments
Issue Links
- relates to
-
DOC-5492 Kafka Connector failure scenarios
- Closed