Uploaded image for project: 'Couchbase Documentation'
  1. Couchbase Documentation
  2. DOC-5702

Kafka Connector: Clarify message handling during a failure

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Won't Fix
    • Major
    • None
    • None
    • connectors
    • 1

    Description

      We don't have any documentation details how the Kafka connector handles crashes or other unexpected shutdowns. Answered from a recent discussion with David Nault

      Q: If the Kafka connector unexpectedly shuts down, are the messages in-flight completely lost ? Does the connector perform a retry if we stop/reboot it ?

      No, the connector will not lose messages if it terminates unexpectedly. Every message it sends to Kafka has the DCP sequence number (seqno) associated with it. The Kafka Connect framework periodically saves the seqno of the last message it committed to the topic. When the connector starts up, the framework passes this seqno back to the connector and the connector uses it to resume the DCP stream.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              richard.smedley Richard Smedley
              ianmccloy Ian McCloy (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty