Sink: Missed updates when using N1QL mode

Description

A connector configured to use N1QL mode could mistakenly ignore updates if Kafka records targeting the same document(s) arrive in rapid succession.

This happened because the connector was using an event deduplication algorithm based on the document ID / Kafka record key: if two records with the key/ID arrive in the same batch, the first record was dropped, and the connecter never applied the updates from the dropped record.

Even if two Kafka records have different IDs, the updates could be applied out of order since N1QL queries are executed concurrently. As a result of this race condition, it was theoretically possible for the updates from the first record to be applied after the updates from the second record.

Environment

None

Gerrit Reviews

None

Release Notes Description

None

Activity

Show:
Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Story Points

Fix versions

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created June 24, 2021 at 6:27 PM
Updated June 25, 2021 at 10:01 PM
Resolved June 25, 2021 at 10:01 PM
Instabug