Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.2.0
-
Untriaged
-
0
-
Unknown
Description
CDC can include duplicates (different version) of a key in a flush batch.
KV has always written flush batches to the KVStore sorted as by key, then by seqno. This has been a key part of the flusher de-duplication since its creation.
input order to flusher:
|
key: b a b a
|
seq: 1 2 3 4
|
|
sorted:
|
key: a a b b
|
seq: 2 1 4 3
|
|
final set written to KVStore:
|
key: a b
|
seq: 2 4
|
With CDC, the final set can now include duplicates, e.g. if the b keys are from a collection with history enabled the output to the KVStore includes two 'b' keys in descending order
final set written to KVStore:
|
key: a b b
|
seq: 2 4 3
|
When processing such a batch, magma assumed it was working in ascending key order, e.g. it may ignore b:4 or assume it's older than b:3... (note that the seqnos are mostly abstract for magma).
This MB tracks this issue.
Note that the simplest way forward in the short-term is for KV to sort the final batch by seqno which will get everything functional.
Note note: couchstore already (attempts to) sort the batch we handover, twice! Each id/seq index and couchstore sorts the flush input by id or seq.
Attachments
For Gerrit Dashboard: MB-55199 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
185328,2 | MB-55199: Remove duplicate value_t docBody from MagmaRequest | neo | kv_engine | Status: MERGED | +2 | +1 |
185329,6 | MB-55199: Sort magma pendingRequests into key and ascending seqno order | neo | kv_engine | Status: MERGED | +2 | +1 |
186786,3 | MB-55199: New WriteDocs ordering only required when History::Enabled | neo | kv_engine | Status: MERGED | +2 | +1 |
187426,1 | Merge neo/d0452196a into master | master | kv_engine | Status: MERGED | +2 | +1 |
187706,4 | Merge neo/4b6a1791d into master | master | kv_engine | Status: ABANDONED | -1 | -1 |
187777,2 | Merge commit neo/6b697af92 into master | master | kv_engine | Status: ABANDONED | -1 | +1 |
187787,2 | Merge commit neo/8d301c5af into master | master | kv_engine | Status: MERGED | +2 | +1 |
191495,3 | Merge commit neo/99bf690ab into master | master | kv_engine | Status: MERGED | +2 | +1 |