Details
-
Bug
-
Resolution: Unresolved
-
Major
-
5.5.0, 5.5.1, 5.5.2, 5.5.3, 5.5.4, 5.5.5, 5.5.6, 5.5.5-MP2, 6.0.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.5.1, 6.6.0, 6.6.1, 6.6.2, 6.5.2, 6.5.0, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 7.0.0-Beta1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.1.4, 7.0.5, 7.1.0, 7.1.1, 7.1.2, 7.2.0, 7.1.3
-
None
-
Untriaged
-
0
-
No
Description
Summary: XDCR’s Advanced Filtering has a limitation where if a document key is an invalid JSON string, XDCR’s filter will not be able to parse it and the document will not be replicated regardless of whether or not the key matches the regular expression or not.
The detail of this is based on how XDCR performs key filtering.
When users enter a key based filter, such as REGEXP_CONTAINS(META().id, “^abc”), and when XDCR receives a document, it will perform some processing, depending on whether or not body or xattribute are being used.
In a general case, if only keys are being used, then XDCR will create a small temporary JSON object like the following:
{“[$%XDCRInternalKey*%$]” : “<documentKey>”}
|
This JSON object is then passed to gojsonsm to perform matching.
The issue here is that the <documentKey> would need to be a valid JSON string. Most of the time, customers have simple keys that fit the criteria. But, because it is not a requirement, some customers have document keys that don’t conform to JSON string standards.
When this happens, the gojsonsm is not able to parse the JSON object and filtering will fail, leading to potential false negatives (i.e. docs that should have passed the filter but did not).
The right thing to do here would be to handle document IDs and make sure that the generated string is a valid JSON string, and that the key being used/passed in as part of the user's input is also valid such that it matches the constructed string.
In the meantime, it would be good to watch for warnings such as:
2023-03-22T10:39:41.112-07:00 WARN GOXDCR.GenericSupervisor: 616d5c35209295d13fb634bf77dd33ee/B1/B2 - Last 1 filtering errors: Filter error: in string escape code - gojsonsm filter returned err in string escape code () for document <ud>something123\abc</ud>, data: <ud>{"[$%XDCRInternalKey*%$]”:”something123\abc"}</ud>
|
as evidence that this issue is being seen in a deployment.