Details
Description
First
2022-09-30T03:26:56.471Z WARN GOXDCR.Utils: Can't find high seqno for vbno=895 in stats map. Source topology may have changed.
|
The message is actually misleading.
As part of XDCR pipeline creation, XDCR actually gets a list of target nodes and their VBs:
https://src.couchbase.org/source/xref/6.6.5/goproj/src/github.com/couchbase/goxdcr/factory/xdcr_factory.go#180
The target_kv_vb_map is then passed to checkpoint manager:
https://src.couchbase.org/source/xref/6.6.5/goproj/src/github.com/couchbase/goxdcr/factory/xdcr_factory.go#240
https://src.couchbase.org/source/xref/6.6.5/goproj/src/github.com/couchbase/goxdcr/factory/xdcr_factory.go#866
So anytime this message shows up:
2022-09-30T03:23:49.538Z WARN GOXDCR.Utils: Can't find high seqno for vbno=895 in stats map. Source topology may have changed.
|
It really should mean “Target Topology may have changed”.
Secondly
The Topology Change Detector technically gets a kv_vb map and then filters the un-necessary VB's out (i.e. those that are not related to source VBs), and only watch for those.
This means that if Target rebalances and gains or loses VBs that are not pertaining to the original target VB map, there's no issue.
However, Checkpoint manager does not. It caches the original "target_kv_vb_map" and never filters it. So if some VBs move away from certain target node, but are not part of the pipeline (since source don't coincide), then we'll start seeing the spammy "Source topology may have changed" message incorrectly.