Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54446

XDCR - Spammy and incorrect "Source Topology may have changed"

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • Morpheus
    • 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.1.0, 7.1.1, 7.1.2
    • XDCR
    • None
    • Untriaged
    • 1
    • No

    Description

      First

      2022-09-30T03:26:56.471Z WARN GOXDCR.Utils: Can't find high seqno for vbno=895 in stats map. Source topology may have changed.
      

      The message is actually misleading.

      As part of XDCR pipeline creation, XDCR actually gets a list of target nodes and their VBs:
      https://src.couchbase.org/source/xref/6.6.5/goproj/src/github.com/couchbase/goxdcr/factory/xdcr_factory.go#180

      The target_kv_vb_map is then passed to checkpoint manager:
      https://src.couchbase.org/source/xref/6.6.5/goproj/src/github.com/couchbase/goxdcr/factory/xdcr_factory.go#240
      https://src.couchbase.org/source/xref/6.6.5/goproj/src/github.com/couchbase/goxdcr/factory/xdcr_factory.go#866

      So anytime this message shows up:

      2022-09-30T03:23:49.538Z WARN GOXDCR.Utils: Can't find high seqno for vbno=895 in stats map. Source topology may have changed.
      

      It really should mean “Target Topology may have changed”.

      Secondly
      The Topology Change Detector technically gets a kv_vb map and then filters the un-necessary VB's out (i.e. those that are not related to source VBs), and only watch for those.
      This means that if Target rebalances and gains or loses VBs that are not pertaining to the original target VB map, there's no issue.

      However, Checkpoint manager does not. It caches the original "target_kv_vb_map" and never filters it. So if some VBs move away from certain target node, but are not part of the pipeline (since source don't coincide), then we'll start seeing the spammy "Source topology may have changed" message incorrectly.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sumukh.bhat Sumukh Bhat
            neil.huang Neil Huang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty