Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47923

XDCR - incremental topology change: Rebalance

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Fixed
    • Major
    • 7.1.0
    • 7.1.0
    • XDCR
    • 1

    Description

      The first part of the deliverable for XDCR to behave well on topology change is to handle the rebalancing case. This MB will encompass the few other task MBs which provides the functionality for VB rebalancing support.

      Using Peer-To-Peer communication, when an XDCR replication starts, The requester node performs the following any of the checkpoints are read or any parts are started:

      1. Figures out who its peer nodes are
      2. Figures out which VBs that this current node owns (and will be replicating)
      3. Sends a request packet listing the VBs that this node owns to all peer nodes and wait to hear back.

      A peer node would:

      1. Receive the request
      2. Check to make sure that the each VB in the request is not owned by this node. If so, respond back saying there's a conflict.
      3. Retrieve from its local simple_store for stored checkpoints for the VBs from the request.
      4. Retrieve from its local simple_store for stored backfill replication/tasks for the VBs.
      5. Retrieve any broken mappings and/or collection manifests associated with checkpoints and backfill replication specs.
      6. Retrieve checkpoints for backfill replication, if any.
      7. Compile the information, compress, and send the information back to the requester node.

      The requester node receives information from peers and:

      1. Checks to make sure that none of the other nodes believe they are VB masters for the requested VBs (if so, bail and try again)
      2. Merge checkpoints information, sort them (by recent-ness), and then truncate to max number of record per VB, and then persist the truncated checkpoints to its local simple_store
      3. Merge backfill replication information and persist them to local simple_store
      4. Persist any non-referred collection manifests and/or broken mapping from the peers.

      At this point, any newly merged backfill replication info will trigger backfill pipelines to start. And, the main pipeline will start after the ckpt merges are done. Using the peer node's checkpoint metadata, a node that is rebalanced-in should no longer need to restream from seqno 0.

      The whole peer-to-peer communication sequence will introduce amount of processing and time to start a replication. There are a new knobs introduced:

      Replication Setting
      Each replication will now have a new setting called preReplicateVBMasterCheck (can consider renaming if necessary), which defaults to true. Setting this false will skip the peer-to-peer information pull + merge, and start pipeline the way it did before this MB.

      Internal Settings
      A few internal settings are introduced with this change:

      1. PipelineTimeoutP2PProtocolSec - The amount of time the whole peer-to-peer communication should finish before declaring execution timed out for pipeline start. (Currently defaults to 60 sec)
      2. P2PVBRelatedGCIntervalHour - Once a node no longer owns a VB, it will keep checkpoints and backfill replication tasks for this VB for this period. Once this period has passed, it assumes that any other node that own said VB(s) have already pulled the metadata, and the metadata will then be purged. (Defaults to 24 hours).
      3. P2PCommTimeoutSec - The amount of time a HTTP REST POST command should take from one node to another peer. (Defaults to 15 seconds).

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavithra.mahamani Pavithra Mahamani (Inactive)
              neil.huang Neil Huang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty