Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19217

[FTS] MCP: after stop-rebalance, and then another rebalance, replica pindexes not cleaned up

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 4.5.0
    • 4.5.0
    • fts
    • None
    • Untriaged
    • Unknown

    Description

      Found this while playing around with OSX cluster-run dev envionment.

      Setup a multi-node cluster with FTS enabled, like node A & B.

      Also setup an FTS index with 0 replicas configured (xxx).

      Next, in ns-server's web admin UI, do a Remove Server on a FTS node B, then hit Rebalance.

      Next, in the midst of the rebalancing, hit Stop Rebalance.

      At that point, you might have some pindex (X) be in replica state on node A (which staying in the cluster) and master state on node B (which was leaving the cluster before the rebalance was canceled).

      Next, in ns-server's web admin UI, cancel the pending removal of node B, and hit the Rebalance button again.

      After this second rebalancing is done, you might be in a state (see attached screen shot) where the replica assignment has (incorrectly) to node A not been removed. Further attempts at Rebalance (while the topology is not changing) at this point also do not help fix this situation.

      First guess on diagnosis, this looks to be an unintended consequence of a recent MCP fix/change which "moves" a pindex (X) from node B to node A by first assigning X to node A in replica state (as opposed to the old behavior of assigning X to node A in master state). So, when the user hits Stop Rebalance, pindex X on node A is left in replica state. Also, a following Rebalance doesn't clean up those leftover replicas, because the FTS index never had replicas configured (at step xxx), so the planner doesn't even look at replicas.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            steve Steve Yen
            steve Steve Yen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty