Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-102

Detect when mobile service node topology changes

    XMLWordPrintable

Details

    • Task
    • Status: Closed
    • Critical
    • Resolution: Won't Do
    • None
    • 3.0
    • MobileService
    • None

    Attachments

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

      Activity

        There is a basic implementation in the prototype code, however it has the following bug:

        1. Remove mobile-service-2 node from the cluster, and rebalance
        2. mobile-service-2 will get the PrepareTopologyChange() where it is listed as an ejected node, but it currently ignores it.
        3. mobile-service-1 gets a StartTopologyChange() message, and cleans up the /mobile/state/mobile-service-2/gw-x metakv entry, since it knows mobile-service-2 is going away
        4. mobile-service-2 gets stats heartbeat messages from the gateway still connected to it, and restores the /mobile/state/mobile-service-2/gw-x metakv entry
        5. mobile-service-2 node is completely removed from the cluster, the mobile-service process is restarted and disconnect all gateway nodes, which reconnect to other mobile service nodes
        6. the /mobile/state/mobile-service-2/gw-x entry is never removed
        traun Traun Leyden (Inactive) added a comment - There is a basic implementation in the prototype code, however it has the following bug: Remove mobile-service-2 node from the cluster, and rebalance mobile-service-2 will get the PrepareTopologyChange() where it is listed as an ejected node, but it currently ignores it. mobile-service-1 gets a StartTopologyChange() message, and cleans up the /mobile/state/mobile-service-2/gw-x metakv entry, since it knows mobile-service-2 is going away mobile-service-2 gets stats heartbeat messages from the gateway still connected to it, and restores the /mobile/state/mobile-service-2/gw-x metakv entry mobile-service-2 node is completely removed from the cluster, the mobile-service process is restarted and disconnect all gateway nodes, which reconnect to other mobile service nodes the /mobile/state/mobile-service-2/gw-x entry is never removed

        Proposed fix:

        • When a node receives a PrepareTopologyChange() callback where it has been ejected, it should:
          • Disconnect all connected gateways
          • Stop accepting new gateway incoming connections until further notice – for example another topology change that adds the node back into the cluster.
        traun Traun Leyden (Inactive) added a comment - Proposed fix: When a node receives a PrepareTopologyChange() callback where it has been ejected, it should: Disconnect all connected gateways Stop accepting new gateway incoming connections until further notice – for example another topology change that adds the node back into the cluster.

        People

          The One The One
          traun Traun Leyden (Inactive)
          Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

            Created:
            Updated:
            Resolved:

            Gerrit Reviews

              There are no open Gerrit changes

              PagerDuty