Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35852

Wait for downstream processes to start while retrying the "leave" procedure

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      When a node is ejected out of the cluster, the following set of operations are performed by the ns_cluster process on the node leaving the cluster:

      1. Creates a “leave marker” to indicate the beginning on the leave procedure.
      2. Empties the users storage.
      3. Stops most of the ns_server processes.
      4. Wipes out the stats archive.
      5. Disconnects from the other nodes in the cluster by regenerating a new Erlang cookie.
      6. Resets the address of the node back to 127.0.0.1
      7. Clears the ns_config copy and reinitialize with default values.
      8. Updates the nodes_wanted list to self.
      9. Creates a “start processes marker” to indicate that the ns_server processes were being restarted (the ones stopped earlier).
      10. Deletes the “leave marker”.
      11. Restarts all the stopped processes.
      12. Restarts memcached.
      13. Deletes the “start processes marker” to signify the end of the leave procedure.

      If for some reason ns_cluster crashes before completing step 10, then when ns_cluster is restarted it attempts perform the leave again as the marker has not been cleared. As part of the leave when ns_cluster tries to perform step 2 it crashes with noproc exception as the downstream process responsible for clearing the user storage wouldn't have started yet.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            dfinlay Dave Finlay
            ajit.yagaty Ajit Yagaty [X] (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty