Uploaded image for project: 'Couchbase Documentation'
  1. Couchbase Documentation
  2. DOC-9606

Document query service graceful shutdown on rebalance remove

    XMLWordPrintable

Details

    • Task
    • Status: Closed
    • Major
    • Resolution: Done
    • Neo
    • Neo
    • query
    • None
    • DOC-2022-S6
    • 1

    Description

      When a node is removed and rebalanced, the query service will allow existing queries and transactions to complete before shutting down.
      This may result in an elongated rebalance operation.  The query service diagnostic log on the node(s) being removed will contain messages indicating how many transactions and queries are still running.  Any new connection attempts to nodes that are shutting down will receive error 1180 and may receive error 1181 in the brief period between the completion of the last statement or transaction and the service exiting.  Such rejected requests will have HTTP status code 503 (service unavailable) set too.

      This does not affect failover.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ray Offiah , the text of the errors is here:

            https://github.com/couchbase/query/blob/cc51ddcbe6ac2a28de7ddd35166f0f59aa3e8d85/errors/service.go#L113

            https://github.com/couchbase/query/blob/cc51ddcbe6ac2a28de7ddd35166f0f59aa3e8d85/errors/service.go#L118

            E_SERVICE_SHUTTING_DOWN (1180) is reported when a node has been instructed to shut down gracefully and a new connection to the node is attempted.

            E_SERVICE_SHUT_DOWN (1181) is reported when the graceful shutdown has completed but ns_server has yet to kill the node.  (In reality the window for hitting this error is small.)

            Donald.haggart Donald Haggart added a comment - Ray Offiah , the text of the errors is here: https://github.com/couchbase/query/blob/cc51ddcbe6ac2a28de7ddd35166f0f59aa3e8d85/errors/service.go#L113 https://github.com/couchbase/query/blob/cc51ddcbe6ac2a28de7ddd35166f0f59aa3e8d85/errors/service.go#L118 E_SERVICE_SHUTTING_DOWN (1180) is reported when a node has been instructed to shut down gracefully and a new connection to the node is attempted. E_SERVICE_SHUT_DOWN (1181) is reported when the graceful shutdown has completed but ns_server has yet to kill the node.  (In reality the window for hitting this error is small.)
            ray.offiah Ray Offiah added a comment -

            Donald Haggart  <-- Just what I was looking for. Thanks very much.

            ray.offiah Ray Offiah added a comment - Donald Haggart   <-- Just what I was looking for. Thanks very much.
            ray.offiah Ray Offiah added a comment -

            Fixed and merged

            ray.offiah Ray Offiah added a comment - Fixed and merged
            simon.dew Simon Dew added a comment -

            Reopened to fix resolution

            simon.dew Simon Dew added a comment - Reopened to fix resolution
            simon.dew Simon Dew added a comment -

            Closed and resolved.

            simon.dew Simon Dew added a comment - Closed and resolved.

            People

              ray.offiah Ray Offiah
              Donald.haggart Donald Haggart
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty