Description
When a node is removed and rebalanced, the query service will allow existing queries and transactions to complete before shutting down.
This may result in an elongated rebalance operation. The query service diagnostic log on the node(s) being removed will contain messages indicating how many transactions and queries are still running. Any new connection attempts to nodes that are shutting down will receive error 1180 and may receive error 1181 in the brief period between the completion of the last statement or transaction and the service exiting. Such rejected requests will have HTTP status code 503 (service unavailable) set too.
This does not affect failover.
Attachments
Issue Links
- relates to
-
MB-40533 Implement cluster aware rebalance protocol
-
- Closed
-
- links to
Ray Offiah , the text of the errors is here:
https://github.com/couchbase/query/blob/cc51ddcbe6ac2a28de7ddd35166f0f59aa3e8d85/errors/service.go#L113
https://github.com/couchbase/query/blob/cc51ddcbe6ac2a28de7ddd35166f0f59aa3e8d85/errors/service.go#L118
E_SERVICE_SHUTTING_DOWN (1180) is reported when a node has been instructed to shut down gracefully and a new connection to the node is attempted.
E_SERVICE_SHUT_DOWN (1181) is reported when the graceful shutdown has completed but ns_server has yet to kill the node. (In reality the window for hitting this error is small.)