In case if /-/reload takes more time than configured timeout (currently 5 sec), ns_server keeps retrying indefinitely which leads to backed up tcp backlog in prometheus's web server. Which makes it impossible to run any http request against prometheus which we should avoid.
Probably we should use exponential back off when retrying to reload config (at least in case when
happens - but this is debatable)
|An issue occurred where the Cluster Manager instructed Prometheus to reload the configuration and the reload timeout impacted other requests.||The Cluster Manager has been improved to handle timeouts when instructing Prometheus to reload the configuration.|
|For Gerrit Dashboard: MB-56464|
|190593,2||MB-56464 [BP] Reload prometheus config in separate process||neo||ns_server||Status: MERGED||+2||+1|
|190666,1||Merge remote-tracking branch 'couchbase/neo'||master||ns_server||Status: MERGED||+2||+1|