Details
-
Bug
-
Resolution: Fixed
-
Major
-
Morpheus, 7.0.3, 7.1.0
-
Untriaged
-
1
-
Unknown
Description
Since we serialize all calls to prometheus http server in one queue (in stats_reader or ns_server_stats) the high volume of stats GET requests might cause the queues of these processes to grow infinitely.
We need to protect stats API's from such DOS attack
repro:
$ tcpkali -em "GET /pools/default/buckets/beer-sample/stats HTTP/1.1\r\nHost: node0.localhost\r\nAuthorization: Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\r\n\r\n" -r 1 -c 50 node0.localhost:9000 -T 600s
|
Ramped up to 50 connections.
|
:clock12: Traffic 3.211↓, 0.051↑ Mbps (conns 0↓ 50↑ 0⇡; seen 50)
|
(n_0@node0.localhost)123> erlang:process_info(whereis('stats_reader-beer-sample'), message_queue_len).
|
{message_queue_len,151}
|