Details
-
Improvement
-
Resolution: Fixed
-
Major
-
master
Description
Raised after a discussion with Deepkaran Salooja and Dave Finlay.
When there's a very high density of indexes/buckets on a single node, this causes a huge stat bloat within ns_server.
This is because it's storing all of the stats per entity (bucket or index) in an ETS table, which is all in-memory.
I've reproduced this by creating ~850 indexes on a node and leaving it running for a few days (cbcollect attached in comments, apologies that the cluster is a bit messy), which may sound extreme, but we have seen out in the wild.
This problem will only get worse in Cheshire Cat where the theme is high application and data density, so seeing this many (small) indexes will be common. Depending on what stats we collect and store for collections this feature may also have a similar effect.
Obviously the scalability of stats in general is debatable but I didn't want this MB to go out of scope, so this just focusses on the memory usage.
Another observation was that with such a high density of indexes that the CPU spent by ns_server just collecting these stats was ~2 cores (out of 8).