Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34580

Historical stats consume large amounts of memory with high app density

    XMLWordPrintable

Details

    Description

      Raised after a discussion with Deepkaran Salooja and Dave Finlay.

      When there's a very high density of indexes/buckets on a single node, this causes a huge stat bloat within ns_server.
      This is because it's storing all of the stats per entity (bucket or index) in an ETS table, which is all in-memory.

      I've reproduced this by creating ~850 indexes on a node and leaving it running for a few days (cbcollect attached in comments, apologies that the cluster is a bit messy), which may sound extreme, but we have seen out in the wild.
      This problem will only get worse in Cheshire Cat where the theme is high application and data density, so seeing this many (small) indexes will be common. Depending on what stats we collect and store for collections this feature may also have a similar effect.

      Obviously the scalability of stats in general is debatable but I didn't want this MB to go out of scope, so this just focusses on the memory usage.
      Another observation was that with such a high density of indexes that the CPU spent by ns_server just collecting these stats was ~2 cores (out of 8).

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sumedh.basarkod Sumedh Basarkod (Inactive)
            matt.carabine Matt Carabine (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty