Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35700

[BP MB-34674] - Optimize collatejson JoinArray to mitigate high memory usage

    XMLWordPrintable

Details

    • Untriaged
    • No

    Description

      collatejson package's JoinArray function which is joins multiple encoded entries to form one encoded entry uses append function which increases buffer capacity more than what is required to append all individual entries. This causes a bloat of memory usage of the process, most of it is garbage which will eventually be GC'd but can cause momentary high memory spike leading to OOM.

      To repro: Create a document that is 1.5MB large. Create an array index with array size over 3000.  Resulting size of all entries should be in the tune of 3.5GB for single document. large_covering_field itself (below) should be around 1.5MB. 
      CREATE INDEX idx ON default(ALL array, large_covering_field)

      One possible optimization is to pre-calculate length of all entries to be indexed from a single document and allocate a buffer of that size, so that append does not need to increase the capacity of the buffer passed. 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              prathibha Prathibha Bisarahalli (Inactive)
              jeelan.poola Jeelan Poola
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty