Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36406

Implement defragmenter for high jemalloc fragmentation

    XMLWordPrintable

Details

    Description

      For MOI, specific workloads and deletion patterns can cause high jemalloc fragmentation. For example, with the below below repro, we can get the utilization in some size classes to 50%. With other doc size, we can get the utilization to as low as 10%.

       

      1) Setup cluster with 1 node:kv+n1ql+index with MOI index storage.

      2) Load 10M docs with cbworloadgen: $ ./cbworkloadgen -r 99999999 -n 127.0.0.1:9000 -s 56 -j -u Administrator -p asdasd -i 10000000

      3) Create primary index and an index on just the field `body` and let them finish building.

      4) Delete approximately 5M docs with by running: DELETE FROM default WHERE age >= 50;

       

      We can now observe that memory_used_storage reduces, but memory_total_storage does not. If the experiment is repeated by replacing the statement in step (4) with DELETE FROM default LIMIT 5000000;, we see that there is no fragmentation and both the stats reduce.

       

      Attached are jemalloc stats after repro where we can the low utilization. Also attached are jemalloc stats from an experiment where utilization was very low, at around 10%. Also attached is a plot of memory_used_storage and memory_total_storage after deleting about half the docs using both the above methods. Note that before issuing the 2nd statement to delete items, all items were reinserted.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jeelan.poola Jeelan Poola
            akhil.mundroy Akhil Mundroy
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty