Details
-
Improvement
-
Resolution: Unresolved
-
Minor
-
None
Description
For MOI, specific workloads and deletion patterns can cause high jemalloc fragmentation. For example, with the below below repro, we can get the utilization in some size classes to 50%. With other doc size, we can get the utilization to as low as 10%.
1) Setup cluster with 1 node:kv+n1ql+index with MOI index storage.
2) Load 10M docs with cbworloadgen: $ ./cbworkloadgen -r 99999999 -n 127.0.0.1:9000 -s 56 -j -u Administrator -p asdasd -i 10000000
3) Create primary index and an index on just the field `body` and let them finish building.
4) Delete approximately 5M docs with by running: DELETE FROM default WHERE age >= 50;
We can now observe that memory_used_storage reduces, but memory_total_storage does not. If the experiment is repeated by replacing the statement in step (4) with DELETE FROM default LIMIT 5000000;, we see that there is no fragmentation and both the stats reduce.
Attached are jemalloc stats after repro where we can the low utilization. Also attached are jemalloc stats from an experiment where utilization was very low, at around 10%. Also attached is a plot of memory_used_storage and memory_total_storage after deleting about half the docs using both the above methods. Note that before issuing the 2nd statement to delete items, all items were reinserted.