Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
7.1.0-1250
-
1
-
Yes
-
Magma May 2022
Description
At the very high load of 10TB, the total number of files are already quire large in number. We try to maintain the files count per level and per kvstore to 300. In order to do that we end up compacting the sstables into larger ones and during this process we get an opportunity to deduplicate the data which is essentially actual compaction process although the overall fragmentation is much below the configured value.
Here is a load of 5B create -> 5B upsert -> 5B create -> 5B upsert
As we can see in the image fragmentation never hit 50% throughout. there was a stage when fragmentation is at 20% which is much below 50% configured value. And as soon as we started upserting the data(last 5B items) the fragmentation started reducing which means aggressive compaction has started kicking in.
Attach the magma stats for better analysis of this test.
Attachments
For Gerrit Dashboard: MB-48414 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
161583,2 | MB-48414 magma_dump: Fix incorrect per level statistics | master | magma | Status: MERGED | +2 | +1 |
162445,5 | MB-48414 magma: Fix writeamp optimisation for L0 local compaction | master | magma | Status: MERGED | +2 | +1 |
162532,19 | MB-48414 magma: Introduce local compactions for non L0 levels | master | magma | Status: MERGED | +2 | +1 |
162550,5 | MB-48414 magma: Fix file count compaction stat | master | magma | Status: MERGED | +2 | +1 |
162592,10 | MB-48414 magma: Count data level file count compacts | master | magma | Status: MERGED | +2 | +1 |
162639,3 | MB-48414 magma: Fix compaction trigger reason | master | magma | Status: MERGED | +2 | +1 |
163064,5 | MB-48414 magma: Avoid L3->L4 file count compaction | master | magma | Status: ABANDONED | 0 | -1 |
163259,8 | MB-48414 magma: Remove file count limit on range extension | master | magma | Status: MERGED | +2 | +1 |
163462,9 | MB-48414 magma: Count internal fragmentation compactions | master | magma | Status: MERGED | +2 | +1 |