Details
-
Improvement
-
Resolution: Fixed
-
Major
-
None
-
Security Level: Public
-
None
Description
By prefixing keys for the id btree with the vbucket Id, incremental index updates, especially large ones such as those after/during a rebalance, get significantly faster with the added benefit of generating less file fragmentation. Also it speeds up cleanup and better overall performance (due to slower fragmentation, less compactions).
Example test, using lucky6.conf evperf dataset. Index was initially built with 0.5M documents and then 512 new vbuckets were added, which brought 1M new documents:
Before
initial build: 3m55.637s (final file size 85Mb)
incremental update: 54m47.857s (final file size 11Gb)
with patch
initial build: 3m59.553s (final file size 89 Mb)
incremental update: 45m1.006s (final file size 8.7Gb)
Gains even higher, when used together with MB-7131, example for a small test of 0.5M + 0.5M documents:
Before
initial build: 3m58.688s ( final file size 85 Mb )
incremental update: 20m11.722s ( final file size 4967 Mb )
After
initial build: 3m57.042s ( final file size 89Mb )
incremental update: 16m0.273s ( final file size 4109Mb )
After + with MB-7131
initial build: 3m59.265s ( final file size 89 Mb )
incremental update: 15m10.753s ( final file size 3819 Mb )
All the tests naturally had view compaction not enabled.
Gains more significant for larger datasets.