By prefixing keys for the id btree with the vbucket Id, incremental index updates, especially large ones such as those after/during a rebalance, get significantly faster with the added benefit of generating less file fragmentation. Also it speeds up cleanup and better overall performance (due to slower fragmentation, less compactions).
Example test, using lucky6.conf evperf dataset. Index was initially built with 0.5M documents and then 512 new vbuckets were added, which brought 1M new documents:
initial build: 3m55.637s (final file size 85Mb)
incremental update: 54m47.857s (final file size 11Gb)
initial build: 3m59.553s (final file size 89 Mb)
incremental update: 45m1.006s (final file size 8.7Gb)
Gains even higher, when used together with
MB-7131, example for a small test of 0.5M + 0.5M documents:
initial build: 3m58.688s ( final file size 85 Mb )
incremental update: 20m11.722s ( final file size 4967 Mb )
initial build: 3m57.042s ( final file size 89Mb )
incremental update: 16m0.273s ( final file size 4109Mb )
After + with
initial build: 3m59.265s ( final file size 89 Mb )
incremental update: 15m10.753s ( final file size 3819 Mb )
All the tests naturally had view compaction not enabled.
Gains more significant for larger datasets.