Details
-
Bug
-
Resolution: Unresolved
-
Major
-
6.0.0
-
None
-
Untriaged
-
Unknown
Description
roaring bitmap library has a recently introduced API that let's you add an offset to each entry in a bitmap, which may be useful in a scorch file segment merging optimization, for the case when there are no (or not many) obsoleted docs (no updates, no deletions). This ticket is to research this optimization idea and implement it if it works.
https://github.com/RoaringBitmap/roaring/issues/190
https://github.com/RoaringBitmap/roaring/blob/master/roaring.go#L425
The handwave, when you have segments A, B, C, then you have bitmaps for each, where they have bit-ranges like...
A: [0...len(A))
|
B: [0...len(B))
|
C: [0...len(C))
|
And, when there are no obsoleted docs in those, you might want to logically Shift-&-Append those bitmaps, so they'll end up more like...
A': [0...len(A)] // No shifting
|
B': [len(A)+0 ... len(A)+len(B)) // Shifted over by len(A)
|
C': [len(A)+len(B)+0 ... len(A)+len(B)+len(C)) // Shifted over by len(A)+len(B)
|
Next, your new final "merged" bitmap is A' Or B' Or C'
The new AndOffset() API in roaring might allow us to have that optimized shifting capability.