Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32600

[FTS] - optimize scorch segment merging with roaring AddOffset() API

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • feature-backlog
    • 6.0.0
    • fts
    • None
    • Untriaged
    • Unknown

    Description

      roaring bitmap library has a recently introduced API that let's you add an offset to each entry in a bitmap, which may be useful in a scorch file segment merging optimization, for the case when there are no (or not many) obsoleted docs (no updates, no deletions). This ticket is to research this optimization idea and implement it if it works.

      https://github.com/RoaringBitmap/roaring/issues/190

      https://github.com/RoaringBitmap/roaring/blob/master/roaring.go#L425

      The handwave, when you have segments A, B, C, then you have bitmaps for each, where they have bit-ranges like...

        A: [0...len(A))
        B: [0...len(B))
        C: [0...len(C))
      

      And, when there are no obsoleted docs in those, you might want to logically Shift-&-Append those bitmaps, so they'll end up more like...

        A': [0...len(A)]                                // No shifting
        B': [len(A)+0 ... len(A)+len(B))                // Shifted over by len(A)
        C': [len(A)+len(B)+0 ... len(A)+len(B)+len(C))  // Shifted over by len(A)+len(B)
      

      Next, your new final "merged" bitmap is A' Or B' Or C'

      The new AndOffset() API in roaring might allow us to have that optimized shifting capability.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            marty.schoch Marty Schoch (Inactive)
            steve Steve Yen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty