Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58531

Allow OSO backfill to support more than one collection

    XMLWordPrintable

Details

    • 0

    Description

      Raised by Michael Blow, currently DCP clients have a challenge when trying to perform DCP backfills on multiple collections in an optimal manner - they must either:

      • (a) Create a single DCP stream with multiple collections - this inhibits out-of-seqno order (OSO) backfill so may be slower if OSO would otherwise be faster (collections are small vs total bucket size - see MB-56547).
      • (b) Create a DCP stream per collection - This permits OSO, but requires multiple streams to be created and hence multiple scans of the underlying KVStore, and hence actually may be slower than (a) if OSO is not used - i.e. we could end up scanning the entire vBucket in seqno order multiple times.

      it would be better for DCP clients if they could just do (a), but OSO would be utilised by KV-Engine (for multiple collections) if expected to be beneficial.

      From a brief conversation with Jim Walker we believe this should be possible:

      Dave: any fundamental reason why OSO couldn’t support multiple collections in a single filter - I’m thinking we could just perform multiple by-key scans ?

      Jim: I don’t recall a fundamental reason, some of the code might just have a single cid parameter in places, the scan ByKey API has a vector of key ranges so that layer certainly supports. Note that when we do an OSO for say cid 8, we scan for all keys in the 8 namespace, and also scan in the system event namespace (getting all system events for collection.
      This is all done by just adding two ranges to the ByKey scan create.
      I don’t recall any fundamental issue, just that;s the way we made the code fit into the 7.0 schedule.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jwalker Jim Walker
            drigby Dave Rigby (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:

              PagerDuty