Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49779

Collections: OSO - opt-in method to be advanced to the high-seqno

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 7.0.0, 7.0.0-Beta1, 7.1.0, 7.0.2, 7.0.1
    • 7.1.0
    • couchbase-bucket
    • None
    • Untriaged
    • 1
    • Unknown
    • KV 2021-Nov, KV 2021-Dec

    Description

      Raising this issue to assist GSI who have a performance issue with DCP and a change to OSO.

      GSI have structured their ingestion so that a bucket level maintenance stream provides data to the indexes. When a new collection is encountered a new collection only DCP stream is created to get the disk snapshot and then they 'join' into the bucket level stream.

      Using OSO is good for overall reduction in I/O for each data node but currently means that GSI have no choice but to join to a lower seqno than is optimal.

      E.g. consider two collections c1 and c2, the disk snapsshot is as follows.

      • c1 occupies seqno 1 to 10
      • c2 occupies seqno 11 to 100

      OSO snapshot for c1 will skip all of the c2 keys and return seqnos 1 to 10, GSI has no choice but to then attempt to join to the bucket level stream at seqno 10.

      KV internally knows that 100 is the high-seqno of the snapshot (and would use that for cursor registration for disk to memory streaming), but it would be advantageous for GSI to also know this, so they can just join up from at least 100.

      KV can provide this value easily by including a seqno advance, this will be opt-in though and not part of a default OSO snapshot.

      • Update the existing control so it can accept both true and true_with_seq_adv.

      Thus in our example KV will transmit the following if the client has successfull requested DCP control enable_out_of_order_snapshots=true_with_seq_adv

      [OSO start] [seqnos 1 to 10 in some order ] [seqno-advanced 100] [OSO end]

      From KV perspective this affects 7.0+ but GSI only enable OSO in Neo - so only fix in Neo.

      cc Deepkaran Salooja

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.1.0-1819 contains kv_engine commit 4c52a5b with commit message:
            MB-49779: Optionally end OSO snapshot with SeqnoAdvance

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1819 contains kv_engine commit 4c52a5b with commit message: MB-49779 : Optionally end OSO snapshot with SeqnoAdvance

            People

              jwalker Jim Walker
              jwalker Jim Walker
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty