Collections: OSO - opt-in method to be advanced to the high-seqno

Raising this issue to assist GSI who have a performance issue with DCP and a change to OSO.

GSI have structured their ingestion so that a bucket level maintenance stream provides data to the indexes. When a new collection is encountered a new collection only DCP stream is created to get the disk snapshot and then they 'join' into the bucket level stream.

Using OSO is good for overall reduction in I/O for each data node but currently means that GSI have no choice but to join to a lower seqno than is optimal.

E.g. consider two collections c1 and c2, the disk snapsshot is as follows.

• c1 occupies seqno 1 to 10
• c2 occupies seqno 11 to 100

OSO snapshot for c1 will skip all of the c2 keys and return seqnos 1 to 10, GSI has no choice but to then attempt to join to the bucket level stream at seqno 10.

KV internally knows that 100 is the high-seqno of the snapshot (and would use that for cursor registration for disk to memory streaming), but it would be advantageous for GSI to also know this, so they can just join up from at least 100.

KV can provide this value easily by including a seqno advance, this will be opt-in though and not part of a default OSO snapshot.

• Update the existing control so it can accept both true and true_with_seqno_advanced.

Thus in our example KV will transmit the following if the client has successfull requested DCP control enable_out_of_order_snapshots=true_with_seqno_advanced

[OSO start] [seqnos 1 to 10 in some order ] [seqno-advanced 100] [OSO end]

From KV perspective this affects 7.0+ but GSI only enable OSO in Neo - so only fix in Neo.

cc Deepkaran Salooja