FTS: handle out-of-order DCP seq-num's and have a "building" mode

Typing these two ideas into a ticket before I forget, as DF mentioned these recently over lunch...

A) FTS should ideally be able to handle so-called "out-of-order" DCP seq-num's during DCP ingest.

Especially, an interesting case is with the very first (potentially huge) DCP snapshot that's sent over the wire during the "backfill" phase. In this, the KV server can purportedly have a substantial optimization if it doesn't need to send DCP mutations in monotonically increasing seq-num ordering. (Part of the work here would be figuring out what "purportedly" and the actual details mean here.)

Aside: There may or may not be a new DCP handshake flag (in cheshire cat?) where DCP consumers may signal that they're okay with out-of-order DCP mutations.

FTS, currently, is somewhat loosey and goosey w.r.t. DCP snapshot boundaries, where it incorporates incoming mutations whenever it sees enough mutations (bleve batch size is reached). And, I believe it does not really care whether the mutations that are received within a snapshot are "in-order" w.r.t. seq-num's – out-of-order is "okay" w.r.t. FTS.

B) For FTS using in the context of N1QL-n1fty-flex... where we want FTS to behave more like yet another GSI index... FTS might need to provide a "building" state or signal, similar to GSI's "building" vs "ready-to-use" states.

In mad-hatter and previous, FTS doesn't have any complex lifecycle states. Put another way: the claim is that FTS has just one state – "we're always building", or perhaps more bluntly... "we're always in a state of always behind"

One approach might be to implement an actual lifecycle state machine for FTS.

Perhaps another approach that seems easier would be more brute force... let's look at or consider how many DCP snapshots an FTS index has completely ingested? If it's < 1 (e.g., still ingesting the very first DCP snapshot...which ought to be the backfill snapshot?), then we might claim that FTS is in building mode or "state"?

And, then, as long as any of the pindexes is building, then using weakest-link-in-the-chain thinking, consider the index as building too?

As soon as all pindexes have completely ingested their very first DCP snapshot, they're the FTS can claim that it's now in "ready to use" mode?

Finally, this whole braindump might be worth breaking up into separate tickets.