Collections: vbucket failover and drop consistency

Logging this to ensure we have a solution for this.

This scenario involves a client switching to a promoted vbucket just after they observed a new manifest (which dropped a collection).

• Client:1 is connected to VB:1 and has received seqno 1 to 100.
• Manifest 6 triggers VB:1 to drop collection:x at seqno 101, client:1 receives drop(id=6, x).
• Failure occurs, VB:1 is fails before it has replicated the drop(id=6, x).

The following scenario could happen, we need to figure out if it can.

• VB:1 is promoted to active and has 1 to 100, collection:x is still open
• VB:1 receives mutations to collections:x 101 to 150.
• Client:1 tries to stream-request(start=101, manifest=6) but is denied "manifest-ahead", 6 is greater than 5.
• VB:1 receives manifest 6 and drops collection:x at seqno 151.
• Client:1 tries to stream-request(start=101, manifest=6) and is failed and told to actually rolled back, we know the 101 they recorded is on a different 'time-line', so rollback to at most 100 occurs.
• Client:1 can't undo the drop(id=6, x) and the design says they don't have to...
• Client:1 tries to stream-request(start=100, manifest=6) and is successful.
• Client:1 sees 101 to 150, the x mutations and is rightly confused, they assume x was dropped.

Overall the vbucket promotion step needs to prevent this and is being designed as a co-operative step between KV and ns_server, ideally in this situation before vbucket promotion occurs manifest:6 is pushed to VB:1, VB:1 does a drop(id=6, x) as the very first thing it does as an active VB, i.e. it can never accept more mutations to collection:x.