[Cursor Dropping]: Potential data loss due to cursor dropping

In cursor dropping, we switch the stream to backfill state from in-memory state and drop the replication cursor for that stream on the checkpoint.

We re-register the replication cursor only during backfill when we mark the disk snapshot (http://src.couchbase.org/source/xref/trunk/ep-engine/src/dcp/stream.cc#350). However by this point the persistence cursors on checkpoints could have moved ahead and one or more checkpoints might be dropped. This would create a hole when we switch over to in-memory streaming from backfilling.

Example:

1. Say a stream is in in-memory state and has sent items upto seqno 100.

2. Now we switch to backfill state at seqno 100 and the replication cursor on the checkpoint is dropped. The disk then gives us a snapshot of 100 to 150.

3. We then re-register the replication cursor with checkpoint with start seqno of 150.

However if between 2 and 3 more items (say upto seqno 200) from checkpoint(s) can be persisted and checkpoint might only have items from 200. We would miss items from 151 to 200 in this case.

We must hence check the result of http://src.couchbase.org/source/xref/trunk/ep-engine/src/dcp/stream.cc#350 to see if we need more backfills.

Further,

Q: Why this doesn't happen during regular backfill ?

A: Because in regular backfill we register the replication cursor on the checkpoint before the backfill starts. We re-register the already registered one during step 3 above.

   But in cursor dropping at step 3 we re-register a dropped cursor. 

Maybe this should be solved after we have resolved MB-22451.