Don't be placed in a pending state during backfilling phase

Summary

When an active vbucket is to be moved to another node, the steps are as follows:

1. Create vbucket in state pending - request originates from ns_server
2. Add NON-takeover stream for the vbucket - this is for the backfill phase
3. close stream
4. Add takeover stream for vbucket - this is for the takeoverphase
5. Move to pending state (Note: already in pending state) - originates from DCP stream
6. Move to active state - originates from the DCP stream
7. close stream
8. move to active state (Note: already in active state) - originates from ns_server

The issue is that a vbucket can be in a pending state for potentially a long period of time, whilst waiting for the backfill phase and persistence of the highest seqno to complete.

This is not ideal because when a vbucket is in a pending phase KV does not respond to any operations sent to it. Instead it adds the operation to a pending operations list, which are processed when the vbucket becomes active.

Details

My understanding is that ns_server made the change in 4.5 to set the vbucket state to pending prior to initiating the backfilling phase. The change was made based on discussions with Chiyoung where he had mentioned that items in replica vBuckets are more likely to be evicted.

In 5.5.0 we introduced a new eviction policy and while it is still true that we are more aggressive evicting from replica buckets than active, we don't believe this requires us to remain in a pending state whilst backfill and persistence and persistence is performed. Instead we would much prefer the vbucket to be in a replica state.