Interruption in failover and then retrying failover causes future delta recovery to fail

When failover is interrupted it is possible that a subsequent delta node recovery will not succeed (as the appropriate metadata that allows delta node recovery to proceed may not be written.) This is part of the design of delta node recovery - it is not guaranteed to work in all cases. Failover has to complete normally. In the event that delta node recovery is not possible, full recovery should be used.

Nonetheless, there are a number of improvements that would reduce the probability of delta recovery not being possible or would improve the user experience around it. It's worth considering these. Examples are:

1. allow users in the UI to delta recover the buckets that are delta recoverable and full recover the others
2. change when we write the delta recovery metadata to reduce the window where the delta recovery metadata is lost (note this can't be eliminated completely)
3. remove the need for delta recovery metadata and at the time of delta recovery ask nodes being delta recovered which vbuckets it's in possession of to determine if delta recovery is possible
4. and there maybe more