rollback may leave hashtable inconsistent with on-disk data

When ep-engine rolls back a vbucket to a particular sequence number (I'm going to call it rollback_seqno) it does the following:

• It iterates through database headers to find the last one whose high_seqno is before rollback_seqno. I'm going to call the b-tree rooted at this header "rollback tree". And the b-tree rooted in the last written header the "original tree"
• So now it iterates through all the documents in the original tree that where updated after high_seqno of the rollback tree.
• For each document, it looks up the value for this document in the rollback tree. And updates the hashtable in accordance.
• Once all of this is done, it appends rollback tree header to the vbucket file.

The problem with this is that some items might not have been persisted to disk yet when ep-engine reconciles the hash table with the on-disk state.

It's pretty easy to reproduce:

• start two node cluster
• upload some data
• stop persistance on both nodes
• upload more data with non-intersecting set of keys (also there's some logic that looks decides whether to entirely reset the vbucket based on number of changes, so I uploaded two times fewer items here)
• kill memcached on first node
• once memcached is restarted, replications will be recreated and replica will have to rollback
• you can start the persistance back on the second node once rollbacks are done; you can see that the second node has more replica items that there are active items on node 1
• failover node 1 and see that it's possible to access items that should have been rolled back