PDM might never seqnoAck if persistence lags behind (1/3) [ETA 2019/7/10]

If

• we are regularly receiving snapshots
• persistence is lagging >1 snapshot behind
• we see a persist level prepare

We might never seqnoAck.

At the moment, if we see a Persist level Prepare, persistence must pass the most recent snapshotEnd before the HPS is allowed to advance. If a new snapshot arrives before persistence reaches the end of the previous snapshot, the snapshotEnd will be updated, and now persistence must exceed the new snapshotEnd to advance the HPS.

Proposed Solution:

Add a flag to the last SyncWrite of a snapshot to mark that when persistence passes the seqno of this SyncWrite, the HPS can be updated to this seqno and then seqnoAck'ed back to the active.

Track a queue of snapshot end seqnos and ack at that seqno when persistence passes it.