Minor Description
In investigating expiry opcode output, we identified scenarios that would output mutations over DCP when we might have expected expirations.
If an item with an expiry time is written to memory and logically expires after the item is persisted to disk, or does not get persisted at all (i.e. the flusher does not run on the item), when a DCP stream request is received this item will be sent as a mutation. However, if the item expires and is then persisted, the flusher expires the document and a DCP stream will either send a deletion (or an expiration, if that has been enabled). I have attached a rough flow diagram of what path triggers certain DCP outputs, but it is both not fully encompassing and is somewhat vague with memory/disk, so read with a pinch of salt.
As ephemeral does not persist to disk, all the logically expired items will appear as mutations if you DCP stream them out with performing any gets/expiry pager runs prior.
This may seem like abnormal behaviour, but as part of setting an item with an expiry time, it is up to the consumer to deal with expiration. The mutation still includes the correct expiration time so say the item was streamed to another node, a attempted get on that node would still result in a failure and deletion/expiration of that item.
It is important to note that if you call a get on the item, or the expiry pager runs when item is logically expired, this will cause the item to be deleted/expired as you would expect and the DCP output corresponds as such.
Although this is all correct in terms of performance, this could be explained better in documentation to avoid future confusion.
