HCS invariant broken after promotion to active

Separated out from MB-40286.

cbcollect and pcaps:- https://uploads.couchbase.com/MB-40286/test_4.zip or s3://cb-customers-secure/mb-40286/2020-07-13/test_4.zip
coredump :- 3031358a-8ed0-42e2-9337f0a0-8372c34b.dmp @ 172.23.121.10

Logs, pcaps, and core dumps indicate a node rejected deletions (due to a magma issue MB-40366) which should have removed prepares from the PDM. The vb was later promoted to active, and attempted to recommit the lingering prepares.

Underlying issue expected to be resolved by MB-40366, but additional code changes may be made under this MB to avoid similar situations in future.

Relevant comments from MB-40286 cherry-picked below.