Shutdown race causes SyncWrite prepare to get "lost" and out of order completions on replica

'2021-05-12T06:02:59.405617-07:00 ERROR 76: Exception occurred during packet execution. Closing connection: PassiveDurabilityMonitor::completeSyncWrite vb:149 Pending resolution for \'SW @0x7f451af11050 qi:[key:\'<ud>cid:0x0:test_docs-26008268</ud>\' seqno:14648 reqs:{majority, timeout=infinite}]  status:Pending\', but received unexpected commit for key <ud>cid:0x0:test_docs-6195935</ud> and prepare seqno: 14652, highSeqno:14937, HPS:14936, HCS:12441, foundKeyWithSeqno:\'SW @0x7f451af11080 qi:[key:\'<ud>cid:0x0:test_docs-6195935</ud>\' seqno:14652 reqs:{majority, timeout=infinite}]  status:Pending\' lastReceivedSnapEnd:\'none\'. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"127.0.0.1\\",\\"port\\":38744} - {\\"ip\\":\\"127.0.0.1\\",\\"port\\":11209} (System, <ud>@ns_server</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":34,"cas":0,"datatype":"raw","extlen":16,"key":"<ud>.test_docs-6195935</ud>","keylen":18,"magic":"ClientRequest","opaque":66,"opcode":"DCP_COMMIT","vbucket":149},"refcount":1}]\n', '2021-05-12T06:03:01.218538-07:00 ERROR 2081: Exception occurred during packet execution. Closing connection: PassiveDurabilityMonitor::completeSyncWrite vb:149 Pending resolution for \'SW @0x7f451af11050 qi:[key:\'<ud>cid:0x0:test_docs-26008268</ud>\' seqno:14648 reqs:{majority, timeout=infinite}]  status:Pending\', but received unexpected commit for key <ud>cid:0x0:test_docs-85393274</ud> and prepare seqno: 14653, highSeqno:14938, HPS:14936, HCS:12441, foundKeyWithSeqno:\'SW @0x7f451af110b0 qi:[key:\'<ud>cid:0x0:test_docs-85393274</ud>\' seqno:14653 reqs:{majority, timeout=infinite}]  status:Pending\' lastReceivedSnapEnd:\'none\'. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"127.0.0.1\\",\\"port\\":57185} - {\\"ip\\":\\"127.0.0.1\\",\\"port\\":11209} (System, <ud>@ns_server</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":35,"cas":0,"datatype":"raw","extlen":16,"key":"<ud>.test_docs-85393274</ud>","keylen":19,"magic":"ClientRequest","opaque":66,"opcode":"DCP_COMMIT","vbucket":149},"refcount":1}]\n', '2021-05-12T06:03:06.923068-07:00 ERROR 2079: Exception occurred during packet execution. Closing connection: PassiveDurabilityMonitor::completeSyncWrite vb:154 Pending resolution for \'SW @0x7f45a87bb690 qi:[key:\'<ud>cid:0x0:test_docs-85392583</ud>\' seqno:14546 reqs:{majority, timeout=infinite}]  status:Pending\', but received unexpected commit for key <ud>cid:0x0:test_docs-60637898</ud> and prepare seqno: 14554, highSeqno:15078, HPS:14855, HCS:12457, foundKeyWithSeqno:\'SW @0x7f452b502ac0 qi:[key:\'<ud>cid:0x0:test_docs-60637898</ud>\' seqno:14554 reqs:{majority, timeout=infinite}]  status:Pending\' lastReceivedSnapEnd:\'15011{Disk}\'. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"127.0.0.1\\",\\"port\\":55175} - {\\"ip\\":\\"127.0.0.1\\",\\"port\\":11209} (System, <ud>@ns_server</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":35,"cas":0,"datatype":"raw","extlen":16,"key":"<ud>.test_docs-60637898</ud>","keylen":19,"magic":"ClientRequest","opaque":71,"opcode":"DCP_COMMIT","vbucket":154},"refcount":1}]\n']

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.37008.ini bucket_storage=couchstore,rerun=false,GROUP=P0;long_graceful,randomize_value=true,doc_size=256,bucket_eviction_policy=fullEviction,replicas=1,nodes_init=4,upgrade_version=7.0.0-5153 -t magma.magma_crash_recovery.MagmaCrashTests.test_crash_during_ops,num_items=1000000,doc_size=256,sdk_timeout=60,graceful=True,doc_ops=create:expiry,replicas=1,GROUP=P0;long_graceful,durability=MAJORITY,multiplier=100'