Caught unhandled std::exception-derived exception. what(): Monotonic<l> (PassiveDM(vb:216)::State::highPreparedSeqno.lastWriteSeqno) invariant failed: new value (467003) breaks invariant on current value (467008)

Steps:

1. Create a 9 node cluster
2. Create required buckets and collections.
3. Create 1000000 items sequentially
4. Create 1000000 items sequentially
5. Update 1000000 RandonKey keys to create 50 percent fragmentation
6. Create 1000000 items sequentially
7. Update 1000000 RandonKey keys to create 50 percent fragmentation
8. Rebalance in with Loading of docs
9. Crash Magma/memc with Loading of docs
10. Rebalance Out with Loading of docs
11. Crash Magma/memc with Loading of docs
12. Rebalance In_Out with Loading of docs
13. Crash Magma/memc with Loading of docs
14. Swap with Loading of docs
15. Crash Magma/memc with Loading of docs
16. Failover a node and RebalanceOut that node with loading in parallel
17. Crash Magma/memc with Loading of docs
18. Failover a node and FullRecovery that node
19. Crash Magma/memc with Loading of docs
20. Failover a node and DeltaRecovery that node with loading in parallel
21. Crash Magma/memc with Loading of docs

    172.23.121.135
    2021-12-21 11:21:07,793 | infra | DEBUG | Thread-6636 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.121.135: rm -rf /opt/couchbase/var/lib/couchbase/crash/eabc363e-c137-4f74-c048a1ad-a5d50af7.core
    running: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/eabc363e-c137-4f74-c048a1ad-a5d50af7.dmp > /opt/couchbase/var/lib/couchbase/crash/eabc363e-c137-4f74-c048a1ad-a5d50af7.core
    2021-12-21 11:21:07,946 | infra | DEBUG | Thread-6636 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.121.135: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/eabc363e-c137-4f74-c048a1ad-a5d50af7.dmp > /opt/couchbase/var/lib/couchbase/crash/eabc363e-c137-4f74-c048a1ad-a5d50af7.core
    running: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/eabc363e-c137-4f74-c048a1ad-a5d50af7.core -ex "bt full" -ex quit
    2021-12-21 11:21:08,209 | infra | DEBUG | Thread-6636 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.121.135: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/eabc363e-c137-4f74-c048a1ad-a5d50af7.core -ex "bt full" -ex quit
    172.23.121.135: Stack Trace of first crash - eabc363e-c137-4f74-c048a1ad-a5d50af7.dmp
    Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
    #0 0x00007f79ecd34387 in raise () from /lib64/libc.so.6
    #0 0x00007f79ecd34387 in raise () from /lib64/libc.so.6
    No symbol table info available.
    #1 0x00007f79ecd35a78 in abort () from /lib64/libc.so.6
    No symbol table info available.
    #2 0x00007f79ed67f63c in __gnu_cxx::__verbose_terminate_handler() [clone .cold] () from /opt/couchbase/bin/../lib/libstdc++.so.6
    No symbol table info available.
    #3 0x0000000000b4cc5b in backtrace_terminate_handler() ()
    No symbol table info available.
    #4 0x00007f79ed68a8f6 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
    No symbol table info available.
    #5 0x00007f79ed68a961 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
    No symbol table info available.
    #6 0x00007f79ed68abf4 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
    No symbol table info available.
    #7 0x000000000053b34b in void cb::throwWithTrace<std::logic_error>(std::logic_error const&) ()
    No symbol table info available.
    #8 0x00000000008113bd in ThrowExceptionPolicy<long>::nonMonotonic ()
    No symbol table info available.
    #9 0x0000000000813717 in PassiveDurabilityMonitor::State::updateHighPreparedSeqno() ()
    No symbol table info available.
    #10 0x0000000000813ba3 in PassiveDurabilityMonitor::notifyLocalPersistence() ()
    No symbol table info available.
    #11 0x0000000000728b6d in VBucket::notifyPersistenceToDurabilityMonitor() ()
    No symbol table info available.
    #12 0x000000000082a6c8 in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) ()
    No symbol table info available.
    #13 0x000000000082b07f in EPBucket::flushVBucket(Vbid) ()
    No symbol table info available.
    #14 0x00000000006dc060 in Flusher::flushVB() ()
    No symbol table info available.
    #15 0x00000000006dc9b0 in Flusher::step(GlobalTask*) ()
    No symbol table info available.
    #16 0x0000000000abd3a2 in GlobalTask::execute() ()
    No symbol table info available.
    #17 0x0000000000aba5b5 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const ()
    No symbol table info available.
    #18 0x0000000000c0f010 in folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) ()
    No symbol table info available.
    #19 0x0000000000bf6dca in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
    No symbol table info available.
    #20 0x0000000000c11fc9 in void folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) ()
    No symbol table info available.
    #21 0x0000000000aba244 in void folly::detail::function::FunctionTraits<void ()>::callBig<CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) ()
    No symbol table info available.
    #22 0x00007f79ed6b3d40 in execute_native_thread_routine () from /opt/couchbase/bin/../lib/libstdc++.so.6
    No symbol table info available.
    #23 0x00007f79ef4bbea5 in start_thread () from /lib64/libpthread.so.0
    No symbol table info available.
    #24 0x00007f79ecdfc8dd in clone () from /lib64/libc.so.6
    No symbol table info available.

    {code:title=QE Test}

    guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False,iterations=2,sdk_timeout=60,log_level=debug,infra_log_level=debug,get-cbcollect-info=True,collect_pcaps=False,collect_data=True,skip_cleanup=True -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=9,graceful=True,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,key_size=18,assert_crashes_on_load=True,num_collections=20,maxttl=10,num_indexes=5,pc=10,index_nodes=5,query_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=50000,doc_ops=create:update:delete:read,durability=Majority,crashes=10 -m rest'

NOTE: Unfortunately i lost the product logs
I do have the thread dumps and stack traces attached in the file. Here is the job as well: http://qa.sc.couchbase.com/job/magma_temp_job4/1270/consoleFull