[Magma-CDC] Magma has multiple live prepares and completes for the same key in the seqIndex

• 5 node cluster
• Create one magma bucket with 10 scopes and 25 collections under each scope
• Load initial data of 10K items/collections (Total 2,500,000 items in the bucket)
• Now enable history on 90% of the collections
• Start graceful failover task on "172.23.107.102" and failover completes successfully
• Start dedupe data load (continuous updates of 200 docs for 1000 iterations) on the following collections. (Cluster is in steady state with one node in failed over state)
  Note: All load are with sync_write=MAJORITY
  
  

  0x8:0x15:name:                  collection-226

  0x9:0x25:name:                  collection-210

  0x9:0x38:name:                  collection-205

  
  * Ran magma dump and observed there is nothing on level 0, 1 2 and 3 of seq tree but there are multiple copies of same key at level 4 (946 copies (including prepares and default)

• Key tree is only showing single copy

Below is the link for Complete output of tree data , --doc by seq and doc-by-key

Magma Dump
https://cb-engineering.s3.amazonaws.com/magma_dump_outputs.tar.gz

QE-TEST:

TAF test:

bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,doc_size=512,data_load_stage=during,bucket_history_retention_seconds=300,bucket_spec=magma_dgm.20_percent_dgm.5_node_2_replica_magma_512,bucket_history_retention_bytes=2000000000,nodes_failover=1,durability=MAJORITY,recovery_type=delta,skip_validations=False,nodes_init=5,default_history_retention_for_collections=false,disk_optimized_thread_settings=True,autoCompactionDefined=true,randomize_value=True,override_spec_params=durability,disk_optimized_thread_settings=True,get-cbcollect-info=False,autoCompactionDefined=true,log_level=debug,infra_log_level=debug