[Transaction] Seeing keys missing in doc_loading during swap_rebalance

Build: 6.6.0-7883

Scenario:

1. 4 node cluster, 2 Couchbase bucket (replica=2)

   +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+

   | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |

   +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+

   | 172.23.105.205 | kv | 54.025974026 | 4201840640 | 3443564544 | 0 / 3758092288 | 5167 / 10324 | 6.6.0-7883-enterprise |

   | 172.23.105.155 | fts, index, kv, n1ql | 69.6335078534 | 4201840640 | 3388350464 | 0 / 3758092288 | 4938 / 9838 | 6.6.0-7883-enterprise |

   | 172.23.105.206 | kv | 57.1428571429 | 4201840640 | 3453943808 | 0 / 3758092288 | 4806 / 9584 | 6.6.0-7883-enterprise |

   | 172.23.105.159 | kv | 56.9553805774 | 4201840640 | 3429101568 | 0 / 3758092288 | 4801 / 9680 | 6.6.0-7883-enterprise |

   +----------------+----------------------+-----------------+------------+------------+----------------+-------------------+-----------------------+

   +----------+---------+----------+------------+-----+-------+------------+----------+-----------+

   | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |

   +----------+---------+----------+------------+-----+-------+------------+----------+-----------+

   | bucket-0 | membase | 2 | none | 0 | 9483 | 2986344448 | 93145992 | 439373036 |

   | bucket-1 | membase | 2 | none | 0 | 10229 | 2986344448 | 97605584 | 540662594 |

   +----------+---------+----------+------------+-----+-------+------------+----------+-----------+

1. Load few docs into the buckets using transaction
2. Perform swap_rebalance for 2 nodes (172.23.105.155, 172.23.105.206)
3. Start parallel doc_loading using transaction during rebalance phase and validate

Observation:

At the end of the rebalance during data validation seeing keys missing from the bucket.

Pcap files https://cb-jira.s3.us-east-2.amazonaws.com/logs/key_missing/pcaps.zip

Test case to run:

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i nodes.ini -t rebalance_new.swaprebalancetests.SwapRebalanceBasicTests.do_test,nodes_init=4,replicas=2,standard_buckets=2,num-swap=2,swap-orchestrator=True,num_items=10000,doc_size=512,transaction_timeout=150,durability=PERSIST_TO_MAJORITY,atomicity=True -p rerun=False,get-cbcollect-info=False -m rest'