[Volume Test] : Data mismatch after eventing swap rebalance

Steps

1. Create 10M docs average of 1k docs.
2. Deploy 4 functions parallely with dcp_stream_boundary=”everything”. Fan out scenario.
3. Wait for function bootstrap to complete - Wait for it to catch up.
4. Verify Results that 10M docs gets transferred to all destination buckets using primary index.
5. Create 3M docs.
6. When 5 is in progress do a KV Rebalance in of 2 nodes.
7. Wait for rebalance to complete
8. Verify Results that 3M docs gets transferred to all destination buckets
9. Create 3M docs.
10. When 9 is in progress do a KV Rebalance out of 2 nodes.
11. Wait for rebalance to complete
12. Verify Results that 3M docs gets transferred to all destination buckets
13. Create 3M docs.
14. When 13 is in progress do a KV Swap Rebalance of 2 nodes.
15. Wait for rebalance to complete
16. Verify Results that 3M docs gets transferred to all destination buckets
17. Create 3M docs.
18. When 17 is in progress do a Eventing Rebalance in of 2 nodes.
19. Wait for eventing rebalance to complete
20. Verify Results that 3M docs gets transferred to all destination buckets
21. Create 3M docs.
22. When 21 is in progress do a Eventing Rebalance out of 2 nodes.
23. Wait for eventing rebalance to complete.
24. Verify Results that 3M docs gets transferred to all destination buckets
25. Create 3M docs.
26. When 25 is in progress do a Eventing Swap Rebalance of 2 nodes
27. Wait for eventing rebalance to complete.
28. Verify Results that 3M docs gets transferred to all destination buckets
29. Create 3M docs.
30. When 29 is in progress do a Eventing + KV Rebalance in of (1+1) nodes.
31. Wait for eventing rebalance to complete
32. Verify Results that 3M docs gets transferred to all destination buckets
33. Create 3M docs.
34. When 33 is in progress do a Rebalance out of Eventing + KV Rebalance in of (1+1) nodes.
35. Wait for eventing rebalance to complete.
36. Verify Results that 3M docs gets transferred to all destination buckets
37. Create 3M docs.
38. When 37 is in progress do a Swap Rebalance of Eventing + KV Rebalance in of (1+1) nodes.
39. Wait for eventing rebalance to complete.
40. Verify Results that 3M docs gets transferred to all destination buckets
41. Undeploy all the functions.
42. Delete all the functions.

Currently test fails at step no 28. I used the primary index for data validation and it has the same values as in

Stats

Stats.txt