Private Links: Throughput drops to 0 and Timeouts seen on rebalance/upgrade

🧵 Story

QE FIT-SIT runs on CI have shown errors when running against the latest release (3.5.2) of the .NET SDK.

Using private links, upgrade tests (upgrading nodes in the cluster to superior server versions) and rebalance tests show that both Kv and Query throughput drops to 0 and both experience timeouts, similar to what we used to see with NCBC-3716. This time this is observed with both server 7.2 and 7.6.

Each recover after the situation is done with. Logs from one of these runs can be downloaded in the resource section. These drops and errors aren't seen on runs without using private links.

🛠️ Acceptance Criteria

Similar to the work done on NCBC-3716, a drop in throughput during rebalances/upgrades is expected but no errors/drop to 0 should be observed.

📚 Resources

• Rebalance 3 to 4 nodes w/ Private Links run (requires VPN)
  • Associated logs (download link - 212MB)
• Rebalance 3 to 5 nodes w/ Private Links run
  • Associated logs (download link - 248MB)
• GitHub Actions workflow for FIT-SIT runs