[Upgrade] - ERROR 211: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

Script to Repro
1. Run the following 6.6.2 longevity test for 3-4 days. We will have 27 node cluster at the end of it.

./sequoia -client 172.23.96.162:2375 -provider file:centos_third_cluster.yml -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml -scale 3 -repeat 0 -log_level 0 -version 6.6.2-9588 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true

2. Run the script [^create_drop.sh] on 6.6.2 nodes on the cluster. And this was run on the 7.0.0 nodes as well that will be brought into the cluster using swap rebalance for upgrade
3. Swap rebalance 6(1 of each service) 6.6.2 nodes with 7.0.0 nodes.
4. Graceful failover 6 node (1 of each service), upgrade, do a recovery and start rebalance.
5. Graceful failover 6 node (1 of each service), upgrade, do a recovery and start rebalance.

However because of MB-46778 and MB-46787 we could not continue with online upgrade. So at this point I upgraded remaining nodes in the cluster using offline method.

Saw the following error on
172.23.104.214

[root@localhost logs]# grep ERROR memcached.log.0002* | grep -v XERROR

memcached.log.000254.txt:2021-06-08T21:44:28.285234-07:00 ERROR 211: Exception occurred during packet execution. Closing connection: Connection::isPacketAvailable(): Invalid packet header detected. Cookies: []

[root@localhost logs]# 

All the previous upgrades on sys test cluster was done using online technique unless we ran into some blockers. But I don't think we had seen this error before.

cbcollect_info attached.