[CX] intermittent failure UpgradeFromAlice600FailoverITD

Seeing something perhaps related on one of our upgrade tests today. This test does the following:

starts four alice nodes (two kv, two cbas)
ingests all beers
fails over both cbas nodes, upgrades them both to 6.6 simultaneously
ensures all beers are (still / again) ingested
failover & upgrade both kv nodes, one at a time
ensures all beers are (still / again) ingested <<<---- FAILS
At the last step, the number of beers is only 2942 instead of the expected 5891, throughout the entire period:

Expected result to be [ {

  "$1" : 5891

} ] but last result acquired = [ {

  "$1" : 2942

} ]

see failoverUpgradeAll.zip console.log

It looks like the upgrade in step 5 failed due to the following:

  "completionMessage": "Rebalance exited with reason {pre_rebalance_janitor_run_failed,\"beer-sample\",\n                                 {error,wait_for_memcached_failed,\n                                     ['ns_1@kv1.couchbase.host']}}."

I checked the Analytics logs and we keep failing to get the failover logs due to the kv nodes issue. We probably need to perform additional checks on the test and file an issue on ns_server/kv if we keep encountering this.