Flex Index: Rebalance out/upgrade/rebalance in - n1fty: initClient failed when query run during rebalance in

Build: 6.5.1-6296 --> 6.6.0 build 7883

Steps for upgrade:

1. Have 3 node cluster with 6.5.1-6296 with kv, n1ql,index and fts on all the nodes
2. Rebalance out one of the node
3. updgrade the rebalanced out node to 6.6.0-7883
4. Rebalance in the upgraded node with same services
5. Repeat the same for all the nodes

In 4th step, run Flex query like below:

SELECT META().id, address.applicationId FROM `default` USE INDEX (USING FTS, USING GSI) WHERE address.applicationId = 1 AND address.deviceTypeId=9 AND address.deviceStatus=0 ORDER BY address.applicationId

we get error like below:

[

  {

    "code": 5000,

    "msg": "n1fty: initClient failed - cause: client: cbauth err: Unable to find given hostport in cbauth database: `172.23.121.65:9130'",

    "query": "SELECT META().id FROM `default`  USE INDEX (USING FTS, USING GSI) WHERE address.ecpdId=\"1\" ORDER BY META().id"

  }

]

Logs:

https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance-upgrade/collectinfo-2020-07-17T000940-ns_1%40172.23.121.65.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance-upgrade/collectinfo-2020-07-17T000940-ns_1%40172.23.121.66.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance-upgrade/collectinfo-2020-07-17T000940-ns_1%40172.23.121.67.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance-upgrade/collectinfo-2020-07-17T000940-ns_1%40172.23.121.68.zip