TLS:Mixed mode: N1fty:err: bleve: QueryBleve remote client returned status: 503 body: null

Upgrade from 6.6.4-9961 to 6.6.5 build 10072

• 4 node 6.6.4 cluster with n1: kv+n1ql, n2: kv+n1ql+index, n3: query+index+search, n4: n1ql+index+search
• Load travel-sample
• Create a fts index

  {

  "type": "fulltext-index",

  "name": "idx3",

  "uuid": "26bfecb1ee1975e5",

  "sourceType": "couchbase",

  "sourceName": "travel-sample",

  "sourceUUID": "338ddd9636cc20b2e7cd7ef6c0d2a5bc",

  "planParams": {

  "maxPartitionsPerPIndex": 171,

  "indexPartitions": 6,

  "numReplicas": 1

  },

  "params": {

  "doc_config": {

  "docid_prefix_delim": "",

  "docid_regexp": "",

  "mode": "type_field",

  "type_field": "type"

  },

  "mapping": {

  "analysis": {},

  "default_analyzer": "keyword",

  "default_datetime_parser": "dateTimeOptional",

  "default_field": "_all",

  "default_mapping": {

  "dynamic": false,

  "enabled": true,

  "properties": {

  "country": {

  "dynamic": false,

  "enabled": true,

  "fields": [

  {

  "docvalues": true,

  "include_in_all": true,

  "include_term_vectors": true,

  "index": true,

  "name": "country",

  "type": "text"

  }

  ]

  }

  }

  },

  "default_type": "_default",

  "docvalues_dynamic": true,

  "index_dynamic": true,

  "store_dynamic": false,

  "type_field": "_type"

  },

  "store": {

  "indexType": "scorch"

  }

  },

  "sourceParams": {}

  }

• Turn on node-node encryption and encryption level to control and make sure below query works

  select meta().id from `travel-sample` USE INDEX(USING FTS) where country = "France"

• Failover n3 and upgrade to 6.6.5 and add back
• Update the encryption level to strict on n3
• Now the above query from n1 returns below error:

  [

  {

  "code": 5000,

  "msg": "n1fty: response_handler: stream.Recv, err - cause: rpc error: code = DeadlineExceeded desc = grpc_server: Search searchInContext err: bleve: QueryBleve remote client returned status: 503 body: null",

  "query": "select meta().id from `travel-sample` USE INDEX(USING FTS) where country = \"France\""

  }

  ]

• If we run the same query from n3, sometimes we do get the right results but sometimes we get the above error.

Logs:

https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-01-07T201111-ns_1%40172.23.120.174.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-01-07T201111-ns_1%40172.23.96.141.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-01-07T201111-ns_1%40172.23.97.211.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-01-07T201111-ns_1%40172.23.97.212.zip