[2] Wrong status code (500) for rejected queries

Test to reproduce:

• Create cluster with following config:
  • 
    D : D+F: D+F: D : F: F with fts RAM as 990mb

• Create 2  buckets, 1 with normal data and other with vector-siftsmall data
• Create 2 fts indexes
• Wait for indexing to complete
• Run 5 queries
• Update index partitions to 3
• Run query. Query fails with 500 query request rejected

• [2024-05-17 12:37:47,798] - [fts_vector_search:234] INFO - ******************** Running Query # 5 - on index i1 ********************

  [2024-05-17 12:37:47,798] - [fts_vector_search:240] INFO - Running n1ql Query - SELECT COUNT(*) FROM `b1`.s1.c1 AS t1 WHERE SEARCH(t1, {'query': {'match_none': {}}, 'explain': True, 'fields': ['*'], 'knn': [{'field': 'vector_data', 'k': 10, 'vector': [1.0, 3.0, 11.0, 110.0, 62.0, 22.0, 4.0, 0.0, 43.0, 21.0, 22.0, 18.0, 6.0, 28.0, 64.0, 9.0, 11.0, 1.0, 0.0, 0.0, 1.0, 40.0, 101.0, 21.0, 20.0, 2.0, 4.0, 2.0, 2.0, 9.0, 18.0, 35.0, 1.0, 1.0, 7.0, 25.0, 108.0, 116.0, 63.0, 2.0, 0.0, 0.0, 11.0, 74.0, 40.0, 101.0, 116.0, 3.0, 33.0, 1.0, 1.0, 11.0, 14.0, 18.0, 116.0, 116.0, 68.0, 12.0, 5.0, 4.0, 2.0, 2.0, 9.0, 102.0, 17.0, 3.0, 10.0, 18.0, 8.0, 15.0, 67.0, 63.0, 15.0, 0.0, 14.0, 116.0, 80.0, 0.0, 2.0, 22.0, 96.0, 37.0, 28.0, 88.0, 43.0, 1.0, 4.0, 18.0, 116.0, 51.0, 5.0, 11.0, 32.0, 14.0, 8.0, 23.0, 44.0, 17.0, 12.0, 9.0, 0.0, 0.0, 19.0, 37.0, 85.0, 18.0, 16.0, 104.0, 22.0, 6.0, 2.0, 26.0, 12.0, 58.0, 67.0, 82.0, 25.0, 12.0, 2.0, 2.0, 25.0, 18.0, 8.0, 2.0, 19.0, 42.0, 48.0, 11.0]}]});

  [2024-05-17 12:37:47,995] - [on_prem_rest_client:1133] ERROR - POST http://172.23.121.140:8093/query?statement=qeury%body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "b6cd4c2b-d20b-413a-9131-c0ec98da581c",\n"signature": {"$1":"number"},\n"results": [\n],\n"errors": [{"code":5000,"msg":"n1fty: response_handler: err - cause: search err summary: [partition: i1_222858ce458f7fae_64a02f35, err: query request rejected, partition: i1_222858ce458f7fae_8ebf8158, err: query request rejected, ]"}],\n"status": "fatal",\n"metrics": {"elapsedTime": "159.973256ms","executionTime": "159.829198ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password

Another instance:

In this case, the test just executed successfully for 13 queries, after which the queries were getting rejected.

• {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "5e8cfb43-e869-498f-8732-46b1dbc293d1",\n"signature": {"$1":"number"},\n"results": [\n],\n"errors": [{"code":5000,"msg":"n1fty: response_handler: stream.Recv, err - cause: rpc error: code = Unknown desc = query request rejected"}],\n"status": "fatal",\n"metrics": {"elapsedTime": "4.379039ms","executionTime": "4.263307ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password

  Logs
  https://cb-engineering.s3.amazonaws.com/logs/FTS/MB-61956/MB.zip