N1QL search function not returning results while FTS and ES does.

Build => 7.6.0-1970

parameters={"OS": "debian", "version_number": "7.6.0-1970", "suite": "12hr_weekly", "component": "fts", "subcomponent": "None", "subcomponent_regex": "None", "serverPoolId": "regression", "addPoolId": "elastic-fts", "extraParameters": "get-cbcollect-info=True", "Test": false, "url": "", "branch": "master", "cherrypick": "", "retries": "0", "fresh_run": false, "rerun_condition": "ALL", "rerun_params": "", "executor_suffix": "", "executor_job_parameters": "", "check_vm": false, "initial_version": "", "serverType": "ON_PREM", "skip_install": false, "use_dockerized_dispatcher": false, "dispatcher_url": "http://qa.sc.couchbase.com/job/test_suite_dispatcher/"} 

2023-12-26 23:56:50 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 96 ----------------------------------------------------

2023-12-26 23:56:50 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "custom_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1990-09-08T07:48:00", "end": "2023-12-26T23:55:28.752984", "inclusive_start": false, "inclusive_end": false}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.217.88 : Administrator:

2023-12-26 23:56:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://172.23.217.88:8094/api/index/custom_index/query verb=POST params=b'{"indexName": "custom_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1990-09-08T07:48:00", "end": "2023-12-26T23:55:28.752984", "inclusive_start": false, "inclusive_end": false}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False

2023-12-26 23:56:50 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 1, 'failed': 0, 'successful': 1, 'errors': {}}

2023-12-26 23:56:50 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "join_date", "start": "1990-09-08T07:48:00", "end": "2023-12-26T23:55:28.752984", "inclusive_start": false, "inclusive_end": false} is 408 (took 1.811519ms)

2023-12-26 23:56:50 | INFO | MainProcess | Cluster_Thread | [es_base.search] ES query '{'query': {'filtered': {'filter': {'range': {'join_date': {'gt': '1990-09-08T07:48:00', 'lt': '2023-12-26T23:55:28.752984'}}}}}}' 

2023-12-26 23:56:50 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"gt": "1990-09-08T07:48:00", "lt": "2023-12-26T23:55:28.752984"}}}}}} on es_index is 408 (took 13ms)

2023-12-26 23:56:50 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified

2023-12-26 23:56:50 | INFO | MainProcess | Cluster_Thread | [task.execute] Running N1QL query: select meta().id from default where type='emp' and search(default, {"field": "join_date", "start": "1990-09-08T07:48:00", "end": "2023-12-26T23:55:28.752984", "inclusive_start": false, "inclusive_end": false},{"index": "custom_index"})

2023-12-26 23:56:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=select+meta%28%29.id+from+default+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22join_date%22%2C+%22start%22%3A+%221990-09-08T07%3A48%3A00%22%2C+%22end%22%3A+%222023-12-26T23%3A55%3A28.752984%22%2C+%22inclusive_start%22%3A+false%2C+%22inclusive_end%22%3A+false%7D%2C%7B%22index%22%3A+%22custom_index%22%7D%29

2023-12-26 23:56:51 | INFO | MainProcess | Cluster_Thread | [task.execute] N1QL hits for query: "select meta().id from default where type='emp' and search(default, {\"field\": \"join_date\", \"start\": \"1990-09-08T07:48:00\", \"end\": \"2023-12-26T23:55:28.752984\", \"inclusive_start\": false, \"inclusive_end\": false},{\"index\": \"custom_index\"})" is 0 (took 258.159001ms)

2023-12-26 23:56:51 | ERROR | MainProcess | Cluster_Thread | [task.execute] FAIL: FTS hits: 408, while N1QL hits: 0

2023-12-26 23:56:51 | ERROR | MainProcess | Cluster_Thread | [task.execute] FAIL: Following 408 doc(s) were not returned by N1QL,but FTS, printing 50: ['emp10000876', 'emp10000253', 'emp10000172', 'emp10000401', 'emp10000241', 'emp10000638', 'emp10000538', 'emp10000363', 'emp10000507', 'emp10000113', 'emp10000917', 'emp10000831', 'emp10000281', 'emp10000964', 'emp10000305', 'emp10000183', 'emp10000631', 'emp10000528', 'emp10000208', 'emp10000242', 'emp10000059', 'emp10000718', 'emp10000765', 'emp10000612', 'emp10000073', 'emp10000742', 'emp10000737', 'emp10000328', 'emp10000304', 'emp10000111', 'emp10000001', 'emp10000056', 'emp10000869', 'emp10000958', 'emp10000243', 'emp10000175', 'emp10000413', 'emp10000927', 'emp10000336', 'emp10000689', 'emp10000991', 'emp10000669', 'emp10000088', 'emp10000924', 'emp10000696', 'emp10000735', 'emp10000913', 'emp10000755', 'emp10000017', 'emp10000130']

2023-12-26 23:56:51 | INFO | MainProcess | Cluster_Thread | [task.execute] Running N1QL query: select meta().id,* from default where type='emp' and search(default, {"field": "join_date", "start": "1990-09-08T07:48:00", "end": "2023-12-26T23:55:28.752984", "inclusive_start": false, "inclusive_end": false},{"index": "custom_index"})

2023-12-26 23:56:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=select+meta%28%29.id%2C%2A+from+default+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22join_date%22%2C+%22start%22%3A+%221990-09-08T07%3A48%3A00%22%2C+%22end%22%3A+%222023-12-26T23%3A55%3A28.752984%22%2C+%22inclusive_start%22%3A+false%2C+%22inclusive_end%22%3A+false%7D%2C%7B%22index%22%3A+%22custom_index%22%7D%29

2023-12-26 23:56:51 | INFO | MainProcess | Cluster_Thread | [task.execute] N1QL hits for query: "select meta().id,* from default where type='emp' and search(default, {\"field\": \"join_date\", \"start\": \"1990-09-08T07:48:00\", \"end\": \"2023-12-26T23:55:28.752984\", \"inclusive_start\": false, \"inclusive_end\": false},{\"index\": \"custom_index\"})" is 0 (took 262.551209ms) 

36 out of 100 queries failed! - [4, 6, 8, 9, 14, 16, 17, 19, 24, 25, 27, 28, 38, 39, 43, 44, 46, 48, 49, 53, 54, 59, 60, 62, 65, 66, 71, 75, 76, 77, 78, 80, 89, 92, 93, 96]