Description
7.2.4-7067 |
Test to reproduce:
./testrunner -i /data/workspace/debian-p0-fts-vset00-00-custom-map-n1ql-rqg-scorch-collections-3_7.0_P1/testexec.35747.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,cluster=D+F,index_type=scorch,fts_quota=1000,run_via_n1ql=True,custom_map_add_non_indexed_fields=False,EXCLUDE_GROUP=SKIP_FOR_N1QL,text_analyzer=keyword,java_sdk_client=True,GROUP=COLLECTIONS,container_type=collection,sirius_url=http://172.23.120.103:4000 -t fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping,items=1000,custom_map=True,num_custom_analyzers=1,compare_es=True,cm_id=4,num_queries=100,GROUP=BUCKETS;P0;SKIP_FOR_N1QL;N1QL_MATCH_PHRASE;COLLECTIONS |
Index Definition:
{"type": "fulltext-index", "name": "custom_index", "uuid": "", "params": {"mapping": {"default_mapping": {"enabled": false, "dynamic": true, "default_analyzer": ""}, "type_field": "type", "default_type": "_default", "default_analyzer": "standard", "default_datetime_parser": "dateTimeOptional", "default_field": "_all", "analysis": {"analyzers": {"customAnalyzer1": {"char_filters": ["mapping"], "token_filters": ["apostrophe"], "tokenizer": "whitespace", "type": "custom"}}, "token_filters": {"back_edge_ngram": {"back": true, "max": 5, "min": 3, "type": "edge_ngram"}, "dict_compound_en": {"dict_token_map": "stop_en", "type": "dict_compound"}, "dict_compound_fr": {"dict_token_map": "articles_fr", "type": "dict_compound"}, "front_edge_ngram": {"back": false, "max": 5, "min": 3, "type": "edge_ngram"}, "keyword_marker": {"keywords_token_map": "stopwords", "type": "keyword_marker"}, "stopwords": {"stop_token_map": "stopwords", "type": "stop_tokens"}, "length": {"max": 5, "min": 3, "type": "length"}, "ngram": {"max": 5, "min": 3, "type": "ngram"}, "shingle": {"filler": "", "max": 5, "min": 2, "output_original": "false", "separator": "", "type": "shingle"}, "truncate": {"length": 10, "type": "truncate_token"}}, "token_maps": {"stopwords": {"tokens": ["i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "would", "should", "could", "ought", "i'm", "you're", "he's", "she's", "it's", "we're", "they're", "i've", "you've", "we've", "they've", "i'd", "you'd", "he'd", "she'd", "we'd", "they'd", "i'll", "you'll", "he'll", "she'll", "we'll", "they'll", "isn't", "aren't", "wasn't", "weren't", "hasn't", "haven't", "hadn't", "doesn't", "don't", "didn't", "won't", "wouldn't", "shan't", "shouldn't", "can't", "cannot", "couldn't", "mustn't", "let's", "that's", "who's", "what's", "here's", "there's", "when's", "where's", "why's", "how's", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very"], "type": "custom"}}, "char_filters": {"mapping": {"regexp": "[f]", "replace": "ph", "type": "regexp"}}, "tokenizers": {"alphanumeric": {"regexp": "[0-9a-zA-Z_]*", "type": "regexp"}}}, "types": {"scope1.collection1.emp": {"dynamic": false, "enabled": true, "fields": [], "properties": {"manages": {"dynamic": false, "enabled": true, "fields": [], "properties": {"team_size": {"dynamic": false, "enabled": true, "properties": {}, "fields": [{"include_in_all": true, "include_term_vectors": true, "index": true, "name": "team_size", "store": false, "type": "number", "analyzer": ""}]}}}, "join_date": {"dynamic": false, "enabled": true, "properties": {}, "fields": [{"include_in_all": true, "include_term_vectors": true, "index": true, "name": "join_date", "store": false, "type": "datetime", "analyzer": ""}]}, "email": {"dynamic": false, "enabled": true, "properties": {}, "fields": [{"include_in_all": true, "include_term_vectors": true, "index": true, "name": "email", "store": false, "type": "text", "analyzer": "customAnalyzer1"}]}}}}}, "store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "scorch"}, "doc_config": {"mode": "scope.collection.type_field", "type_field": "type"}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}} |
2024-01-03 04:36:00 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=http://172.23.121.197:8094/api/index/custom_index verb=PUT params={"type": "fulltext-index", "name": "custom_index", "uuid": "", "params": {"mapping": {"default_mapping": {"enabled": false, "dynamic": true, "default_analyzer": ""}, "type_field": "type", "default_type": "_default", "default_analyzer": "standard", "default_datetime_parser": "dateTimeOptional", "default_field": "_all", "analysis": {"analyzers": {"customAnalyzer1": {"char_filters": ["mapping"], "token_filters": ["apostrophe"], "tokenizer": "whitespace", "type": "custom"}}, "token_filters": {"back_edge_ngram": {"back": true, "max": 5, "min": 3, "type": "edge_ngram"}, "dict_compound_en": {"dict_token_map": "stop_en", "type": "dict_compound"}, "dict_compound_fr": {"dict_token_map": "articles_fr", "type": "dict_compound"}, "front_edge_ngram": {"back": false, "max": 5, "min": 3, "type": "edge_ngram"}, "keyword_marker": {"keywords_token_map": "stopwords", "type": "keyword_marker"}, "stopwords": {"stop_token_map": "stopwords", "type": "stop_tokens"}, "length": {"max": 5, "min": 3, "type": "length"}, "ngram": {"max": 5, "min": 3, "type": "ngram"}, "shingle": {"filler": "", "max": 5, "min": 2, "output_original": "false", "separator": "", "type": "shingle"}, "truncate": {"length": 10, "type": "truncate_token"}}, "token_maps": {"stopwords": {"tokens": ["i", "me", "my", "myself", "we", "our", "ours", "ourselves", "you", "your", "yours", "yourself", "yourselves", "he", "him", "his", "himself", "she", "her", "hers", "herself", "it", "its", "itself", "they", "them", "their", "theirs", "themselves", "what", "which", "who", "whom", "this", "that", "these", "those", "am", "is", "are", "was", "were", "be", "been", "being", "have", "has", "had", "having", "do", "does", "did", "doing", "would", "should", "could", "ought", "i'm", "you're", "he's", "she's", "it's", "we're", "they're", "i've", "you've", "we've", "they've", "i'd", "you'd", "he'd", "she'd", "we'd", "they'd", "i'll", "you'll", "he'll", "she'll", "we'll", "they'll", "isn't", "aren't", "wasn't", "weren't", "hasn't", "haven't", "hadn't", "doesn't", "don't", "didn't", "won't", "wouldn't", "shan't", "shouldn't", "can't", "cannot", "couldn't", "mustn't", "let's", "that's", "who's", "what's", "here's", "there's", "when's", "where's", "why's", "how's", "a", "an", "the", "and", "but", "if", "or", "because", "as", "until", "while", "of", "at", "by", "for", "with", "about", "against", "between", "into", "through", "during", "before", "after", "above", "below", "to", "from", "up", "down", "in", "out", "on", "off", "over", "under", "again", "further", "then", "once", "here", "there", "when", "where", "why", "how", "all", "any", "both", "each", "few", "more", "most", "other", "some", "such", "no", "nor", "not", "only", "own", "same", "so", "than", "too", "very"], "type": "custom"}}, "char_filters": {"mapping": {"regexp": "[f]", "replace": "ph", "type": "regexp"}}, "tokenizers": {"alphanumeric": {"regexp": "[0-9a-zA-Z_]*", "type": "regexp"}}}, "types": {"scope1.collection1.emp": {"dynamic": false, "enabled": true, "fields": [], "properties": {"manages": {"dynamic": false, "enabled": true, "fields": [], "properties": {"team_size": {"dynamic": false, "enabled": true, "properties": {}, "fields": [{"include_in_all": true, "include_term_vectors": true, "index": true, "name": "team_size", "store": false, "type": "number", "analyzer": ""}]}}}, "join_date": {"dynamic": false, "enabled": true, "properties": {}, "fields": [{"include_in_all": true, "include_term_vectors": true, "index": true, "name": "join_date", "store": false, "type": "datetime", "analyzer": ""}]}, "email": {"dynamic": false, "enabled": true, "properties": {}, "fields": [{"include_in_all": true, "include_term_vectors": true, "index": true, "name": "email", "store": false, "type": "text", "analyzer": "customAnalyzer1"}]}}}}}, "store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "scorch"}, "doc_config": {"mode": "scope.collection.type_field", "type_field": "type"}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}} |
Some n1ql queries are failing with
No index available on keyspace `default`:`default`.`scope1`.`collection1` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`default`.`scope1`.`collection1` to create a primary index, or check that your expected index is online. |
Example query:
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 64 ---------------------------------------------------- |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "custom_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "email", "wildcard": "*"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.216.216 : Administrator: |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [rest_client.urllib_request] Making a rest request api=http://172.23.216.216:8094/api/index/custom_index/query verb=POST params=b'{"indexName": "custom_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "email", "wildcard": "*"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 1, 'failed': 0, 'successful': 1} |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "email", "wildcard": "*"} is 1000 (took 3.471185ms) |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [es_base.search] ES query '{'query': {'wildcard': {'email': '*'}}}' |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"query": {"wildcard": {"email": "*"}}} on es_index is 1000 (took 52ms) |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] Running N1QL query: select meta().id from default:default.scope1.collection1 where type='emp' and search(default, {"field": "email", "wildcard": "*"}) |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=select+meta%28%29.id+from+default%3Adefault.scope1.collection1+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22email%22%2C+%22wildcard%22%3A+%22%2A%22%7D%29 |
2024-01-03 02:53:30 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.109.155:8093/query?statement=select+meta%28%29.id+from+default%3Adefault.scope1.collection1+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22email%22%2C+%22wildcard%22%3A+%22%2A%22%7D%29 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'{\n"requestID": "5095300a-9680-4ab2-a589-e897701fec80",\n"errors": [{"code":4000,"msg":"No index available on keyspace `default`:`default`.`scope1`.`collection1` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`default`.`scope1`.`collection1` to create a primary index, or check that your expected index is online."}],\n"status": "fatal",\n"metrics": {"elapsedTime": "3.371217ms","executionTime": "3.254224ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] N1QL query execution is failed. |
2024-01-03 02:53:30 | ERROR | MainProcess | Cluster_Thread | [task.execute] No index available on keyspace `default`:`default`.`scope1`.`collection1` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`default`.`scope1`.`collection1` to create a primary index, or check that your expected index is online. |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] Running N1QL query: select meta().id from default:default.scope1.collection1 where type='emp' and search(default, {"field": "email", "wildcard": "*"},{"index": "custom_index"}) |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=select+meta%28%29.id+from+default%3Adefault.scope1.collection1+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22email%22%2C+%22wildcard%22%3A+%22%2A%22%7D%2C%7B%22index%22%3A+%22custom_index%22%7D%29 |
2024-01-03 02:53:30 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.109.155:8093/query?statement=select+meta%28%29.id+from+default%3Adefault.scope1.collection1+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22email%22%2C+%22wildcard%22%3A+%22%2A%22%7D%2C%7B%22index%22%3A+%22custom_index%22%7D%29 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'{\n"requestID": "c9d11e14-4311-4438-b6bd-8f1257a322f0",\n"errors": [{"code":4000,"msg":"No index available on keyspace `default`:`default`.`scope1`.`collection1` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`default`.`scope1`.`collection1` to create a primary index, or check that your expected index is online."}],\n"status": "fatal",\n"metrics": {"elapsedTime": "1.794002ms","executionTime": "1.699069ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] N1QL query execution is failed. |
2024-01-03 02:53:30 | ERROR | MainProcess | Cluster_Thread | [task.execute] No index available on keyspace `default`:`default`.`scope1`.`collection1` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`default`.`scope1`.`collection1` to create a primary index, or check that your expected index is online. |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] Running N1QL query: select meta().id,* from default:default.scope1.collection1 where type='emp' and search(default, {"field": "email", "wildcard": "*"},{"index": "custom_index"}) |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=select+meta%28%29.id%2C%2A+from+default%3Adefault.scope1.collection1+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22email%22%2C+%22wildcard%22%3A+%22%2A%22%7D%2C%7B%22index%22%3A+%22custom_index%22%7D%29 |
2024-01-03 02:53:30 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.109.155:8093/query?statement=select+meta%28%29.id%2C%2A+from+default%3Adefault.scope1.collection1+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22email%22%2C+%22wildcard%22%3A+%22%2A%22%7D%2C%7B%22index%22%3A+%22custom_index%22%7D%29 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'{\n"requestID": "5de0dcdf-7992-4b4c-a28e-f26cfc078287",\n"errors": [{"code":4000,"msg":"No index available on keyspace `default`:`default`.`scope1`.`collection1` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`default`.`scope1`.`collection1` to create a primary index, or check that your expected index is online."}],\n"status": "fatal",\n"metrics": {"elapsedTime": "3.013404ms","executionTime": "2.819137ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] N1QL query execution is failed. |
2024-01-03 02:53:30 | ERROR | MainProcess | Cluster_Thread | [task.execute] No index available on keyspace `default`:`default`.`scope1`.`collection1` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`default`.`scope1`.`collection1` to create a primary index, or check that your expected index is online. |
While latter queries pass (not always, this is completely random)
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 67 ---------------------------------------------------- |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "custom_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "2003-03-16T19:45:00", "end": "2024-01-03T02:52:51.863703", "inclusive_start": false, "inclusive_end": false}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.216.216 : Administrator: |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [rest_client.urllib_request] Making a rest request api=http://172.23.216.216:8094/api/index/custom_index/query verb=POST params=b'{"indexName": "custom_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "2003-03-16T19:45:00", "end": "2024-01-03T02:52:51.863703", "inclusive_start": false, "inclusive_end": false}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 1, 'failed': 0, 'successful': 1} |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "join_date", "start": "2003-03-16T19:45:00", "end": "2024-01-03T02:52:51.863703", "inclusive_start": false, "inclusive_end": false} is 214 (took 1.289702ms) |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [es_base.search] ES query '{'query': {'filtered': {'filter': {'range': {'join_date': {'gt': '2003-03-16T19:45:00', 'lt': '2024-01-03T02:52:51.863703'}}}}}}' |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"gt": "2003-03-16T19:45:00", "lt": "2024-01-03T02:52:51.863703"}}}}}} on es_index is 214 (took 16ms) |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] Running N1QL query: select meta().id from default:default.scope1.collection1 where type='emp' and search(default, {"field": "join_date", "start": "2003-03-16T19:45:00", "end": "2024-01-03T02:52:51.863703", "inclusive_start": false, "inclusive_end": false},{"index": "custom_index"}) |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=select+meta%28%29.id+from+default%3Adefault.scope1.collection1+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22join_date%22%2C+%22start%22%3A+%222003-03-16T19%3A45%3A00%22%2C+%22end%22%3A+%222024-01-03T02%3A52%3A51.863703%22%2C+%22inclusive_start%22%3A+false%2C+%22inclusive_end%22%3A+false%7D%2C%7B%22index%22%3A+%22custom_index%22%7D%29 |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] N1QL hits for query: "select meta().id from default:default.scope1.collection1 where type='emp' and search(default, {\"field\": \"join_date\", \"start\": \"2003-03-16T19:45:00\", \"end\": \"2024-01-03T02:52:51.863703\", \"inclusive_start\": false, \"inclusive_end\": false},{\"index\": \"custom_index\"})" is 214 (took 17.684696ms) |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by N1QL, doc_ids verified |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] Running N1QL query: select meta().id,* from default:default.scope1.collection1 where type='emp' and search(default, {"field": "join_date", "start": "2003-03-16T19:45:00", "end": "2024-01-03T02:52:51.863703", "inclusive_start": false, "inclusive_end": false},{"index": "custom_index"}) |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=select+meta%28%29.id%2C%2A+from+default%3Adefault.scope1.collection1+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22join_date%22%2C+%22start%22%3A+%222003-03-16T19%3A45%3A00%22%2C+%22end%22%3A+%222024-01-03T02%3A52%3A51.863703%22%2C+%22inclusive_start%22%3A+false%2C+%22inclusive_end%22%3A+false%7D%2C%7B%22index%22%3A+%22custom_index%22%7D%29 |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] N1QL hits for query: "select meta().id,* from default:default.scope1.collection1 where type='emp' and search(default, {\"field\": \"join_date\", \"start\": \"2003-03-16T19:45:00\", \"end\": \"2024-01-03T02:52:51.863703\", \"inclusive_start\": false, \"inclusive_end\": false},{\"index\": \"custom_index\"})" is 214 (took 61.850788ms) |
2024-01-03 02:53:30 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by N1QL, doc_ids verified |
It's also not like queries prev to this also never passed, for instance query #15 passed
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 15 ---------------------------------------------------- |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "custom_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "2000-12-17T08:33:00", "end": "2024-01-03T02:52:51.856465", "inclusive_start": true, "inclusive_end": true}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.109.155 : Administrator: |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.urllib_request] Making a rest request api=http://172.23.109.155:8094/api/index/custom_index/query verb=POST params=b'{"indexName": "custom_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "2000-12-17T08:33:00", "end": "2024-01-03T02:52:51.856465", "inclusive_start": true, "inclusive_end": true}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 1, 'failed': 0, 'successful': 1, 'errors': {}} |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "join_date", "start": "2000-12-17T08:33:00", "end": "2024-01-03T02:52:51.856465", "inclusive_start": true, "inclusive_end": true} is 250 (took 1.512801ms) |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [es_base.search] ES query '{'query': {'filtered': {'filter': {'range': {'join_date': {'gte': '2000-12-17T08:33:00', 'lte': '2024-01-03T02:52:51.856465'}}}}}}' |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"gte": "2000-12-17T08:33:00", "lte": "2024-01-03T02:52:51.856465"}}}}}} on es_index is 250 (took 86ms) |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [task.execute] Running N1QL query: select meta().id from default:default.scope1.collection1 where type='emp' and search(default, {"field": "join_date", "start": "2000-12-17T08:33:00", "end": "2024-01-03T02:52:51.856465", "inclusive_start": true, "inclusive_end": true},{"index": "custom_index"}) |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=select+meta%28%29.id+from+default%3Adefault.scope1.collection1+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22join_date%22%2C+%22start%22%3A+%222000-12-17T08%3A33%3A00%22%2C+%22end%22%3A+%222024-01-03T02%3A52%3A51.856465%22%2C+%22inclusive_start%22%3A+true%2C+%22inclusive_end%22%3A+true%7D%2C%7B%22index%22%3A+%22custom_index%22%7D%29 |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [task.execute] N1QL hits for query: "select meta().id from default:default.scope1.collection1 where type='emp' and search(default, {\"field\": \"join_date\", \"start\": \"2000-12-17T08:33:00\", \"end\": \"2024-01-03T02:52:51.856465\", \"inclusive_start\": true, \"inclusive_end\": true},{\"index\": \"custom_index\"})" is 250 (took 14.693841ms) |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by N1QL, doc_ids verified |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [task.execute] Running N1QL query: select meta().id,* from default:default.scope1.collection1 where type='emp' and search(default, {"field": "join_date", "start": "2000-12-17T08:33:00", "end": "2024-01-03T02:52:51.856465", "inclusive_start": true, "inclusive_end": true},{"index": "custom_index"}) |
2024-01-03 02:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=select+meta%28%29.id%2C%2A+from+default%3Adefault.scope1.collection1+where+type%3D%27emp%27+and+search%28default%2C+%7B%22field%22%3A+%22join_date%22%2C+%22start%22%3A+%222000-12-17T08%3A33%3A00%22%2C+%22end%22%3A+%222024-01-03T02%3A52%3A51.856465%22%2C+%22inclusive_start%22%3A+true%2C+%22inclusive_end%22%3A+true%7D%2C%7B%22index%22%3A+%22custom_index%22%7D%29 |
2024-01-03 02:53:24 | INFO | MainProcess | Cluster_Thread | [task.execute] N1QL hits for query: "select meta().id,* from default:default.scope1.collection1 where type='emp' and search(default, {\"field\": \"join_date\", \"start\": \"2000-12-17T08:33:00\", \"end\": \"2024-01-03T02:52:51.856465\", \"inclusive_start\": true, \"inclusive_end\": true},{\"index\": \"custom_index\"})" is 250 (took 58.573643ms) |
2024-01-03 02:53:24 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by N1QL, doc_ids verified |
So, queries are randomly failing with PRIMARY INDEX not available, I think probably somewhere between the test n1ql loses that fts index connection, It gets that fts index doesn't exist and then in turn starts to check for GSI Primary index to execute this query, which also doesn't exist and therefore fails.
Job logs:
7.2.4 - http://qa.sc.couchbase.com/job/test_suite_executor/659199/consoleFull
7.2.3 - http://qa.sc.couchbase.com/job/test_suite_executor/659212/consoleFull