Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
6.0.0
-
Untriaged
-
Centos 64-bit
-
Yes
Description
Build
6.0.0-1489
Testcase
./testrunner -i /tmp/testexec.25109.ini -p get-cbcollect-info=True,get-logs=False,stop-on-failure=False,fts_quota=1024 -t fts.stable_topology_fts.StableTopFTS.test_geo_query,items=1000,skip-cleanup=True,fail-on-errors=False,compare_es=True,cluster=D+F,num_queries=20
A regression between 6.0.0-1432 and 6.0.0-1484.
Please note the same test used to pass on build 6.0.0-1432 with 512mb quota. Now, with query rejection, after increasing quota to 1gb, the test still fails with multiple queries being rejected.
2018-08-16 09:57:03 | INFO | MainProcess | test_thread | [fts_base.create] Creating fulltext-index geo-index on 172.23.104.98 |
2018-08-16 09:57:03 | INFO | MainProcess | test_thread | [rest_client.create_fts_index] {"params": {"mapping": {"types": {"earthquake": {"enabled": true, "properties": {"geo": {"enabled": true, "dynamic": false, "fields": [{"include_in_all": true, "index": true, "type": "geopoint", "name": "geo", "store": false}]}}}}}}, "name": "geo-index", "planParams": {}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "uuid": ""} |
2018-08-16 09:57:03 | INFO | MainProcess | test_thread | [rest_client.create_fts_index] Index geo-index created |
2018-08-16 09:57:03 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validating index distribution for geo-index ... |
2018-08-16 09:57:03 | INFO | MainProcess | test_thread | [fts_base.sleep] sleep for 5 secs. No pindexes found, waiting for index to get created ... |
2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Number of PIndexes = 6 |
2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Every pIndex serves 171 partitions or lesser |
2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Expecting num of partitions in each node in range 853-1024 |
2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Node fb573e49645b8c3a7b38f5ebc60813f3 houses 6 pindexes which serve 1024 partitions |
2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [fts_base.create_geo_index_and_load] Loading earthquakes.json ... |
2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [task.__init__] Starting operation 'create' on Elastic Search ... |
2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.98:11210 default |
2018-08-16 09:57:09 | INFO | MainProcess | test_thread | [fts_base.sleep] sleep for 10 secs. Waiting to load earthquakes.json ... ... |
2018-08-16 09:57:09 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.98:11210 default |
2018-08-16 09:57:09 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.98:11210 default |
2018-08-16 09:57:09 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.98:11210 default |
2018-08-16 09:57:09 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.98:11210 default |
2018-08-16 09:57:09 | INFO | MainProcess | ES_Thread | [task.execute] 1000 documents bulk loaded into ES |
2018-08-16 09:57:09 | INFO | MainProcess | ES_Thread | [task.execute] ES index count for 'es_index': 1000 |
2018-08-16 09:57:19 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.104.98:8091/pools/default/buckets/default/stats?zoom=minute |
2018-08-16 09:57:19 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 1000, docs in FTS index 'geo-index': 1000, docs in ES index: 1000 |
2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 1 ---------------------------------------------------- |
2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-118.77, 34.243], "top_left": [-85.3, 10.02]}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 6, u'failed': 0, u'total': 6} |
2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-118.77, 34.243], "top_left": [-85.3, 10.02]} is 0 (took 1.261266ms) |
2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-118.77, 34.243], "top_left": [-85.3, 10.02]}}}, "query": {"match_all": {}}} on es_index is 0 (took 2ms) |
2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified |
2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 2 ---------------------------------------------------- |
2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-118.77, 34.243], "top_left": [-149.34, 61.28]}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 6, u'failed': 0, u'total': 6} |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-118.77, 34.243], "top_left": [-149.34, 61.28]} is 212 (took 1937.296673ms) |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-118.77, 34.243], "top_left": [-149.34, 61.28]}}}, "query": {"match_all": {}}} on es_index is 212 (took 4ms) |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 3 ---------------------------------------------------- |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-119.23, 39.59], "top_left": [-119.23, 39.59]}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 6, u'failed': 0, u'total': 6} |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-119.23, 39.59], "top_left": [-119.23, 39.59]} is 0 (took 10.530667ms) |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-119.23, 39.59], "top_left": [-119.23, 39.59]}}}, "query": {"match_all": {}}} on es_index is 0 (took 1ms) |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 4 ---------------------------------------------------- |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"distance": "10km", "location": {"lat": 19.59, "lon": -64.35}, "field": "geo"}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 6, u'failed': 0, u'total': 6} |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"distance": "10km", "location": {"lat": 19.59, "lon": -64.35}, "field": "geo"} is 6 (took 13.782679ms) |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_distance": {"distance": "10km", "geo": {"lat": 19.59, "lon": -64.35}}}, "query": {"match_all": {}}} on es_index is 6 (took 1ms) |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 5 ---------------------------------------------------- |
2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"distance": "1000mi", "location": {"lat": 34.243, "lon": -118.77}, "field": "geo"}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 2, u'failed': 4, u'total': 6, u'errors': {u'geo-index_6fc288d1cbf92ab3_f4e0a48a': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_18572d87': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_6ddbfb54': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_13aa53f3': u'search request rejected'}} |
2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS returned partial results...skipping ES validation |
2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"distance": "1000mi", "location": {"lat": 34.243, "lon": -118.77}, "field": "geo"} is 129 (took 2101.908444ms) |
2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_distance": {"distance": "1000mi", "geo": {"lat": 34.243, "lon": -118.77}}}, "query": {"match_all": {}}} on es_index is 405 (took 8ms) |
2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 6 ---------------------------------------------------- |
2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-85.3, 10.02], "top_left": [-122.82, 38.83]}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 0, u'failed': 6, u'total': 6, u'errors': {u'geo-index_6fc288d1cbf92ab3_54820232': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_18572d87': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_f4e0a48a': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_6ddbfb54': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_aa574717': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_13aa53f3': u'search request rejected'}} |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-85.3, 10.02], "top_left": [-122.82, 38.83]} is 0 (took 2545.624492ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-85.3, 10.02], "top_left": [-122.82, 38.83]}}}, "query": {"match_all": {}}} on es_index is 319 (took 10ms) |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [task.execute] FAIL: FTS hits: 0, while ES hits: 319 |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [task.execute] FAIL: Following 319 docs were not returned by FTS, but ES, printing 50: [u'earthquake584', u'earthquake581', u'earthquake580', u'earthquake583', u'earthquake919', u'earthquake316', u'earthquake55', u'earthquake181', u'earthquake182', u'earthquake185', u'earthquake467', u'earthquake921', u'earthquake743', u'earthquake740', u'earthquake995', u'earthquake65', u'earthquake67', u'earthquake66', u'earthquake61', u'earthquake60', u'earthquake346', u'earthquake69', u'earthquake68', u'earthquake690', u'earthquake677', u'earthquake913', u'earthquake859', u'earthquake674', u'earthquake916', u'earthquake917', u'earthquake349', u'earthquake915', u'earthquake853', u'earthquake852', u'earthquake851', u'earthquake177', u'earthquake173', u'earthquake98', u'earthquake99', u'earthquake595', u'earthquake592', u'earthquake207', u'earthquake459', u'earthquake205', u'earthquake325', u'earthquake324', u'earthquake92', u'earthquake454', u'earthquake453', u'earthquake618'] |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 7 ---------------------------------------------------- |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 19.59, "lon": -64.35}, "top_left": {"lat": 38.83, "lon": -122.82}}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 19.59, "lon": -64.35}, "top_left": {"lat": 38.83, "lon": -122.82}}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":{"lat":19.59,"lon":-64.35},"field":"geo","top_left":{"lat":38.83,"lon":-122.82}},"size":10000000},"status":"fail"} auth: Administrator:password |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {} |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": {"lat": 19.59, "lon": -64.35}, "top_left": {"lat": 38.83, "lon": -122.82}} is -1 (took 0.0ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": {"lat": 19.59, "lon": -64.35}, "top_left": {"lat": 38.83, "lon": -122.82}}}}, "query": {"match_all": {}}} on es_index is 316 (took 7ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 8 ---------------------------------------------------- |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [52.9, 27.33], "top_left": [-126, 39]}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [52.9, 27.33], "top_left": [-126, 39]}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":[52.9,27.33],"field":"geo","top_left":[-126,39]},"size":10000000},"status":"fail"} auth: Administrator:password |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {} |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [52.9, 27.33], "top_left": [-126, 39]} is -1 (took 0.0ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [52.9, 27.33], "top_left": [-126, 39]}}}, "query": {"match_all": {}}} on es_index is 350 (took 7ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 9 ---------------------------------------------------- |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"distance": "1000km", "location": [-64.35, 19.59], "field": "geo"}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"distance": "1000km", "location": [-64.35, 19.59], "field": "geo"}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"distance":"1000km","field":"geo","location":[-64.35,19.59]},"size":10000000},"status":"fail"} auth: Administrator:password |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {} |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"distance": "1000km", "location": [-64.35, 19.59], "field": "geo"} is -1 (took 0.0ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_distance": {"distance": "1000km", "geo": [-64.35, 19.59]}}, "query": {"match_all": {}}} on es_index is 236 (took 5ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 10 ---------------------------------------------------- |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 32, "lon": -115.22}, "top_left": {"lat": 38.83, "lon": -122.82}}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 32, "lon": -115.22}, "top_left": {"lat": 38.83, "lon": -122.82}}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":{"lat":32,"lon":-115.22},"field":"geo","top_left":{"lat":38.83,"lon":-122.82}},"size":10000000},"status":"fail"} auth: Administrator:password |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {} |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": {"lat": 32, "lon": -115.22}, "top_left": {"lat": 38.83, "lon": -122.82}} is -1 (took 0.0ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": {"lat": 32, "lon": -115.22}, "top_left": {"lat": 38.83, "lon": -122.82}}}}, "query": {"match_all": {}}} on es_index is 284 (took 5ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 11 ---------------------------------------------------- |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-64.35, 19.59], "top_left": [-73.04, 6.76]}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-64.35, 19.59], "top_left": [-73.04, 6.76]}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":[-64.35,19.59],"field":"geo","top_left":[-73.04,6.76]},"size":10000000},"status":"fail"} auth: Administrator:password |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {} |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-64.35, 19.59], "top_left": [-73.04, 6.76]} is -1 (took 0.0ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-64.35, 19.59], "top_left": [-73.04, 6.76]}}}, "query": {"match_all": {}}} on es_index is 0 (took 1ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 12 ---------------------------------------------------- |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-119.23, 39.59], "top_left": [52.9, 27.33]}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-119.23, 39.59], "top_left": [52.9, 27.33]}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":[-119.23,39.59],"field":"geo","top_left":[52.9,27.33]},"size":10000000},"status":"fail"} auth: Administrator:password |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {} |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-119.23, 39.59], "top_left": [52.9, 27.33]} is -1 (took 0.0ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-119.23, 39.59], "top_left": [52.9, 27.33]}}}, "query": {"match_all": {}}} on es_index is 0 (took 1ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 13 ---------------------------------------------------- |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 39, "lon": -126}, "top_left": {"lat": 27.33, "lon": 52.9}}, "size": 10000000} on node: 172.23.104.98: |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 39, "lon": -126}, "top_left": {"lat": 27.33, "lon": 52.9}}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":{"lat":39,"lon":-126},"field":"geo","top_left":{"lat":27.33,"lon":52.9}},"size":10000000},"status":"fail"} auth: Administrator:password |
2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {} |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": {"lat": 39, "lon": -126}, "top_left": {"lat": 27.33, "lon": 52.9}} is -1 (took 0.0ms) |
2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": {"lat": 39, "lon": -126}, "top_left": {"lat": 27.33, "lon": 52.9}}}}, "query": {"match_all": {}}} on es_index is 0 (took 1ms) |
Attached logs from .66 and .70.
Attachments
Issue Links
- relates to
-
MB-30896 FTS query returns "search request rejected" 6.0.0-1484
- Closed