Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30950

[FTS] Query rejection noted even when fts_quota is doubled

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 6.0.0
    • 6.0.0
    • fts
    • Untriaged
    • Centos 64-bit
    • Yes

    Description

      Build
      6.0.0-1489

      Testcase
      ./testrunner -i /tmp/testexec.25109.ini -p get-cbcollect-info=True,get-logs=False,stop-on-failure=False,fts_quota=1024 -t fts.stable_topology_fts.StableTopFTS.test_geo_query,items=1000,skip-cleanup=True,fail-on-errors=False,compare_es=True,cluster=D+F,num_queries=20

      A regression between 6.0.0-1432 and 6.0.0-1484.

      Please note the same test used to pass on build 6.0.0-1432 with 512mb quota. Now, with query rejection, after increasing quota to 1gb, the test still fails with multiple queries being rejected.

      2018-08-16 09:57:03 | INFO | MainProcess | test_thread | [fts_base.create] Creating fulltext-index geo-index on 172.23.104.98
      		 
      2018-08-16 09:57:03 | INFO | MainProcess | test_thread | [rest_client.create_fts_index] {"params": {"mapping": {"types": {"earthquake": {"enabled": true, "properties": {"geo": {"enabled": true, "dynamic": false, "fields": [{"include_in_all": true, "index": true, "type": "geopoint", "name": "geo", "store": false}]}}}}}}, "name": "geo-index", "planParams": {}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "uuid": ""}
      		 
      2018-08-16 09:57:03 | INFO | MainProcess | test_thread | [rest_client.create_fts_index] Index geo-index created
      		 
      2018-08-16 09:57:03 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validating index distribution for geo-index ...
      		 
      2018-08-16 09:57:03 | INFO | MainProcess | test_thread | [fts_base.sleep] sleep for 5 secs. No pindexes found, waiting for index to get created ...
      		 
      2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Number of PIndexes = 6
      		 
      2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Every pIndex serves 171 partitions or lesser
      		 
      2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Expecting num of partitions in each node in range 853-1024
      		 
      2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Node fb573e49645b8c3a7b38f5ebc60813f3 houses 6 pindexes which serve 1024 partitions
      		 
      2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [fts_base.create_geo_index_and_load] Loading earthquakes.json ...
      		 
      2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [task.__init__] Starting operation 'create' on Elastic Search ...
      		 
      2018-08-16 09:57:08 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.98:11210 default
      		 
      2018-08-16 09:57:09 | INFO | MainProcess | test_thread | [fts_base.sleep] sleep for 10 secs. Waiting to load earthquakes.json ... ...
      		 
      2018-08-16 09:57:09 | INFO | Process-2   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.98:11210 default
      		 
      2018-08-16 09:57:09 | INFO | Process-3   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.98:11210 default
      		 
      2018-08-16 09:57:09 | INFO | Process-4   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.98:11210 default
      		 
      2018-08-16 09:57:09 | INFO | Process-5   | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.98:11210 default
      		 
      2018-08-16 09:57:09 | INFO | MainProcess | ES_Thread  | [task.execute] 1000 documents bulk loaded into ES
      		 
      2018-08-16 09:57:09 | INFO | MainProcess | ES_Thread  | [task.execute] ES index count for 'es_index': 1000
      		 
      2018-08-16 09:57:19 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.104.98:8091/pools/default/buckets/default/stats?zoom=minute
      		 
      2018-08-16 09:57:19 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 1000, docs in FTS index 'geo-index': 1000, docs in ES index: 1000 
      2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 1 ----------------------------------------------------
      		 
      2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-118.77, 34.243], "top_left": [-85.3, 10.02]}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 6, u'failed': 0, u'total': 6}
      		 
      2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-118.77, 34.243], "top_left": [-85.3, 10.02]} is 0 (took 1.261266ms)
      		 
      2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-118.77, 34.243], "top_left": [-85.3, 10.02]}}}, "query": {"match_all": {}}} on es_index is 0 (took 2ms)
      		 
      2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
      		 
      2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 2 ----------------------------------------------------
      		 
      2018-08-16 09:57:19 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-118.77, 34.243], "top_left": [-149.34, 61.28]}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 6, u'failed': 0, u'total': 6}
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-118.77, 34.243], "top_left": [-149.34, 61.28]} is 212 (took 1937.296673ms)
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-118.77, 34.243], "top_left": [-149.34, 61.28]}}}, "query": {"match_all": {}}} on es_index is 212 (took 4ms)
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 3 ----------------------------------------------------
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-119.23, 39.59], "top_left": [-119.23, 39.59]}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 6, u'failed': 0, u'total': 6}
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-119.23, 39.59], "top_left": [-119.23, 39.59]} is 0 (took 10.530667ms)
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-119.23, 39.59], "top_left": [-119.23, 39.59]}}}, "query": {"match_all": {}}} on es_index is 0 (took 1ms)
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 4 ----------------------------------------------------
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"distance": "10km", "location": {"lat": 19.59, "lon": -64.35}, "field": "geo"}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 6, u'failed': 0, u'total': 6}
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"distance": "10km", "location": {"lat": 19.59, "lon": -64.35}, "field": "geo"} is 6 (took 13.782679ms)
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_distance": {"distance": "10km", "geo": {"lat": 19.59, "lon": -64.35}}}, "query": {"match_all": {}}} on es_index is 6 (took 1ms)
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 5 ----------------------------------------------------
      		 
      2018-08-16 09:57:21 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"distance": "1000mi", "location": {"lat": 34.243, "lon": -118.77}, "field": "geo"}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 2, u'failed': 4, u'total': 6, u'errors': {u'geo-index_6fc288d1cbf92ab3_f4e0a48a': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_18572d87': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_6ddbfb54': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_13aa53f3': u'search request rejected'}}
      		 
      2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS returned partial results...skipping ES validation
      		 
      2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"distance": "1000mi", "location": {"lat": 34.243, "lon": -118.77}, "field": "geo"} is 129 (took 2101.908444ms)
      		 
      2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_distance": {"distance": "1000mi", "geo": {"lat": 34.243, "lon": -118.77}}}, "query": {"match_all": {}}} on es_index is 405 (took 8ms)
      		 
      2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 6 ----------------------------------------------------
      		 
      2018-08-16 09:57:23 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-85.3, 10.02], "top_left": [-122.82, 38.83]}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {u'successful': 0, u'failed': 6, u'total': 6, u'errors': {u'geo-index_6fc288d1cbf92ab3_54820232': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_18572d87': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_f4e0a48a': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_6ddbfb54': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_aa574717': u'search request rejected', u'geo-index_6fc288d1cbf92ab3_13aa53f3': u'search request rejected'}}
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-85.3, 10.02], "top_left": [-122.82, 38.83]} is 0 (took 2545.624492ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-85.3, 10.02], "top_left": [-122.82, 38.83]}}}, "query": {"match_all": {}}} on es_index is 319 (took 10ms)
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [task.execute] FAIL: FTS hits: 0, while ES hits: 319
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [task.execute] FAIL: Following 319 docs were not returned by FTS, but ES, printing 50: [u'earthquake584', u'earthquake581', u'earthquake580', u'earthquake583', u'earthquake919', u'earthquake316', u'earthquake55', u'earthquake181', u'earthquake182', u'earthquake185', u'earthquake467', u'earthquake921', u'earthquake743', u'earthquake740', u'earthquake995', u'earthquake65', u'earthquake67', u'earthquake66', u'earthquake61', u'earthquake60', u'earthquake346', u'earthquake69', u'earthquake68', u'earthquake690', u'earthquake677', u'earthquake913', u'earthquake859', u'earthquake674', u'earthquake916', u'earthquake917', u'earthquake349', u'earthquake915', u'earthquake853', u'earthquake852', u'earthquake851', u'earthquake177', u'earthquake173', u'earthquake98', u'earthquake99', u'earthquake595', u'earthquake592', u'earthquake207', u'earthquake459', u'earthquake205', u'earthquake325', u'earthquake324', u'earthquake92', u'earthquake454', u'earthquake453', u'earthquake618']
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 7 ----------------------------------------------------
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 19.59, "lon": -64.35}, "top_left": {"lat": 38.83, "lon": -122.82}}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 19.59, "lon": -64.35}, "top_left": {"lat": 38.83, "lon": -122.82}}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":{"lat":19.59,"lon":-64.35},"field":"geo","top_left":{"lat":38.83,"lon":-122.82}},"size":10000000},"status":"fail"} auth: Administrator:password
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {}
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": {"lat": 19.59, "lon": -64.35}, "top_left": {"lat": 38.83, "lon": -122.82}} is -1 (took 0.0ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": {"lat": 19.59, "lon": -64.35}, "top_left": {"lat": 38.83, "lon": -122.82}}}}, "query": {"match_all": {}}} on es_index is 316 (took 7ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 8 ----------------------------------------------------
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [52.9, 27.33], "top_left": [-126, 39]}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [52.9, 27.33], "top_left": [-126, 39]}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":[52.9,27.33],"field":"geo","top_left":[-126,39]},"size":10000000},"status":"fail"} auth: Administrator:password
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {}
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [52.9, 27.33], "top_left": [-126, 39]} is -1 (took 0.0ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [52.9, 27.33], "top_left": [-126, 39]}}}, "query": {"match_all": {}}} on es_index is 350 (took 7ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 9 ----------------------------------------------------
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"distance": "1000km", "location": [-64.35, 19.59], "field": "geo"}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"distance": "1000km", "location": [-64.35, 19.59], "field": "geo"}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"distance":"1000km","field":"geo","location":[-64.35,19.59]},"size":10000000},"status":"fail"} auth: Administrator:password
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {}
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"distance": "1000km", "location": [-64.35, 19.59], "field": "geo"} is -1 (took 0.0ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_distance": {"distance": "1000km", "geo": [-64.35, 19.59]}}, "query": {"match_all": {}}} on es_index is 236 (took 5ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 10 ----------------------------------------------------
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 32, "lon": -115.22}, "top_left": {"lat": 38.83, "lon": -122.82}}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 32, "lon": -115.22}, "top_left": {"lat": 38.83, "lon": -122.82}}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":{"lat":32,"lon":-115.22},"field":"geo","top_left":{"lat":38.83,"lon":-122.82}},"size":10000000},"status":"fail"} auth: Administrator:password
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {}
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": {"lat": 32, "lon": -115.22}, "top_left": {"lat": 38.83, "lon": -122.82}} is -1 (took 0.0ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": {"lat": 32, "lon": -115.22}, "top_left": {"lat": 38.83, "lon": -122.82}}}}, "query": {"match_all": {}}} on es_index is 284 (took 5ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 11 ----------------------------------------------------
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-64.35, 19.59], "top_left": [-73.04, 6.76]}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-64.35, 19.59], "top_left": [-73.04, 6.76]}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":[-64.35,19.59],"field":"geo","top_left":[-73.04,6.76]},"size":10000000},"status":"fail"} auth: Administrator:password
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {}
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-64.35, 19.59], "top_left": [-73.04, 6.76]} is -1 (took 0.0ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-64.35, 19.59], "top_left": [-73.04, 6.76]}}}, "query": {"match_all": {}}} on es_index is 0 (took 1ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 12 ----------------------------------------------------
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-119.23, 39.59], "top_left": [52.9, 27.33]}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": [-119.23, 39.59], "top_left": [52.9, 27.33]}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":[-119.23,39.59],"field":"geo","top_left":[52.9,27.33]},"size":10000000},"status":"fail"} auth: Administrator:password
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {}
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": [-119.23, 39.59], "top_left": [52.9, 27.33]} is -1 (took 0.0ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": [-119.23, 39.59], "top_left": [52.9, 27.33]}}}, "query": {"match_all": {}}} on es_index is 0 (took 1ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 13 ----------------------------------------------------
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 39, "lon": -126}, "top_left": {"lat": 27.33, "lon": 52.9}}, "size": 10000000} on node: 172.23.104.98:
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.98:8094/api/index/geo-index/query body: {"indexName": "geo-index", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "geo", "bottom_right": {"lat": 39, "lon": -126}, "top_left": {"lat": 27.33, "lon": 52.9}}, "size": 10000000} headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 503 reason: rest_index: Query, indexName: geo-index, err: search request rejected {"error":"rest_index: Query, indexName: geo-index, err: search request rejected","request":{"ctl":{"consistency":{"level":"","vectors":{}},"timeout":60000},"explain":false,"fields":[],"from":0,"indexName":"geo-index","query":{"bottom_right":{"lat":39,"lon":-126},"field":"geo","top_left":{"lat":27.33,"lon":52.9}},"size":10000000},"status":"fail"} auth: Administrator:password
      		 
      2018-08-16 09:57:26 | ERROR | MainProcess | Cluster_Thread | [fts_base.execute_query] Error running query: 'NoneType' object is not iterable
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {}
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "geo", "bottom_right": {"lat": 39, "lon": -126}, "top_left": {"lat": 27.33, "lon": 52.9}} is -1 (took 0.0ms)
      		 
      2018-08-16 09:57:26 | INFO | MainProcess | Cluster_Thread | [task.execute] ES hits for query: {"filter": {"geo_bounding_box": {"geo": {"bottom_right": {"lat": 39, "lon": -126}, "top_left": {"lat": 27.33, "lon": 52.9}}}}, "query": {"match_all": {}}} on es_index is 0 (took 1ms)

      Attached logs from .66 and .70.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-30896 FTS query returns "search request rejected" 6.0.0-1484

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              steve Steve Yen
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty