Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28780

[FTS Scorch] Query on numeric fields yields lesser results in FTS compared to ES

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Build : 5.5.0-2211

      Test : ./testrunner -i /tmp/testexec.498.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,cluster=D+F,index_type=scorch -t fts.stable_topology_fts.StableTopFTS.index_query_custom_mapping,items=1000,custom_map=True,cm_id=3,num_queries=100,compare_es=True,GROUP=P0

      Index Definition :

      {
         "mapping": {
            "default_analyzer": "standard", 
            "default_datetime_parser": "dateTimeOptional", 
            "type_field": "type", 
            "default_mapping": {
               "default_analyzer": "", 
               "dynamic": true, 
               "enabled": false
            }, 
            "default_type": "_default", 
            "analysis": {}, 
            "default_field": "_all", 
            "types": {
               "emp": {
                  "enabled": true, 
                  "dynamic": false, 
                  "properties": {
                     "salary": {
                        "enabled": true, 
                        "dynamic": false, 
                        "properties": {}, 
                        "fields": [
                           {
                              "index": true, 
                              "name": "salary", 
                              "include_in_all": true, 
                              "type": "number", 
                              "analyzer": "", 
                              "include_term_vectors": true, 
                              "store": false
                           }
                        ]
                     }, 
                     "mutated": {
                        "enabled": true, 
                        "dynamic": false, 
                        "properties": {}, 
                        "fields": [
                           {
                              "index": true, 
                              "name": "mutated", 
                              "include_in_all": true, 
                              "type": "number", 
                              "analyzer": "", 
                              "include_term_vectors": true, 
                              "store": false
                           }
                        ]
                     }, 
                     "manages": {
                        "enabled": true, 
                        "dynamic": false, 
                        "properties": {
                           "team_size": {
                              "enabled": true, 
                              "dynamic": false, 
                              "properties": {}, 
                              "fields": [
                                 {
                                    "index": true, 
                                    "name": "team_size", 
                                    "include_in_all": true, 
                                    "type": "number", 
                                    "analyzer": "", 
                                    "include_term_vectors": true, 
                                    "store": false
                                 }
                              ]
                           }
                        }, 
                        "fields": []
                     }
                  }, 
                  "fields": []
               }
            }
         }, 
         "store": {
            "mossStoreOptions": {}, 
            "kvStoreName": "mossStore", 
            "indexType": "scorch"
         }
      }
      

      Failing queries:
      [2018-03-17 21:22:48,630] - [task:1324] INFO - FTS hits for query:

      {"inclusive_max": false, "field": "manages.team_size", "max": 822, "inclusive_min": false, "min": 7}

      is 251 (took 2.346684ms)
      [2018-03-17 21:22:48,651] - [task:1334] INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"manages.team_size": {"lt": 822, "gt": 7}}}}}} on es_index is 259 (took 8ms)
      [2018-03-17 21:22:48,651] - [task:1339] ERROR - FAIL: FTS hits: 251, while ES hits: 259
      [2018-03-17 21:22:48,651] - [task:1354] ERROR - FAIL: Following 8 docs were not returned by FTS, but ES, printing 50: [u'emp10000094', u'emp10000708', u'emp10000841', u'emp10000482', u'emp10000940', u'emp10000236', u'emp10000837', u'emp10000562']

      [2018-03-17 21:22:49,874] - [task:1324] INFO - FTS hits for query:

      {"inclusive_max": false, "field": "manages.team_size", "max": 5241, "inclusive_min": false, "min": 7}

      is 251 (took 5.105681ms)
      [2018-03-17 21:22:49,895] - [task:1334] INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"manages.team_size": {"lt": 5241, "gt": 7}}}}}} on es_index is 259 (took 9ms)
      [2018-03-17 21:22:49,896] - [task:1339] ERROR - FAIL: FTS hits: 251, while ES hits: 259
      [2018-03-17 21:22:49,896] - [task:1354] ERROR - FAIL: Following 8 docs were not returned by FTS, but ES, printing 50: [u'emp10000094', u'emp10000708', u'emp10000841', u'emp10000482', u'emp10000940', u'emp10000236', u'emp10000837', u'emp10000562']

      [2018-03-17 21:22:50,187] - [task:1324] INFO - FTS hits for query:

      {"inclusive_max": false, "field": "manages.team_size", "max": 5919, "inclusive_min": false, "min": 7}

      is 251 (took 6.825116ms)
      [2018-03-17 21:22:50,209] - [task:1334] INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"manages.team_size": {"lt": 5919, "gt": 7}}}}}} on es_index is 259 (took 9ms)
      [2018-03-17 21:22:50,210] - [task:1339] ERROR - FAIL: FTS hits: 251, while ES hits: 259
      [2018-03-17 21:22:50,210] - [task:1354] ERROR - FAIL: Following 8 docs were not returned by FTS, but ES, printing 50: [u'emp10000094', u'emp10000708', u'emp10000841', u'emp10000482', u'emp10000940', u'emp10000236', u'emp10000837', u'emp10000562']

      [2018-03-17 21:22:50,406] - [task:1324] INFO - FTS hits for query:

      {"query": "+manages.team_size:>=9"}

      is 166 (took 2.340861ms)
      [2018-03-17 21:22:50,424] - [task:1334] INFO - ES hits for query: {"query": {"query_string":

      {"query": "+manages.team_size:>=9"}

      }} on es_index is 172 (took 9ms)
      [2018-03-17 21:22:50,425] - [task:1339] ERROR - FAIL: FTS hits: 166, while ES hits: 172
      [2018-03-17 21:22:50,425] - [task:1354] ERROR - FAIL: Following 6 docs were not returned by FTS, but ES, printing 50: [u'emp10000094', u'emp10000708', u'emp10000841', u'emp10000940', u'emp10000837', u'emp10000562']

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mschoch Marty Schoch [X] (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty