Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53050

[FTS] updating a geopoint fts index after upgrade returns wrong results

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 7.1.2
    • 7.1.2
    • fts
    • 7.1.2-3386
    • Untriaged
    • 1
    • Yes

    Description

      the geopoint index is fine post upgrade until it is updated, then it no longer works

      3 nodes on 7.0.4 release version
      1 kv - 172.23.104.110
      1 kv fts - 172.23.104.173
      1 n1ql fts - 172.23.104.174

      load travel-sample
      create geopoint fts index w/2 partitions
      (I added a child field mapping to a default index on field geo)

      curl -XPUT -H "Content-Type: application/json" \
      -u <username>:<password> http://172.23.104.175:8094/api/index/geo-index -d \
      '{
        "type": "fulltext-index",
        "name": "geo-index",
        "uuid": "4e79c8d26e52067a",
        "sourceType": "gocbcore",
        "sourceName": "travel-sample",
        "sourceUUID": "80f8ba408b50f1b0c0d02053ded93caa",
        "planParams": {
          "maxPartitionsPerPIndex": 512,
          "indexPartitions": 2
        },
        "params": {
          "doc_config": {
            "docid_prefix_delim": "",
            "docid_regexp": "",
            "mode": "type_field",
            "type_field": "type"
          },
          "mapping": {
            "analysis": {},
            "default_analyzer": "standard",
            "default_datetime_parser": "dateTimeOptional",
            "default_field": "_all",
            "default_mapping": {
              "dynamic": true,
              "enabled": true,
              "properties": {
                "geo": {
                  "dynamic": false,
                  "enabled": true,
                  "fields": [
                    {
                      "include_in_all": true,
                      "index": true,
                      "name": "geo",
                      "store": true,
                      "type": "geopoint"
                    }
                  ]
                }
              }
            },
            "default_type": "_default",
            "docvalues_dynamic": false,
            "index_dynamic": true,
            "store_dynamic": false,
            "type_field": "_type"
          },
          "store": {
            "indexType": "scorch",
            "segmentVersion": 15
          }
        },
        "sourceParams": {}
      }'
      

      run a query on this index - 847 hits

      curl -XPOST -H "Content-Type: application/json" -u Administrator:password http://172.23.104.173:8094/api/index/default/query -d '{
       "from": 0,
       "size": 10,
       "query": {
        "location": {
         "lon": -2.235143,
         "lat": 53.482358
         },
         "distance": "100mi",
         "field": "geo"
        },
       "sort": [
        {
         "by": "geo_distance",
         "field": "geo",
         "unit": "mi",
         "location": {
         "lon": -2.235143,
         "lat": 53.482358
         }
        }
       ]
      }' | jq
      

      upgrade all nodes to 7.1.2-3386

      run the query again, all results are the same and fine - 847 hits

      update the index to have 1 replica

      run the query again, now its not returning any hits

      if i create a second index post upgrade that is equivalent to the first and query that index, i again receive 847 hits

      these are the steps detailed in MB-49199

      logs and indexes/query are below

      here is the index definition post upgrade and after it was updated

      {
        "type": "fulltext-index",
        "name": "geo-index",
        "uuid": "1bc164388a9c9bd8",
        "sourceType": "gocbcore",
        "sourceName": "travel-sample",
        "sourceUUID": "0fa39e1ea9ea451240772561dececa5d",
        "planParams": {
          "maxPartitionsPerPIndex": 512,
          "indexPartitions": 2,
          "numReplicas": 1
        },
        "params": {
          "doc_config": {
            "docid_prefix_delim": "",
            "docid_regexp": "",
            "mode": "type_field",
            "type_field": "type"
          },
          "mapping": {
            "analysis": {},
            "default_analyzer": "standard",
            "default_datetime_parser": "dateTimeOptional",
            "default_field": "_all",
            "default_mapping": {
              "dynamic": false,
              "enabled": true,
              "properties": {
                "geo": {
                  "dynamic": false,
                  "enabled": true,
                  "fields": [
                    {
                      "include_in_all": true,
                      "index": true,
                      "name": "geo",
                      "store": true,
                      "type": "geopoint"
                    }
                  ]
                }
              }
            },
            "default_type": "_default",
            "docvalues_dynamic": false,
            "index_dynamic": true,
            "store_dynamic": false,
            "type_field": "_type"
          },
          "store": {
            "indexType": "scorch",
            "segmentVersion": 15,
            "spatialPlugin": "s2"
          }
        },
        "sourceParams": {}
      }
      

      here is the query I ran on the index

      curl -XPOST -H "Content-Type: application/json" -u Administrator:password http://172.23.104.173:8094/api/index/geo-index/query -d '{
       "from": 0,
       "size": 10,
       "query": {
        "location": {
         "lon": -2.235143,
         "lat": 53.482358
         },
         "distance": "100mi",
         "field": "geo"
        },
       "sort": [
        {
         "by": "geo_distance",
         "field": "geo",
         "unit": "mi",
         "location": {
         "lon": -2.235143,
         "lat": 53.482358
         }
        }
       ]
      }' | jq
      

      here is the new index i made after the upgrade that is behaving as expected

      {
        "type": "fulltext-index",
        "name": "default",
        "uuid": "1da11aac0153b92e",
        "sourceType": "gocbcore",
        "sourceName": "travel-sample",
        "sourceUUID": "0fa39e1ea9ea451240772561dececa5d",
        "planParams": {
          "maxPartitionsPerPIndex": 512,
          "indexPartitions": 2
        },
        "params": {
          "doc_config": {
            "docid_prefix_delim": "",
            "docid_regexp": "",
            "mode": "type_field",
            "type_field": "type"
          },
          "mapping": {
            "analysis": {},
            "default_analyzer": "standard",
            "default_datetime_parser": "dateTimeOptional",
            "default_field": "_all",
            "default_mapping": {
              "dynamic": true,
              "enabled": true,
              "properties": {
                "geo": {
                  "dynamic": false,
                  "enabled": true,
                  "fields": [
                    {
                      "docvalues": true,
                      "include_in_all": true,
                      "index": true,
                      "name": "geo",
                      "store": true,
                      "type": "geopoint"
                    }
                  ]
                }
              }
            },
            "default_type": "_default",
            "docvalues_dynamic": false,
            "index_dynamic": true,
            "store_dynamic": false,
            "type_field": "_type"
          },
          "store": {
            "indexType": "scorch",
            "segmentVersion": 15,
            "spatialPlugin": "s2"
          }
        },
        "sourceParams": {}
      }
      

      results of query after index is updated

      {
        "status": {
          "total": 2,
          "failed": 0,
          "successful": 2
        },
        "request": {
          "query": {
            "location": [
              -2.235143,
              53.482358
            ],
            "distance": "100mi",
            "field": "geo"
          },
          "size": 10,
          "from": 0,
          "highlight": null,
          "fields": null,
          "facets": null,
          "explain": false,
          "sort": [
            {
              "by": "geo_distance",
              "field": "geo",
              "location": {
                "lat": 53.482358,
                "lon": -2.235143
              },
              "unit": "mi"
            }
          ],
          "includeLocations": false,
          "search_after": null,
          "search_before": null
        },
        "hits": [],
        "total_hits": 0,
        "max_score": 0,
        "took": 2476340,
        "facets": null
      }
      

      expected results of query

      {
        "status": {
          "total": 2,
          "failed": 0,
          "successful": 2
        },
        "request": {
          "query": {
            "location": [
              -2.235143,
              53.482358
            ],
            "distance": "100mi",
            "field": "geo"
          },
          "size": 10,
          "from": 0,
          "highlight": null,
          "fields": null,
          "facets": null,
          "explain": false,
          "sort": [
            {
              "by": "geo_distance",
              "field": "geo",
              "location": {
                "lat": 53.482358,
                "lon": -2.235143
              },
              "unit": "mi"
            }
          ],
          "includeLocations": false,
          "search_after": null,
          "search_before": null
        },
        "hits": [
          {
            "index": "default_1da11aac0153b92e_527bf675",
            "id": "landmark_17411",
            "score": 0.027011391898944172,
            "sort": [
              " \u0001?E#9>N\f\"e"
            ]
          },
          {
            "index": "default_1da11aac0153b92e_b6d0c5f9",
            "id": "landmark_17409",
            "score": 0.027290604112977332,
            "sort": [
              " \u0001?O~i*(kD,"
            ]
          },
          {
            "index": "default_1da11aac0153b92e_b6d0c5f9",
            "id": "landmark_17403",
            "score": 0.027290604112977332,
            "sort": [
              " \u0001?Sg*|/t\u001f\u0002"
            ]
          },
          {
            "index": "default_1da11aac0153b92e_b6d0c5f9",
            "id": "hotel_17413",
            "score": 0.027290604112977332,
            "sort": [
              " \u0001?U]S\\.e\u0002_"
            ]
          },
          {
            "index": "default_1da11aac0153b92e_527bf675",
            "id": "hotel_17414",
            "score": 0.027011391898944172,
            "sort": [
              " \u0001?Z\u0000./\u0007Q\u0012\t"
            ]
          },
          {
            "index": "default_1da11aac0153b92e_b6d0c5f9",
            "id": "landmark_17410",
            "score": 0.027290604112977332,
            "sort": [
              " \u0001?Z3T6 \u0010\u0019@"
            ]
          },
          {
            "index": "default_1da11aac0153b92e_527bf675",
            "id": "landmark_17412",
            "score": 0.027011391898944172,
            "sort": [
              " \u0001?]-\u000fm?\u000b\u0014#"
            ]
          },
          {
            "index": "default_1da11aac0153b92e_527bf675",
            "id": "landmark_17408",
            "score": 0.027011391898944172,
            "sort": [
              " \u0001?^DV7\u0014t:^"
            ]
          },
          {
            "index": "default_1da11aac0153b92e_b6d0c5f9",
            "id": "landmark_17406",
            "score": 0.027290604112977332,
            "sort": [
              " \u0001?_<\u00009\u001eW\u0013\u0012"
            ]
          },
          {
            "index": "default_1da11aac0153b92e_527bf675",
            "id": "landmark_17397",
            "score": 0.027011391898944172,
            "sort": [
              " \u0001?c\u001cx\u0010n\u0016Wl"
            ]
          }
        ],
        "total_hits": 847,
        "max_score": 0.1923674628915241,
        "took": 8705322,
        "facets": null
      }

      Attachments

        For Gerrit Dashboard: MB-53050
        # Subject Branch Project Status CR V

        Activity

          Girish Benakappa , can you please approve this ticket for 7.1.2?

          Sreekanth Sivasankaran Sreekanth Sivasankaran added a comment - Girish Benakappa , can you please approve this ticket for 7.1.2?

          QE - Approved
          We should be able to validate the fix for this issue

          girish.benakappa Girish Benakappa added a comment - QE - Approved We should be able to validate the fix for this issue

          Build couchbase-server-7.1.2-3389 contains cbft commit e2a3dc5 with commit message:
          MB-53050 - [FTS] index after upgrade returns wrong results

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.2-3389 contains cbft commit e2a3dc5 with commit message: MB-53050 - [FTS] index after upgrade returns wrong results

          Build couchbase-server-7.2.0-1677 contains cbft commit e2a3dc5 with commit message:
          MB-53050 - [FTS] index after upgrade returns wrong results

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-1677 contains cbft commit e2a3dc5 with commit message: MB-53050 - [FTS] index after upgrade returns wrong results

          Build couchbase-server-8.0.0-1061 contains cbft commit e2a3dc5 with commit message:
          MB-53050 - [FTS] index after upgrade returns wrong results

          build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1061 contains cbft commit e2a3dc5 with commit message: MB-53050 - [FTS] index after upgrade returns wrong results

          verified in 7.1.2-3390 using the repro in the description

          now after the index is updated post-upgrade the result hits are fine

          ajay.bhullar Ajay Bhullar added a comment - verified in 7.1.2-3390 using the repro in the description now after the index is updated post-upgrade the result hits are fine

          As per Ajay's comment closing the ticket

          chanabasappa.ghali Chanabasappa Ghali added a comment - As per Ajay's comment closing the ticket

          People

            ajay.bhullar Ajay Bhullar
            ajay.bhullar Ajay Bhullar
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty