Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61723

Adding an extra " to base64 encoding results in exact match instead of illegal base64

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • 7.6.2
    • fts
    • Untriaged
    • 0
    • Unknown

    Description

      Having a document in xattrs like

      {
        "meta": {
          "id": "vect1",
          "rev": "44-17cb4209e21c00000000000002000000",
          "expiration": 0,
          "flags": 33554432,
          "type": "json"
        },
        "xattrs": {
          "x": "AAAAAAAAgEEAAAxCAACgQAAAAEIAAPhBAABgQQAAIEEAADBBAACcQgAAXEIAACBBAAA0QgAApkIAADBBAADAQAAAYEEAAGRCAADMQgAAlkIAAKBBAAAAQQAAQEAAAKBAAACGQgAAiEEAAJhBAADQQQAAoEAAAAAAAACAPwAAsEEAAHBCAADQQQAA4EAAAIA/AACQQQAAsEEAAKhCAABUQgAAqkIAAO5CAADuQgAAgEAAAMBBAACQQQAA4EAAAOBAAACAPwAAokIAANRCAADMQgAAkEIAAPBBAADAQAAAAAAAABBBAACAPwAAEEEAAO5CAACQQgAAgD8AAIBAAAAEQgAA7kIAAOhBAADAQAAAgD8AAAAAAACAPwAAYEEAAFBCAADuQgAA8EEAAEBAAAAAAAAAAAAAAFxCAAC4QgAA3kIAAABAAACgQAAAgEAAABBBAACwQQAAskIAAMBCAABgQQAAgD8AAAAAAACAPwAApEIAAGxCAACAQQAAoEEAAKBAAADIQQAAYEEAADBBAACAQAAAAAAAAAAAAACAPwAA0EEAADxCAAC4QQAAgEAAAAAAAAAAAAAAgEAAABhCAACmQgAA8EEAAGBBAAAQQQAAgEAAABBBAACIQQAAuEEAACRCAAAAAAAAAAAAAABAAAAAQQAAmEEAAMhBAAC4QQAAgD8="
        }
      } 

      Indexed this x field, Index Def:

      {
        "type": "fulltext-index",
        "name": "i1",
        "uuid": "4250e1df971f24a6",
        "sourceType": "gocbcore",
        "sourceName": "b1",
        "sourceUUID": "2b6318dcba775ee5adc05a159a93797d",
        "planParams": {
          "maxPartitionsPerPIndex": 1024,
          "indexPartitions": 1
        },
        "params": {
          "doc_config": {
            "docid_prefix_delim": "",
            "docid_regexp": "",
            "mode": "scope.collection.type_field",
            "type_field": "type"
          },
          "mapping": {
            "analysis": {},
            "default_analyzer": "standard",
            "default_datetime_parser": "dateTimeOptional",
            "default_field": "_all",
            "default_mapping": {
              "dynamic": false,
              "enabled": false
            },
            "default_type": "_default",
            "docvalues_dynamic": true,
            "index_dynamic": true,
            "store_dynamic": false,
            "type_field": "_type",
            "types": {
              "s1.c1": {
                "default_analyzer": "standard",
                "dynamic": true,
                "enabled": true,
                "properties": {
                  "_$xattrs": {
                    "dynamic": true,
                    "enabled": true,
                    "properties": {
                      "x": {
                        "dynamic": false,
                        "enabled": true,
                        "fields": [
                          {
                            "dims": 128,
                            "include_in_all": true,
                            "include_term_vectors": true,
                            "index": true,
                            "name": "x",
                            "similarity": "l2_norm",
                            "type": "vector_base64",
                            "vector_index_optimized_for": "recall"
                          }
                        ]
                      }
                    }
                  },
                  "sno": {
                    "dynamic": false,
                    "enabled": true,
                    "fields": [
                      {
                        "include_in_all": true,
                        "include_term_vectors": true,
                        "index": true,
                        "name": "sno",
                        "store": true,
                        "type": "number"
                      }
                    ]
                  }
                }
              }
            }
          },
          "store": {
            "indexType": "scorch",
            "mossStoreOptions": {},
            "segmentVersion": 16
          }
        },
        "sourceParams": {
          "includeXAttrs": true
        }
      } 

      On performing KNN search for the document that already exists (mentioned above)I am getting correct hits

      {     "query": {         "match_none": {}     },     "explain": true,     "knn": [         {             "field": "_$xattrs.x",             "k": 2,             "vector_base64":"AAAAAAAAgEEAAAxCAACgQAAAAEIAAPhBAABgQQAAIEEAADBBAACcQgAAXEIAACBBAAA0QgAApkIAADBBAADAQAAAYEEAAGRCAADMQgAAlkIAAKBBAAAAQQAAQEAAAKBAAACGQgAAiEEAAJhBAADQQQAAoEAAAAAAAACAPwAAsEEAAHBCAADQQQAA4EAAAIA/AACQQQAAsEEAAKhCAABUQgAAqkIAAO5CAADuQgAAgEAAAMBBAACQQQAA4EAAAOBAAACAPwAAokIAANRCAADMQgAAkEIAAPBBAADAQAAAAAAAABBBAACAPwAAEEEAAO5CAACQQgAAgD8AAIBAAAAEQgAA7kIAAOhBAADAQAAAgD8AAAAAAACAPwAAYEEAAFBCAADuQgAA8EEAAEBAAAAAAAAAAAAAAFxCAAC4QgAA3kIAAABAAACgQAAAgEAAABBBAACwQQAAskIAAMBCAABgQQAAgD8AAAAAAACAPwAApEIAAGxCAACAQQAAoEEAAKBAAADIQQAAYEEAADBBAACAQAAAAAAAAAAAAACAPwAA0EEAADxCAAC4QQAAgEAAAAAAAAAAAAAAgEAAABhCAACmQgAA8EEAAGBBAAAQQQAAgEAAABBBAACIQQAAuEEAACRCAAAAAAAAAAAAAABAAAAAQQAAmEEAAMhBAAC4QQAAgD8="         }     ] }

      But to the same query if I add an extra " before vector encoded base 64 I started seeing exact match query result

      {     "query": {         "match_none": {}     },     "explain": true,     "knn": [         {             "field": "_$xattrs.x",             "k": 2,             "vector_base64":"AAAAAAAAgEEAAAxCAACgQAAAAEIAAPhBAABgQQAAIEEAADBBAACcQgAAXEIAACBBAAA0QgAApkIAADBBAADAQAAAYEEAAGRCAADMQgAAlkIAAKBBAAAAQQAAQEAAAKBAAACGQgAAiEEAAJhBAADQQQAAoEAAAAAAAACAPwAAsEEAAHBCAADQQQAA4EAAAIA/AACQQQAAsEEAAKhCAABUQgAAqkIAAO5CAADuQgAAgEAAAMBBAACQQQAA4EAAAOBAAACAPwAAokIAANRCAADMQgAAkEIAAPBBAADAQAAAAAAAABBBAACAPwAAEEEAAO5CAACQQgAAgD8AAIBAAAAEQgAA7kIAAOhBAADAQAAAgD8AAAAAAACAPwAAYEEAAFBCAADuQgAA8EEAAEBAAAAAAAAAAAAAAFxCAAC4QgAA3kIAAABAAACgQAAAgEAAABBBAACwQQAAskIAAMBCAABgQQAAgD8AAAAAAACAPwAApEIAAGxCAACAQQAAoEEAAKBAAADIQQAAYEEAADBBAACAQAAAAAAAAAAAAACAPwAA0EEAADxCAAC4QQAAgEAAAAAAAAAAAAAAgEAAABhCAACmQgAA8EEAAGBBAAAQQQAAgEAAABBBAACIQQAAuEEAACRCAAAAAAAAAAAAAABAAAAAQQAAmEEAAMhBAAC4QQAAgD8="         }     ] } 

      Seems like a UI parsing issue because this is anyways an invalid JSON. But should have been getting an error instead of direct match, or possibly k results instead of 1.

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sarthak.dua Sarthak Dua
            sarthak.dua Sarthak Dua
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty