Uploaded image for project: 'Couchbase Lite'
  1. Couchbase Lite
  2. CBL-324

Function.upper() failed to operate non-ASCII characters

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.6.0
    • 2.7.0
    • LiteCore
    • Security Level: Public
    • CBL Sprint 3
    • 3

    Description

      CBL VERSION:  → 2.6.0-154

      CBS VERSION: 6.0.1-2037

      SGW VERSION: 2.6.0-127

      Steps to Reproduce: 

      1. install cbs server and enable travel-sample bucket
      2. install sgw and connect to cbs server
      3. have an Android app reference CBLite 2.6 libraries, (in my case, use TestServer Android app), and run this statement:
        let searchQuery = Query
        .select(SelectResult.expression(Expression.meta().id),
        SelectResult.expression(Expression.property("email")),
        SelectResult.expression(Function.upper(Expression.property("name"))))
        .from(DataSource.database(db))
        .where(Expression.property("email").and(Function.contains(Expression.property("email"),
        substring: "maisondeverlaine@gmail.com")))

      Actual Result:

      {'$1': 'LA MAISON DE VERLAINE *à* METZ', 'email': 'maisondeverlaine@gmail.com', 'id': 'landmark_17957'}

      Expected Result:

      {'$1': 'LA MAISON DE VERLAINE *À* METZ', 'email': 'maisondeverlaine@gmail.com', 'id': 'landmark_17957'}

       

      non-ascii character à didn't not converted to À, This behavior is only seen on Android, NOT on iOS or .net platforms

      Logs : attached

      Jenkins job Failure  link:

      *http://uberjenkins.sc.couchbase.com:8080/job/Android-Listener-TestServer-Query-tests/336/*

      Pytest command:

      pytest -s -rsx --timeout=1800 --liteserv-version=2.6.0-154 --liteserv-host=192.168.33.126 --liteserv-port=8080 --xattrs --sync-gateway-version=2.6.0-127 --mode=cc --server-version=6.0.1-2037 --liteserv-platform=android --create-db-per-suite=cbl-test --enable-sample-bucket=travel-sample /Users/eunicehuang/dev/mobile-testkit/testsuites/CBLTester/CBL_Functional_tests/SuiteSetup_FunctionalTests/  -k test_query_substring

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          eunice.huang Eunice Huang (Inactive) created issue -
          blake.meike Blake Meike added a comment -

          Jens Alfke : do you have any insight into this.  It looks, at first face, as if the SQLite UPPER function is behaving differently on different platforms?

          blake.meike Blake Meike added a comment - Jens Alfke : do you have any insight into this.  It looks, at first face, as if the SQLite UPPER function is behaving differently on different platforms?

          if query couchbase server through admin UI, you can use this query:

          select meta().id, name, email from `travel-sample` t where email = 'maisondeverlaine@gmail.com'

          eunice.huang Eunice Huang (Inactive) added a comment - if query couchbase server through admin UI, you can use this query: select meta().id, name, email from `travel-sample` t where email = 'maisondeverlaine@gmail.com'
          eunice.huang Eunice Huang (Inactive) made changes -
          Field Original Value New Value
          Attachment function.upper.android-2.5.3.txt [ 72498 ]
          Attachment function.upper.android-2.6.txt [ 72499 ]
          jimb Jim Borden added a comment -

          Yes, the upper implementation is per "platform" (Win32, CoreFoundation, ICU).

          jimb Jim Borden added a comment - Yes, the upper implementation is per "platform" (Win32, CoreFoundation, ICU).
          jimb Jim Borden added a comment - - edited

          Are you sure this passed in 2.5? Not a single thing has changed for this area and I would guess it's been around for a while.

          EDIT: Sorry I missed the logs that show the pass and fail!

          jimb Jim Borden added a comment - - edited Are you sure this passed in 2.5? Not a single thing has changed for this area and I would guess it's been around for a while. EDIT: Sorry I missed the logs that show the pass and fail!
          jimb Jim Borden made changes -
          Component/s LiteCore [ 15347 ]
          Component/s Java-Android [ 15344 ]
          jimb Jim Borden made changes -
          Labels android
          jimb Jim Borden added a comment -

          If this is a failure of LiteCore, this can probably be reproduced in a LiteCore test on Linux since it uses the same string utility engine.

          jimb Jim Borden added a comment - If this is a failure of LiteCore, this can probably be reproduced in a LiteCore test on Linux since it uses the same string utility engine.
          jens Jens Alfke added a comment -

          Shouldn't this isn't be caught by LiteCore unit tests? Do we not have any that test the UPPER() and LOWER() functions?

          jens Jens Alfke added a comment - Shouldn't this isn't be caught by LiteCore unit tests? Do we not have any that test the UPPER() and LOWER() functions?
          eunice.huang Eunice Huang (Inactive) made changes -
          Attachment network down x-android replicator stopped.txt [ 72583 ]
          eunice.huang Eunice Huang (Inactive) made changes -
          Attachment network down x-android replicator stopped.txt [ 72583 ]
          daniel.petersen Daniel Petersen made changes -
          Component/s Java-Android [ 15344 ]
          Component/s LiteCore [ 15347 ]
          daniel.petersen Daniel Petersen made changes -
          Labels android known_issue

          non-ascii stuff is always a problem on Android. We'll doc this for Cobalt and see if there is anything to do for this in Mercury.

          daniel.petersen Daniel Petersen added a comment - non-ascii stuff is always a problem on Android. We'll doc this for Cobalt and see if there is anything to do for this in Mercury.
          daniel.petersen Daniel Petersen made changes -
          Fix Version/s Mercury [ 16188 ]
          jimb Jim Borden made changes -
          Component/s LiteCore [ 15347 ]
          Component/s Java-Android [ 15344 ]
          jimb Jim Borden made changes -
          Labels known_issue android known_issue
          jamiltz James Nocentini made changes -
          Security Private [ 10332 ] Public [ 10333 ]
          daniel.petersen Daniel Petersen made changes -
          Story Points 3
          daniel.petersen Daniel Petersen made changes -
          Assignee The Lite [ cbgto ] Jayahari Vavachan [ jayahari.vavachan ]
          daniel.petersen Daniel Petersen made changes -
          Sprint CBL Sprint 3 [ 891 ]
          daniel.petersen Daniel Petersen made changes -
          Rank Ranked higher
          Jayahari.Vavachan Jay Vavachan added a comment - - edited

          initial findings

          according to this, there is no way to get locale from Android NDK.

          Jayahari.Vavachan Jay Vavachan added a comment - - edited initial findings according to this , there is no way to get locale from Android NDK.
          Jayahari.Vavachan Jay Vavachan made changes -
          Rank Ranked higher
          jens Jens Alfke added a comment -

          I don't think you need a locale to do case transformation. There might be a few language-specific rules, but mostly they're global.

          I just checked and neither the Cocoa nor the ICU-based implementations of UTF8ChangeCase() in LiteCore use locales. It looks like the APIs they call could allow a locale to be set, but they work without one.

          jens Jens Alfke added a comment - I don't think you need a locale to do case transformation. There might be a few language-specific rules, but mostly they're global. I just checked and neither the Cocoa nor the ICU-based implementations of UTF8ChangeCase() in LiteCore use locales. It looks like the APIs they call could allow a locale to be set, but they work without one.
          Jayahari.Vavachan Jay Vavachan added a comment -

          Seems like `LITECORE_USES_ICU` is not set to true, in case of android.

          It started working fine, when i did the below items,

          • if we pass the `LITECORE_USES_ICU`argument, and
          • set the library path to include LiteCore/Android/
          Jayahari.Vavachan Jay Vavachan added a comment - Seems like `LITECORE_USES_ICU` is not set to true, in case of android. It started working fine, when i did the below items, if we pass the `LITECORE_USES_ICU`argument, and set the library path to include LiteCore/Android/
          jimb Jim Borden added a comment -

          It gets set here for Android.

          jimb Jim Borden added a comment - It gets set here for Android.
          Jayahari.Vavachan Jay Vavachan added a comment -

          Support source was not build with `LITECORE_USES_ICU` argument, which was the reason, why the UTF8ChangeCase was going to the default implementation than the StringUtil_icu.

          Now the fix is added in this PR

          Jayahari.Vavachan Jay Vavachan added a comment - Support source was not build with `LITECORE_USES_ICU` argument, which was the reason, why the UTF8ChangeCase was going to the default implementation than the StringUtil_icu. Now the fix is added in this PR
          Jayahari.Vavachan Jay Vavachan made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Jayahari.Vavachan Jay Vavachan made changes -
          Status In Progress [ 3 ] In Review [ 10107 ]
          Jayahari.Vavachan Jay Vavachan made changes -
          Status In Review [ 10107 ] In Progress [ 3 ]
          Jayahari.Vavachan Jay Vavachan made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          Jayahari.Vavachan Jay Vavachan made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Jayahari.Vavachan Jay Vavachan made changes -
          Assignee Jayahari Vavachan [ jayahari.vavachan ] Eunice Huang [ eunice.huang ]
          Jayahari.Vavachan Jay Vavachan added a comment -

           It gets set here for Android.

          this is only setting for building lite core, but not for support files build.

          Jayahari.Vavachan Jay Vavachan added a comment -  It gets set here for Android. this is only setting for building lite core, but not for support files build.

          Build couchbase-lite-core-2.7.0-65 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.7.0-65 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-android-2.7.0-115 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-android-2.7.0-115 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-log-2.6.0-189 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.6.0-189 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-cblite-2.6.0-154 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-cblite-2.6.0-154 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-ios-2.7.0-35 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-ios-2.7.0-35 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          weekly run with android 2.7.0-117 has been verified, the issue has been fixed.

          eunice.huang Eunice Huang (Inactive) added a comment - weekly run with android 2.7.0-117 has been verified, the issue has been fixed.
          eunice.huang Eunice Huang (Inactive) made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          jamiltz James Nocentini made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          jamiltz James Nocentini made changes -
          Labels android known_issue android
          jamiltz James Nocentini made changes -
          Resolution Fixed [ 1 ]
          Status Reopened [ 4 ] Resolved [ 5 ]
          jamiltz James Nocentini made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          jamiltz James Nocentini made changes -
          Resolution Fixed [ 1 ]
          Status Reopened [ 4 ] Resolved [ 5 ]
          jamiltz James Nocentini made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          Build couchbase-lite-net-2.7.0-45 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-net-2.7.0-45 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)
          daniel.petersen Daniel Petersen made changes -
          Link This issue blocks CBL-397 [ CBL-397 ]

          Build couchbase-lite-android-2.6.2-3 contains couchbase-lite-core commit 7c33e8f with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-android-2.6.2-3 contains couchbase-lite-core commit 7c33e8f with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-log-2.6.1-7 contains couchbase-lite-core commit 7c33e8f with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.6.1-7 contains couchbase-lite-core commit 7c33e8f with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-ios-2.6.2-3 contains couchbase-lite-core commit 7c33e8f with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-ios-2.6.2-3 contains couchbase-lite-core commit 7c33e8f with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-cblite-2.6.0-213 contains couchbase-lite-core commit 7c33e8f with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-cblite-2.6.0-213 contains couchbase-lite-core commit 7c33e8f with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-net-2.6.2-2 contains couchbase-lite-core commit 7c33e8f with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-net-2.6.2-2 contains couchbase-lite-core commit 7c33e8f with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-log-2.7.0-79 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.7.0-79 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-cblite-2.7.0-5 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-cblite-2.7.0-5 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          People

            eunice.huang Eunice Huang (Inactive)
            eunice.huang Eunice Huang (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty