Uploaded image for project: 'Couchbase Lite'
  1. Couchbase Lite
  2. CBL-324

Function.upper() failed to operate non-ASCII characters

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.6.0
    • 2.7.0
    • LiteCore
    • Security Level: Public
    • CBL Sprint 3
    • 3

    Description

      CBL VERSION:  → 2.6.0-154

      CBS VERSION: 6.0.1-2037

      SGW VERSION: 2.6.0-127

      Steps to Reproduce: 

      1. install cbs server and enable travel-sample bucket
      2. install sgw and connect to cbs server
      3. have an Android app reference CBLite 2.6 libraries, (in my case, use TestServer Android app), and run this statement:
        let searchQuery = Query
        .select(SelectResult.expression(Expression.meta().id),
        SelectResult.expression(Expression.property("email")),
        SelectResult.expression(Function.upper(Expression.property("name"))))
        .from(DataSource.database(db))
        .where(Expression.property("email").and(Function.contains(Expression.property("email"),
        substring: "maisondeverlaine@gmail.com")))

      Actual Result:

      {'$1': 'LA MAISON DE VERLAINE *à* METZ', 'email': 'maisondeverlaine@gmail.com', 'id': 'landmark_17957'}

      Expected Result:

      {'$1': 'LA MAISON DE VERLAINE *À* METZ', 'email': 'maisondeverlaine@gmail.com', 'id': 'landmark_17957'}

       

      non-ascii character à didn't not converted to À, This behavior is only seen on Android, NOT on iOS or .net platforms

      Logs : attached

      Jenkins job Failure  link:

      *http://uberjenkins.sc.couchbase.com:8080/job/Android-Listener-TestServer-Query-tests/336/*

      Pytest command:

      pytest -s -rsx --timeout=1800 --liteserv-version=2.6.0-154 --liteserv-host=192.168.33.126 --liteserv-port=8080 --xattrs --sync-gateway-version=2.6.0-127 --mode=cc --server-version=6.0.1-2037 --liteserv-platform=android --create-db-per-suite=cbl-test --enable-sample-bucket=travel-sample /Users/eunicehuang/dev/mobile-testkit/testsuites/CBLTester/CBL_Functional_tests/SuiteSetup_FunctionalTests/  -k test_query_substring

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          blake.meike Blake Meike added a comment -

          Jens Alfke : do you have any insight into this.  It looks, at first face, as if the SQLite UPPER function is behaving differently on different platforms?

          blake.meike Blake Meike added a comment - Jens Alfke : do you have any insight into this.  It looks, at first face, as if the SQLite UPPER function is behaving differently on different platforms?

          if query couchbase server through admin UI, you can use this query:

          select meta().id, name, email from `travel-sample` t where email = 'maisondeverlaine@gmail.com'

          eunice.huang Eunice Huang (Inactive) added a comment - if query couchbase server through admin UI, you can use this query: select meta().id, name, email from `travel-sample` t where email = 'maisondeverlaine@gmail.com'
          jimb Jim Borden added a comment -

          Yes, the upper implementation is per "platform" (Win32, CoreFoundation, ICU).

          jimb Jim Borden added a comment - Yes, the upper implementation is per "platform" (Win32, CoreFoundation, ICU).
          jimb Jim Borden added a comment - - edited

          Are you sure this passed in 2.5? Not a single thing has changed for this area and I would guess it's been around for a while.

          EDIT: Sorry I missed the logs that show the pass and fail!

          jimb Jim Borden added a comment - - edited Are you sure this passed in 2.5? Not a single thing has changed for this area and I would guess it's been around for a while. EDIT: Sorry I missed the logs that show the pass and fail!
          jimb Jim Borden added a comment -

          If this is a failure of LiteCore, this can probably be reproduced in a LiteCore test on Linux since it uses the same string utility engine.

          jimb Jim Borden added a comment - If this is a failure of LiteCore, this can probably be reproduced in a LiteCore test on Linux since it uses the same string utility engine.
          jens Jens Alfke added a comment -

          Shouldn't this isn't be caught by LiteCore unit tests? Do we not have any that test the UPPER() and LOWER() functions?

          jens Jens Alfke added a comment - Shouldn't this isn't be caught by LiteCore unit tests? Do we not have any that test the UPPER() and LOWER() functions?

          non-ascii stuff is always a problem on Android. We'll doc this for Cobalt and see if there is anything to do for this in Mercury.

          daniel.petersen Daniel Petersen added a comment - non-ascii stuff is always a problem on Android. We'll doc this for Cobalt and see if there is anything to do for this in Mercury.
          Jayahari.Vavachan Jay Vavachan added a comment - - edited

          initial findings

          according to this, there is no way to get locale from Android NDK.

          Jayahari.Vavachan Jay Vavachan added a comment - - edited initial findings according to this , there is no way to get locale from Android NDK.
          jens Jens Alfke added a comment -

          I don't think you need a locale to do case transformation. There might be a few language-specific rules, but mostly they're global.

          I just checked and neither the Cocoa nor the ICU-based implementations of UTF8ChangeCase() in LiteCore use locales. It looks like the APIs they call could allow a locale to be set, but they work without one.

          jens Jens Alfke added a comment - I don't think you need a locale to do case transformation. There might be a few language-specific rules, but mostly they're global. I just checked and neither the Cocoa nor the ICU-based implementations of UTF8ChangeCase() in LiteCore use locales. It looks like the APIs they call could allow a locale to be set, but they work without one.
          Jayahari.Vavachan Jay Vavachan added a comment -

          Seems like `LITECORE_USES_ICU` is not set to true, in case of android.

          It started working fine, when i did the below items,

          • if we pass the `LITECORE_USES_ICU`argument, and
          • set the library path to include LiteCore/Android/
          Jayahari.Vavachan Jay Vavachan added a comment - Seems like `LITECORE_USES_ICU` is not set to true, in case of android. It started working fine, when i did the below items, if we pass the `LITECORE_USES_ICU`argument, and set the library path to include LiteCore/Android/
          jimb Jim Borden added a comment -

          It gets set here for Android.

          jimb Jim Borden added a comment - It gets set here for Android.
          Jayahari.Vavachan Jay Vavachan added a comment -

          Support source was not build with `LITECORE_USES_ICU` argument, which was the reason, why the UTF8ChangeCase was going to the default implementation than the StringUtil_icu.

          Now the fix is added in this PR

          Jayahari.Vavachan Jay Vavachan added a comment - Support source was not build with `LITECORE_USES_ICU` argument, which was the reason, why the UTF8ChangeCase was going to the default implementation than the StringUtil_icu. Now the fix is added in this PR
          Jayahari.Vavachan Jay Vavachan added a comment -

           It gets set here for Android.

          this is only setting for building lite core, but not for support files build.

          Jayahari.Vavachan Jay Vavachan added a comment -  It gets set here for Android. this is only setting for building lite core, but not for support files build.

          Build couchbase-lite-core-2.7.0-65 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.7.0-65 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-android-2.7.0-115 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-android-2.7.0-115 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-log-2.6.0-189 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.6.0-189 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-cblite-2.6.0-154 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-cblite-2.6.0-154 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-ios-2.7.0-35 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-ios-2.7.0-35 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          weekly run with android 2.7.0-117 has been verified, the issue has been fixed.

          eunice.huang Eunice Huang (Inactive) added a comment - weekly run with android 2.7.0-117 has been verified, the issue has been fixed.

          Build couchbase-lite-net-2.7.0-45 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-net-2.7.0-45 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-android-2.6.2-3 contains couchbase-lite-core commit 7c33e8f with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-android-2.6.2-3 contains couchbase-lite-core commit 7c33e8f with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-log-2.6.1-7 contains couchbase-lite-core commit 7c33e8f with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.6.1-7 contains couchbase-lite-core commit 7c33e8f with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-ios-2.6.2-3 contains couchbase-lite-core commit 7c33e8f with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-ios-2.6.2-3 contains couchbase-lite-core commit 7c33e8f with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-cblite-2.6.0-213 contains couchbase-lite-core commit 7c33e8f with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-cblite-2.6.0-213 contains couchbase-lite-core commit 7c33e8f with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-net-2.6.2-2 contains couchbase-lite-core commit 7c33e8f with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-net-2.6.2-2 contains couchbase-lite-core commit 7c33e8f with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-log-2.7.0-79 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.7.0-79 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          Build couchbase-lite-cblite-2.7.0-5 contains couchbase-lite-core commit 28affdf with commit message:
          CBL-324: fix: support source build (#847)

          build-team Couchbase Build Team added a comment - Build couchbase-lite-cblite-2.7.0-5 contains couchbase-lite-core commit 28affdf with commit message: CBL-324 : fix: support source build (#847)

          People

            eunice.huang Eunice Huang (Inactive)
            eunice.huang Eunice Huang (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty