Uploaded image for project: 'Couchbase Python Client Library'
  1. Couchbase Python Client Library
  2. PYCBC-869

Cattrs library doesn't work with python 3.5.2

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.0.0-rc
    • 3.0.0
    • None
    • None
    • 1
    • SDK13: Coll/Txn/Docs More Chip

    Description

      Currently, our performance testing framework uses python 3.5.2. I was trying to test out the latest python sdk 3.0.0 in our perf framework to see if there are any issues. I ran pip install git+https://github.com/couchbase/couchbase-python-client.git as recommended. I then tried to run 

      from couchbase.exceptions import CouchbaseError

      I did this on two setups: osx with python 3.5.6 and ubuntu with python 3.5.2. In both cases it throws import error:

      Ubuntu -

      Python 3.5.2 (default, Oct  8 2019, 13:06:37) 
      [GCC 5.4.0 20160609] on linux
      Type "help", "copyright", "credits" or "license" for more information.
      >>> from couchbase.exceptions import CouchbaseError
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "/tmp/perfrunner/perfrunner/env/lib/python3.5/site-packages/couchbase/__init__.py", line 3, in <module>
          from .bucket import *
        File "/tmp/perfrunner/perfrunner/env/lib/python3.5/site-packages/couchbase/bucket.py", line 1, in <module>
          from couchbase.management import CollectionManager, ViewIndexManager
        File "/tmp/perfrunner/perfrunner/env/lib/python3.5/site-packages/couchbase/management/__init__.py", line 1, in <module>
          from .collections import CollectionManager
        File "/tmp/perfrunner/perfrunner/env/lib/python3.5/site-packages/couchbase/management/collections.py", line 1, in <module>
          from ..options import OptionBlockTimeOut, forward_args
        File "/tmp/perfrunner/perfrunner/env/lib/python3.5/site-packages/couchbase/options.py", line 14, in <module>
          import cattr
        File "/tmp/perfrunner/perfrunner/env/lib/python3.5/site-packages/cattr/__init__.py", line 2, in <module>
          from .converters import Converter, UnstructureStrategy
        File "/tmp/perfrunner/perfrunner/env/lib/python3.5/site-packages/cattr/converters.py", line 15, in <module>
          from ._compat import (
        File "/tmp/perfrunner/perfrunner/env/lib/python3.5/site-packages/cattr/_compat.py", line 87, in <module>
          from typing import _Union
      ImportError: cannot import name '_Union'
      

      OSX - 

      Python 3.5.6 (default, May 13 2019, 15:48:34) 
      [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
      Type "help", "copyright", "credits" or "license" for more information.
      >>> from couchbase.exceptions import CouchbaseError
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      ImportError: cannot import name 'CouchbaseError'
      

      Not sure why osx python didnt provide stacktrace.

      Certain version of python do not have  typing._Union and this is a known issue with Cattrs:

      https://github.com/Tinche/cattrs/issues/46

      https://github.com/Tinche/cattrs/issues/77

      Cattrs incompatibility is not just for the version I mentioned, 3.5.2 and 3.5.6, but others as well. It seems there are possible work arounds you can implement. Until this is fixed perf team cannot run any tests on latests python sdk. Alternative workaround for perf team would be to upgrade our python version however this is a large task and we would prefer if this is fixed in the sdk.

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          for reference, it looks like cattrs was added to requirements.txt recently in this commit: https://github.com/couchbase/couchbase-python-client/commit/b36c711d25ae9558a560b2705955ed48a25b790a

          korrigan.clark Korrigan Clark (Inactive) added a comment - for reference, it looks like cattrs was added to requirements.txt recently in this commit:  https://github.com/couchbase/couchbase-python-client/commit/b36c711d25ae9558a560b2705955ed48a25b790a

          Where are you getting your python distribution from? There is a bug in Python that indicates building it from source can cause issues similar to this. Perhaps you're seeing this from a cbdeps based Python build? It looks like it's fixed, but you're also using a very old version (a quick search indicates 3.5.2 is from June, 2016).

          Korrigan Clark: since import of _Union to be a general problem in earlier releases of Python 3.5. Can you at least try Python 3.5.9? That shouldn't be arduous to perf tests, I wouldn't think. Also note that 3.5.6 was released in November of 2018.

          ingenthr Matt Ingenthron added a comment - Where are you getting your python distribution from? There is a bug in Python that indicates building it from source can cause issues similar to this. Perhaps you're seeing this from a cbdeps based Python build? It looks like it's fixed, but you're also using a very old version (a quick search indicates 3.5.2 is from June, 2016). Korrigan Clark : since import of _Union to be a general problem in earlier releases of Python 3.5. Can you at least try Python 3.5.9? That shouldn't be arduous to perf tests, I wouldn't think. Also note that 3.5.6 was released in November of 2018.
          ingenthr Matt Ingenthron added a comment - - edited

          Oh, and a further search (one link behind what you posted on cattrs) indicates that 3.5.3 does work: https://github.com/apache/airflow/issues/8162

          Maybe the issue you're seeing on 3.5.6 is actually different. Are you using pyenv there? It's my understanding that it's generally not recommended to use OS X's built in Python since it doesn't hew too closely to most other distributions and is hard to update. That's at least why I've used pyenv.

          All of that said, it does look like cattrs is a little problematic, as it also has issues with Python 3.8 which remain open. Worth evaluating if we really need a new dependency Ellis Breen.

          ingenthr Matt Ingenthron added a comment - - edited Oh, and a further search (one link behind what you posted on cattrs) indicates that 3.5.3 does work: https://github.com/apache/airflow/issues/8162 Maybe the issue you're seeing on 3.5.6 is actually different. Are you using pyenv there? It's my understanding that it's generally not recommended to use OS X's built in Python since it doesn't hew too closely to most other distributions and is hard to update. That's at least why I've used pyenv. All of that said, it does look like cattrs is a little problematic, as it also has issues with Python 3.8 which remain open. Worth evaluating if we really need a new dependency Ellis Breen .

          Matt Ingenthron I was using virtualenv with pyenv installed python3.5.6 and for ubuntu its the apt-get installed python3.5.2. I can run a test locally to see if I can find a pyenv python version that works with _Union. Making any python version changes in perfrunner is a bit of work having to install a new python on all of our perf client servers, making sure nothing breaks in the framework, and checking all tests run without change to the performance numbers reported. If it is truly necessary to upgrade our perf python version, I will have to talk to Wayne Siu and the team and get this work scheduled. The whole process will take several days in best case scenario, so if we want to run perf test with sdk3-rc before GA we will need to move soon.

          korrigan.clark Korrigan Clark (Inactive) added a comment - Matt Ingenthron  I was using virtualenv with pyenv installed python3.5.6 and for ubuntu its the apt-get installed python3.5.2. I can run a test locally to see if I can find a pyenv python version that works with _Union. Making any python version changes in perfrunner is a bit of work having to install a new python on all of our perf client servers, making sure nothing breaks in the framework, and checking all tests run without change to the performance numbers reported. If it is truly necessary to upgrade our perf python version, I will have to talk to Wayne Siu  and the team and get this work scheduled. The whole process will take several days in best case scenario, so if we want to run perf test with sdk3-rc before GA we will need to move soon.
          ingenthr Matt Ingenthron added a comment - - edited

          Thanks for the update.

          Whether it's for this or a future issue, you might want to look into how you can update Python. I don't doubt you that it was apt-get'd at some point, but I'm pretty sure Ubuntu, even LTS, isn't sticking to 3.5.2 from 2016. I'd have to believe that even if we remove cattrs as a dependency, something someday is going to need you to break down whatever multi-day barrier you have in updating pythons within the version. Related you're on EOL software, posted back in 2018:

          Python 3.5 has now entered "security fixes only" mode, and as such the only changes since Python 3.5.4 are security fixes. Also, Python 3.5.6 has only been released in source code form; no more official binary installers will be produced.

          We'll still look into what we can do with this, of course.

          ingenthr Matt Ingenthron added a comment - - edited Thanks for the update. Whether it's for this or a future issue, you might want to look into how you can update Python. I don't doubt you that it was apt-get'd at some point, but I'm pretty sure Ubuntu, even LTS, isn't sticking to 3.5.2 from 2016. I'd have to believe that even if we remove cattrs as a dependency, something someday is going to need you to break down whatever multi-day barrier you have in updating pythons within the version. Related you're on EOL software, posted back in 2018: Python 3.5 has now entered "security fixes only" mode, and as such the only changes since Python 3.5.4 are security fixes. Also, Python 3.5.6 has only been released in source code form; no more official binary installers will be produced. We'll still look into what we can do with this, of course.

          Agreed, we should look into upgrading our python, automating the upgrades across all client servers. Also, need to figure out which version we should upgrade to. Will talk to Wayne Siu. Thanks Matt

          korrigan.clark Korrigan Clark (Inactive) added a comment - Agreed, we should look into upgrading our python, automating the upgrades across all client servers. Also, need to figure out which version we should upgrade to. Will talk to Wayne Siu . Thanks Matt
          david.kelly David Kelly added a comment -

          Hey Korrigan Clark – IIRC we may be using some async apis that were not there in 3.5.2 as well. So 3.5.2 may have runtime errors even if it builds. I'm sure 3.5.0 had that issue, and almost positive 3.5.9 worked. I didn't check in-between systematically, but vaguely recall 3.5.2 was a problem (before cattrs). And that was a few weeks ago – unsure now.

          david.kelly David Kelly added a comment - Hey Korrigan Clark – IIRC we may be using some async apis that were not there in 3.5.2 as well. So 3.5.2 may have runtime errors even if it builds. I'm sure 3.5.0 had that issue, and almost positive 3.5.9 worked. I didn't check in-between systematically, but vaguely recall 3.5.2 was a problem (before cattrs). And that was a few weeks ago – unsure now.

          People

            Ellis.Breen Ellis Breen
            korrigan.clark Korrigan Clark (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty