Uploaded image for project: 'Couchbase Python Client Library'
  1. Couchbase Python Client Library
  2. PYCBC-622

Document/configure Git dependency

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-rc
    • None
    • None
    • 1
    • SDK5: Docs, Pathfind Antorized

    Attachments

      For Gerrit Dashboard: PYCBC-622
      # Subject Branch Project Status CR V

      Activity

        Ellis.Breen Ellis Breen added a comment -

        We should document the Git requirement PYCBC 3.0 has, and ideally configure it in the build requirements, e.g. pyproject.toml, if possible.

        Potentially we could download a tarball of the source (CMake externalproject_add allows for this) without requiring Git, although listing the available tags may not be possible, so we'd have to adapt the version selection code accordingly.

        Ellis.Breen Ellis Breen added a comment - We should document the Git requirement PYCBC 3.0 has, and ideally configure it in the build requirements, e.g. pyproject.toml, if possible. Potentially we could download a tarball of the source (CMake externalproject_add allows for this) without requiring Git, although listing the available tags may not be possible, so we'd have to adapt the version selection code accordingly.
        david.kelly David Kelly added a comment - - edited

        Yes indeed I do need git. What I don't understand is why we need git. AFAIK, we simply need libcouchbase source which we could get with something like:

        wget https://github.com/couchbase/libcouchbase/archive/master.zip

        But of course, that is for master. I think we can get a tag as well, replacing master with the tag. I'll experiment.

        david.kelly David Kelly added a comment - - edited Yes indeed I do need git. What I don't understand is why we need git. AFAIK, we simply need libcouchbase source which we could get with something like: wget https://github.com/couchbase/libcouchbase/archive/master.zip But of course, that is for master. I think we can get a tag as well, replacing master with the tag. I'll experiment.
        david.kelly David Kelly added a comment -

        So, we can certainly get the source code (zipped) from GitHub without need for git. For instance, wget https://github.com/couchbase/libcouchbase/archive/3.0.0-beta.2.zip works just fine.

        So perhaps the simplest thing to do is to have cmake get and unzip the code this way. I'll be looking into that next

        david.kelly David Kelly added a comment - So, we can certainly get the source code (zipped) from GitHub without need for git. For instance, wget https://github.com/couchbase/libcouchbase/archive/3.0.0-beta.2.zip works just fine. So perhaps the simplest thing to do is to have cmake get and unzip the code this way. I'll be looking into that next
        david.kelly David Kelly added a comment -

        It would appear there were some changes made to the CMakeLists.txt that allowed for patches from gerrit to be used. Also, there is a lot of code to decide about various tags one could use, etc...

        My claim is this is unnecessarily complex. If you want a specific gerrit patch, apply it to a checked out version of libcouchbase. Then build using that (there are directions for that on our readme). The exact tag (or SHA for that matter) should be burned in on the python side, and each time there is a new lcb release, we can move to that and make another python release. If we want to override that, we can just set an environment variable before building.

        And most important, no need for someone's machine to need git. Not that it is an unusual thing for a server to have, but I've definitely deployed app servers (ruby in particular) without a git client on the server machine. So - I will strip out some of what I'm claiming is unnecessary complexity out, and make a PR shortly.

        david.kelly David Kelly added a comment - It would appear there were some changes made to the CMakeLists.txt that allowed for patches from gerrit to be used. Also, there is a lot of code to decide about various tags one could use, etc... My claim is this is unnecessarily complex. If you want a specific gerrit patch, apply it to a checked out version of libcouchbase. Then build using that (there are directions for that on our readme). The exact tag (or SHA for that matter) should be burned in on the python side, and each time there is a new lcb release, we can move to that and make another python release. If we want to override that, we can just set an environment variable before building. And most important, no need for someone's machine to need git. Not that it is an unusual thing for a server to have, but I've definitely deployed app servers (ruby in particular) without a git client on the server machine. So - I will strip out some of what I'm claiming is unnecessary complexity out, and make a PR shortly.
        Ellis.Breen Ellis Breen added a comment -

        I agree the code is pretty complex, though it served its purpose well during rapid changes in LCB, and our focus right now is on delivering for the end user, who need not have git on their machine. So as we are targeting LCB 3.0.0 this should be fine for now.

        Checking out git patches is actually really useful for testing Gerrit changesets in Jenkins, something that has on several occasions shown up problems that might not otherwise be caught until after the LCB change is merged, potentially breaking PYCBC deployments or builds.Perhaps we can reinstate this facility in some form when we need to test new LCB changes, post LCB 3.0.0.

        The semver matching stuff is also pretty useful for explicitly syncing a given version of the PYCBC code with a range of LCB versions, without having to release a new version of PYCBC just to deliver a new LCB fix. Though this is more of a nice-to-have than the Gerrit checkout functionality.

        Ultimately, this sort of dependency management (if considered necessary) probably should be dealt with by an off-the-shelf system, rather than this ad hoc one. I looked at numerous options for binary dep management, and the one that looked most appropriate was Conan, which allows pulling things from a binary/source repository, with semver, but from what I can see it would entail packaging LCB as a Conan package, and I think I hit some snags when I played with it before (and apparently there are some issues with Python 3.8). However, it would move the burden of source/binary package acquisition/caching to Conan, which would probably be better.

        Ellis.Breen Ellis Breen added a comment - I agree the code is pretty complex, though it served its purpose well during rapid changes in LCB, and our focus right now is on delivering for the end user, who need not have git on their machine. So as we are targeting LCB 3.0.0 this should be fine for now. Checking out git patches is actually really useful for testing Gerrit changesets in Jenkins, something that has on several occasions shown up problems that might not otherwise be caught until after the LCB change is merged, potentially breaking PYCBC deployments or builds.Perhaps we can reinstate this facility in some form when we need to test new LCB changes, post LCB 3.0.0. The semver matching stuff is also pretty useful for explicitly syncing a given version of the PYCBC code with a range of LCB versions, without having to release a new version of PYCBC just to deliver a new LCB fix. Though this is more of a nice-to-have than the Gerrit checkout functionality. Ultimately, this sort of dependency management (if considered necessary) probably should be dealt with by an off-the-shelf system, rather than this ad hoc one. I looked at numerous options for binary dep management, and the one that looked most appropriate was Conan, which allows pulling things from a binary/source repository, with semver, but from what I can see it would entail packaging LCB as a Conan package, and I think I hit some snags when I played with it before (and apparently there are some issues with Python 3.8). However, it would move the burden of source/binary package acquisition/caching to Conan, which would probably be better.
        david.kelly David Kelly added a comment -

        Lots of what was there is super convenient, for sure. In fact we can easily add it back, as long as we have conditionals around it such that it only is used when git is really there. I'd thought of starting with that - but wanted to make things simpler in the short run. Lets think about popping it in, albeit with protection for no git, in a later commit.

        But - Conan seems like an interesting way to go as well. Maybe after GA we can explore it?

        david.kelly David Kelly added a comment - Lots of what was there is super convenient, for sure. In fact we can easily add it back, as long as we have conditionals around it such that it only is used when git is really there. I'd thought of starting with that - but wanted to make things simpler in the short run. Lets think about popping it in, albeit with protection for no git, in a later commit. But - Conan seems like an interesting way to go as well. Maybe after GA we can explore it?
        david.kelly David Kelly added a comment -

        With the caveat that we should consider adding the git stuff back in, conditionally, after GA. Definitely could be convenient later.

        david.kelly David Kelly added a comment - With the caveat that we should consider adding the git stuff back in, conditionally, after GA. Definitely could be convenient later.

        People

          david.kelly David Kelly
          Ellis.Breen Ellis Breen
          Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

            Created:
            Updated:
            Resolved:

            Gerrit Reviews

              There are no open Gerrit changes

              PagerDuty