Uploaded image for project: 'Couchbase Python Client Library'
  1. Couchbase Python Client Library
  2. PYCBC-984

Some N1QL query settings may not get set

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Test Blocker
    • Resolution: Fixed
    • 3.0.3
    • 3.0.4
    • library
    • 1
    • SDK28: Txn Loop/Rb,PfQueryColl, SDK32: QueryCollections, SDK30: Connect,Txns

    Description

      https://github.com/couchbase/couchbase-python-client/blob/master/couchbase/cluster.py#L194

              for k in self.VALID_OPTS:
                  v = args.get(k, None)
                  if v:
                      if k == 'adhoc':
                          query.adhoc = v
      

      If I want 'adhoc' to be False, then v = False and adhoc never gets set. The default value for 'adhoc' is set to True: https://github.com/couchbase/couchbase-python-client/blob/master/couchbase_core/n1ql.py#L124

      This will also affect any of the parameters that are bools or ints, and maybe others as well. Not just adhoc

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          So I tried to hard code adhoc to False in the python sdk code. This resulted in no prepared statements being created. So adhoc=True and adhoc=False both behave like ahdoc=True and thus no prepared statments are ever created or used.

          Looking a bit deeper, if adhoc=False:

          https://github.com/couchbase/couchbase-python-client/blob/ef37d587c702c69cc95fbaf97d745b82b3a48f6e/couchbase_core/n1ql.py#L414

           

          def _submit_query(self): 
          return self._parent._n1ql_query(self._params.encoded, not self._params.adhoc, cross_bucket=self._params.cross_bucket)
          

          not self._params.adhoc become equal to True

          This code is then called:

          https://github.com/couchbase/couchbase-python-client/blob/ef37d587c702c69cc95fbaf97d745b82b3a48f6e/src/n1ql.c#L472

          This will set int prepared to 1 corresponding to "not adhoc" being true.

          Then this is called:

          https://github.com/couchbase/couchbase-python-client/blob/fa95702086f2cb5b90657b42401c547d8ebd579a/src/n1ql.c#L282

          This will call lcb_cmdquery_adhoc(cmd, 1) if prepared=1, which it does.

          The lcb_cmdquery_adhoc code: https://github.com/couchbase/libcouchbase/blob/7a5e90542e59fc32f410a8d6c23fc3e597afc45f/src/n1ql/n1ql.cc#L203

          This will turn tell LCB to run the query as an adhoc statement.

           

          So by setting adhoc=False in python sdk, lcb will actually see a request for adhoc=True. This looks like it comes from the python sdk c library that switches terminology up a bit by translating adhoc to a variable called prepared by negativing which is then transalted to adhoc in lcb.

          If I set adhoc=False in the sdk, then each query should check in the prepared cache for a corresponding prepared statement. If a statment is found, execute and if not found, prepare then execute.

           

          korrigan.clark Korrigan Clark (Inactive) added a comment - So I tried to hard code adhoc to False in the python sdk code. This resulted in no prepared statements being created. So adhoc=True and adhoc=False both behave like ahdoc=True and thus no prepared statments are ever created or used. Looking a bit deeper, if adhoc=False: https://github.com/couchbase/couchbase-python-client/blob/ef37d587c702c69cc95fbaf97d745b82b3a48f6e/couchbase_core/n1ql.py#L414   def _submit_query(self): return self._parent._n1ql_query(self._params.encoded, not self._params.adhoc, cross_bucket=self._params.cross_bucket) not self._params.adhoc become equal to True This code is then called: https://github.com/couchbase/couchbase-python-client/blob/ef37d587c702c69cc95fbaf97d745b82b3a48f6e/src/n1ql.c#L472 This will set int prepared to 1 corresponding to "not adhoc" being true. Then this is called: https://github.com/couchbase/couchbase-python-client/blob/fa95702086f2cb5b90657b42401c547d8ebd579a/src/n1ql.c#L282 This will call lcb_cmdquery_adhoc(cmd, 1) if prepared=1, which it does. The lcb_cmdquery_adhoc code:  https://github.com/couchbase/libcouchbase/blob/7a5e90542e59fc32f410a8d6c23fc3e597afc45f/src/n1ql/n1ql.cc#L203 This will turn tell LCB to run the query as an adhoc statement.   So by setting adhoc=False in python sdk, lcb will actually see a request for adhoc=True. This looks like it comes from the python sdk c library that switches terminology up a bit by translating adhoc to a variable called prepared by negativing which is then transalted to adhoc in lcb. If I set adhoc=False in the sdk, then each query should check in the prepared cache for a corresponding prepared statement. If a statment is found, execute and if not found, prepare then execute.  
          Ellis.Breen Ellis Breen added a comment -

          Thanks for the update - will take a close look at this ASAP.

          Ellis.Breen Ellis Breen added a comment - Thanks for the update - will take a close look at this ASAP.
          korrigan.clark Korrigan Clark (Inactive) added a comment - - edited

          ran a test with the pending code changes for this issue: http://perf.jenkins.couchbase.com/job/iris-collections/50/

          Seems that adhoc is still not working properly as the reported latency is still too high ~ 3.5ms whereas with adhoc=true we expect a latency in the range of 1-2ms

          Using this: pip install git+http://review.couchbase.org/couchbase-python-client@refs/changes/47/133447/3

          korrigan.clark Korrigan Clark (Inactive) added a comment - - edited ran a test with the pending code changes for this issue:  http://perf.jenkins.couchbase.com/job/iris-collections/50/ Seems that adhoc is still not working properly as the reported latency is still too high ~ 3.5ms whereas with adhoc=true we expect a latency in the range of 1-2ms Using this:  pip install git+ http://review.couchbase.org/couchbase-python-client@refs/changes/47/133447/3

          This code seemed to fix the adhoc issue:

          pip install git+http://review.couchbase.org/couchbase-python-client@refs/changes/46/133446/12

          Here is a job with latency around 1.5ms which is expected for prepared queries of this nature:http://perf.jenkins.couchbase.com/job/iris-collections/51/console

          korrigan.clark Korrigan Clark (Inactive) added a comment - This code seemed to fix the adhoc issue: pip install git+ http://review.couchbase.org/couchbase-python-client@refs/changes/46/133446/12 Here is a job with latency around 1.5ms which is expected for prepared queries of this nature: http://perf.jenkins.couchbase.com/job/iris-collections/51/console

          Ellis Breen Do you know when this fix will be merged into mainline? Any ETA is appreciated thank you

          raju Raju Suravarjjala added a comment - Ellis Breen Do you know when this fix will be merged into mainline? Any ETA is appreciated thank you
          Ellis.Breen Ellis Breen added a comment -

          Hi, it's in master now, and was released as part of 3.0.4 yesterday.

          Ellis.Breen Ellis Breen added a comment - Hi, it's in master now, and was released as part of 3.0.4 yesterday.

          People

            Ellis.Breen Ellis Breen
            korrigan.clark Korrigan Clark (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty