Uploaded image for project: 'Couchbase Python Client Library'
  1. Couchbase Python Client Library
  2. PYCBC-206

streaming=True - drops last } from json response object resulting in error.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: library
    • Security Level: Public
    • Labels:
      None
    • Environment:
      libcouchbase 2.1
      python client 1.1

      Description

      We had been experimenting with using the streaming=True for our view iteration as we do not want to load the entire result set into memory before beginning processing and noticed that if a value is present in the view emission, we get an error in the json scan of the return object. This is the error that python spits back:

      Traceback (most recent call last):
      File "/Users/belbis/Desktop/stream_test.py", line 20, in <module>
      for q in query1:
      File "build/bdist.macosx-10.8-intel/egg/couchbase/views/iterator.py", line 390, in _iter_
      File "build/bdist.macosx-10.8-intel/egg/couchbase/views/iterator.py", line 361, in _get_page
      File "build/bdist.macosx-10.8-intel/egg/couchbase/views/iterator.py", line 361, in <genexpr>
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/_init_.py", line 326, in loads
      return _default_decoder.decode(s)
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
      obj, end = self.scan_once(s, idx)
      ValueError: Expecting object: line 1 column 108 (char 108)

      We did some investigation into the matter and if we catch this exception in the json.loads method and append a '}' to the end of the object being scanned, the error goes away. We are not sure how this occurs but we believe that the streaming=True parameter being passed to the view causes it to drop a the closing json character (or something else we're not exactly sure).

      We are able to replicate this behavior in the sample bucket 'beer-sample' with a simple test.

      I created a design doc 'b' and added two views 'stream_test1' and 'stream_test2'. The attached stream_test.py will attempt to iterate over the two views (both with streaming enabled and disabled). What you will see is that the view that emits a value will fail with streaming=True enabled. This error will also happen if you use the view iterator constructor so we assume that the problem is in the handling of the result object and not the iterator itself.

      We would appreciate any input you guys could give on what's going on here. For the time being we are able to work around this by using only keys in views we need to stream. However certain processes are being bogged down by this issue.

      1. stream_test.txt
        1.0 kB
        Alex Ma
      2. stream_test1.txt
        0.1 kB
        Alex Ma
      3. stream_test2.txt
        0.1 kB
        Alex Ma
      # Subject Project Status CR V
      For Gerrit Dashboard: &For+PYCBC-206=message:PYCBC-206

        Activity

        Hide
        mnunberg Mark Nunberg added a comment -

        I did fix a bit of the streaming stuff in the next release. I'll see if it's broken there too.

        Show
        mnunberg Mark Nunberg added a comment - I did fix a bit of the streaming stuff in the next release. I'll see if it's broken there too.
        Hide
        mnunberg Mark Nunberg added a comment - - edited

        def test_pycbc_206(self):

        1. Streaming with values
          view = self.cb.query("beer", "by_location", streaming=True, reduce=False)
          rows = list(view)
          self.assertTrue(len(rows))

        Seems to work. However I'm quite sure it also works with 1.1.0 - so i'm digging in a bit more..

        Show
        mnunberg Mark Nunberg added a comment - - edited def test_pycbc_206(self): Streaming with values view = self.cb.query("beer", "by_location", streaming=True, reduce=False) rows = list(view) self.assertTrue(len(rows)) Seems to work. However I'm quite sure it also works with 1.1.0 - so i'm digging in a bit more..
        Hide
        mnunberg Mark Nunberg added a comment -

        So I've reproduced it by usign the exact view mentioned. Let's see if this still happens in the latest version..

        Show
        mnunberg Mark Nunberg added a comment - So I've reproduced it by usign the exact view mentioned. Let's see if this still happens in the latest version..
        Hide
        mnunberg Mark Nunberg added a comment -

        And it does..

        Show
        mnunberg Mark Nunberg added a comment - And it does..
        Show
        mnunberg Mark Nunberg added a comment - http://review.couchbase.org/30805
        Show
        mnunberg Mark Nunberg added a comment - https://github.com/couchbase/couchbase-python-client/commit/15dd72aa297da40284cc90cc648274c4d0abe132

          People

          • Assignee:
            mnunberg Mark Nunberg
            Reporter:
            alex Alex Ma
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes