Uploaded image for project: 'Couchbase Python Client Library'
  1. Couchbase Python Client Library
  2. PYCBC-206

streaming=True - drops last } from json response object resulting in error.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 1.2.0
    • 1.1.0
    • library
    • Security Level: Public
    • None
    • libcouchbase 2.1
      python client 1.1

    Description

      We had been experimenting with using the streaming=True for our view iteration as we do not want to load the entire result set into memory before beginning processing and noticed that if a value is present in the view emission, we get an error in the json scan of the return object. This is the error that python spits back:

      Traceback (most recent call last):
      File "/Users/belbis/Desktop/stream_test.py", line 20, in <module>
      for q in query1:
      File "build/bdist.macosx-10.8-intel/egg/couchbase/views/iterator.py", line 390, in _iter_
      File "build/bdist.macosx-10.8-intel/egg/couchbase/views/iterator.py", line 361, in _get_page
      File "build/bdist.macosx-10.8-intel/egg/couchbase/views/iterator.py", line 361, in <genexpr>
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/_init_.py", line 326, in loads
      return _default_decoder.decode(s)
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
      obj, end = self.scan_once(s, idx)
      ValueError: Expecting object: line 1 column 108 (char 108)

      We did some investigation into the matter and if we catch this exception in the json.loads method and append a '}' to the end of the object being scanned, the error goes away. We are not sure how this occurs but we believe that the streaming=True parameter being passed to the view causes it to drop a the closing json character (or something else we're not exactly sure).

      We are able to replicate this behavior in the sample bucket 'beer-sample' with a simple test.

      I created a design doc 'b' and added two views 'stream_test1' and 'stream_test2'. The attached stream_test.py will attempt to iterate over the two views (both with streaming enabled and disabled). What you will see is that the view that emits a value will fail with streaming=True enabled. This error will also happen if you use the view iterator constructor so we assume that the problem is in the handling of the result object and not the iterator itself.

      We would appreciate any input you guys could give on what's going on here. For the time being we are able to work around this by using only keys in views we need to stream. However certain processes are being bogged down by this issue.

      Attachments

        1. stream_test.txt
          1.0 kB
        2. stream_test1.txt
          0.1 kB
        3. stream_test2.txt
          0.1 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mnunberg Mark Nunberg (Inactive)
            alex Alex Ma [X] (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty