Details

    • Type: Task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.1.5
    • Fix Version/s: .backlog1.x
    • Component/s: Core
    • Security Level: Public
    • Labels:
      None
    • Environment:
      Java application with large view

      Description

      We had some complain from developer that are using large view for example 500k elements. (with small content id+url for example). The total size of the view is in this case around 52Mb.

      The result out of memory.
      2013-05-01 16:47:06.536 INFO
      com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback:
      /192.168.63.99:8092 - Session request successful
      Exception in thread "I/O dispatcher 3" java.lang.OutOfMemoryError: Java
      heap space
      at org.apache.http.util.CharArrayBuffer.expand(CharArrayBuffer.java:61)
      at org.apache.http.util.CharArrayBuffer.append(CharArrayBuffer.java:91)
      at org.apache.http.util.EntityUtils.toString(EntityUtils.java:200)
      at org.apache.http.util.EntityUtils.toString(EntityUtils.java:221)
      at
      com.couchbase.client.protocol.views.HttpOperationImpl.getEntityString(Http
      OperationImpl.java:106)
      at
      com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewO
      perationImpl.java:60)

      Note from our community member

      After digging into the SDK and apache code, it occurred that the viewResponse actually contains Strings (the json objects). I think the "problem" lies in handleResponse method in ViewOperationImpl.
      Here the SDK are using the apache http core to get the json String via the call
      String json = getEntityString(response);
      Would it be smarter to give back a InputStream instead? This could quite
      easily be fixed because the response could give back a HttpEntity that
      again could give the developer a InputStream.
      In this way the SDK did not have to copy each Char, which it actually
      does in the apache http core library (see expand in
      CharArrayBufffer.class ). It does the following: char newbuffer[] = new
      char[Math.max(this.buffer.length << 1, newlen)];
      I think it kind of chokes when it get 52MB it have to copy from Char to a
      String and then deliver it back.

      BTW: I've tried calling the same view and operating on it form node.js
      (using baseview and the official sdk), it just worked like a charm. And
      this is prob. because you get a stream of data chunks that you can work
      on, which seems more effective.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        daschl Michael Nitschinger added a comment -

        assigning this to .next, because this potentially requires larger changes and needs more investigations.

        Show
        daschl Michael Nitschinger added a comment - assigning this to .next, because this potentially requires larger changes and needs more investigations.

          People

          • Assignee:
            daschl Michael Nitschinger
            Reporter:
            tgrall Tug Grall (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Gerrit Reviews

              There are no open Gerrit changes