We had some complain from developer that are using large view for example 500k elements. (with small content id+url for example). The total size of the view is in this case around 52Mb.
The result out of memory.
2013-05-01 16:47:06.536 INFO
/192.168.63.99:8092 - Session request successful
Exception in thread "I/O dispatcher 3" java.lang.OutOfMemoryError: Java
Note from our community member
After digging into the SDK and apache code, it occurred that the viewResponse actually contains Strings (the json objects). I think the "problem" lies in handleResponse method in ViewOperationImpl.
Here the SDK are using the apache http core to get the json String via the call
String json = getEntityString(response);
Would it be smarter to give back a InputStream instead? This could quite
easily be fixed because the response could give back a HttpEntity that
again could give the developer a InputStream.
In this way the SDK did not have to copy each Char, which it actually
does in the apache http core library (see expand in
CharArrayBufffer.class ). It does the following: char newbuffer = new
char[Math.max(this.buffer.length << 1, newlen)];
I think it kind of chokes when it get 52MB it have to copy from Char to a
String and then deliver it back.
BTW: I've tried calling the same view and operating on it form node.js
(using baseview and the official sdk), it just worked like a charm. And
this is prob. because you get a stream of data chunks that you can work
on, which seems more effective.