Uploaded image for project: 'Java Couchbase JVM Core'
  1. Java Couchbase JVM Core
  2. JVMCBC-446

N1ql query result parsing has poor performance for big document

    XMLWordPrintable

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.5.0
    • N1QL
    • java 8

    Description

      The environment :

      • couchbase server 4.6
      • java client

      on the same machine (dev), plenty of ram, SSD disk, 4 cores

      Simple bucket with only 2 (heavy) documents.
      Simple query select b.* FROM thebucket b

      Execute the query in the couchbase admin : 120 ms
      the same query with the java client : 12 seconds
      the thread doing the computation is using a core at 100%

      the document is huge and badly designed but that's not the point
      the performances should be nearly the same between java and the admin console.

      After investigation the performance bottleneck is in QueryHandler#parseQueryRows > ByteBufJsonHelper#findSectionClosingPosition
      Looks like data in being appended to a buffer but the scan with forEachByte is always done from the beginning of the buffer.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          daschl Michael Nitschinger added a comment - - edited

          We have a new experimental parser in. Can you try setting the sys property "com.couchbase.enableYasjlQueryResponseParser" to "true" before initializing the client and see if it is better?

          daschl Michael Nitschinger added a comment - - edited We have a new experimental parser in. Can you try setting the sys property "com.couchbase.enableYasjlQueryResponseParser" to "true" before initializing the client and see if it is better?
          benoit.wiart Benoit Wiart added a comment -

          Will try the new parser

          I have a haskish patch for this bug. In QueryHandler#parseQueryRows I reuse the ClosingPositionBufProcessor and restart the scan when when it ends at the last run with int forEachByte(int index, int length, ByteBufProcessor processor)
          It brings the performance on par with the query launched from the console.

          benoit.wiart Benoit Wiart added a comment - Will try the new parser I have a haskish patch for this bug. In QueryHandler#parseQueryRows I reuse the ClosingPositionBufProcessor and restart the scan when when it ends at the last run with int forEachByte(int index, int length, ByteBufProcessor processor) It brings the performance on par with the query launched from the console.
          benoit.wiart Benoit Wiart added a comment -

          Launch with -Dcom.couchbase.enableYasjlQueryResponseParser=true

          Got an exception

          Exception in thread "main" java.lang.IllegalStateException: The content of this Observable (querySignature.bf900f3f-311a-4222-9f2b-c4d71d8dfe48) is already released. Subscribe earlier or tune the CouchbaseEnvironment#autoreleaseAfter() setting.
          	at com.couchbase.client.core.utils.UnicastAutoReleaseSubject$OnSubscribeAction.call(UnicastAutoReleaseSubject.java:243)
          	at com.couchbase.client.core.utils.UnicastAutoReleaseSubject$OnSubscribeAction.call(UnicastAutoReleaseSubject.java:207)
          	at rx.Observable.unsafeSubscribe(Observable.java:10346)
          	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
          	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
          	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
          	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
          	at rx.Observable.unsafeSubscribe(Observable.java:10346)
          	at rx.internal.operators.OperatorZip$Zip.start(OperatorZip.java:202)
          	at rx.internal.operators.OperatorZip$ZipSubscriber.onNext(OperatorZip.java:143)
          	at rx.internal.operators.OperatorZip$ZipSubscriber.onNext(OperatorZip.java:109)
          	at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276)
          	at rx.Subscriber.setProducer(Subscriber.java:209)
          	at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138)
          	at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129)
          	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
          	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
          	at rx.Observable.unsafeSubscribe(Observable.java:10346)
          	at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
          	at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
          	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
          	at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:511)
          	at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:466)
          	at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:244)
          	at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
          	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
          	at rx.observers.Subscribers$5.onNext(Subscribers.java:235)
          	at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
          	at rx.Subscriber.setProducer(Subscriber.java:211)
          	at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
          	at rx.Subscriber.setProducer(Subscriber.java:205)
          	at rx.subjects.AsyncSubject.onCompleted(AsyncSubject.java:103)
          	at com.couchbase.client.core.endpoint.AbstractGenericHandler.completeResponse(AbstractGenericHandler.java:390)
          	at com.couchbase.client.core.endpoint.AbstractGenericHandler.access$000(AbstractGenericHandler.java:72)
          	at com.couchbase.client.core.endpoint.AbstractGenericHandler$1.call(AbstractGenericHandler.java:408)
          	at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:745)
          

          benoit.wiart Benoit Wiart added a comment - Launch with -Dcom.couchbase.enableYasjlQueryResponseParser=true Got an exception Exception in thread "main" java.lang.IllegalStateException: The content of this Observable (querySignature.bf900f3f-311a- 4222 -9f2b-c4d71d8dfe48) is already released. Subscribe earlier or tune the CouchbaseEnvironment#autoreleaseAfter() setting. at com.couchbase.client.core.utils.UnicastAutoReleaseSubject$OnSubscribeAction.call(UnicastAutoReleaseSubject.java: 243 ) at com.couchbase.client.core.utils.UnicastAutoReleaseSubject$OnSubscribeAction.call(UnicastAutoReleaseSubject.java: 207 ) at rx.Observable.unsafeSubscribe(Observable.java: 10346 ) at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java: 48 ) at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java: 33 ) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java: 48 ) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java: 30 ) at rx.Observable.unsafeSubscribe(Observable.java: 10346 ) at rx.internal.operators.OperatorZip$Zip.start(OperatorZip.java: 202 ) at rx.internal.operators.OperatorZip$ZipSubscriber.onNext(OperatorZip.java: 143 ) at rx.internal.operators.OperatorZip$ZipSubscriber.onNext(OperatorZip.java: 109 ) at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java: 276 ) at rx.Subscriber.setProducer(Subscriber.java: 209 ) at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java: 138 ) at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java: 129 ) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java: 48 ) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java: 30 ) at rx.Observable.unsafeSubscribe(Observable.java: 10346 ) at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java: 248 ) at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java: 148 ) at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java: 77 ) at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java: 511 ) at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java: 466 ) at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java: 244 ) at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java: 148 ) at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java: 77 ) at rx.observers.Subscribers$ 5 .onNext(Subscribers.java: 235 ) at rx.internal.producers.SingleProducer.request(SingleProducer.java: 65 ) at rx.Subscriber.setProducer(Subscriber.java: 211 ) at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java: 102 ) at rx.Subscriber.setProducer(Subscriber.java: 205 ) at rx.subjects.AsyncSubject.onCompleted(AsyncSubject.java: 103 ) at com.couchbase.client.core.endpoint.AbstractGenericHandler.completeResponse(AbstractGenericHandler.java: 390 ) at com.couchbase.client.core.endpoint.AbstractGenericHandler.access$ 000 (AbstractGenericHandler.java: 72 ) at com.couchbase.client.core.endpoint.AbstractGenericHandler$ 1 .call(AbstractGenericHandler.java: 408 ) at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java: 55 ) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java: 511 ) at java.util.concurrent.FutureTask.run(FutureTask.java: 266 ) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$ 201 (ScheduledThreadPoolExecutor.java: 180 ) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java: 293 ) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1142 ) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 617 ) at java.lang.Thread.run(Thread.java: 745 )
          benoit.wiart Benoit Wiart added a comment - See https://github.com/couchbase/couchbase-jvm-core/pull/9

          thanks!

          daschl Michael Nitschinger added a comment - thanks!

          People

            daschl Michael Nitschinger
            benoit.wiart Benoit Wiart
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty