Uploaded image for project: 'Couchbase Spark Connector'
  1. Couchbase Spark Connector
  2. SPARKC-105

Spark-Couchbase streaming job fails after some time of lauching

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • None
    • 2.4.0
    • Core
    • None
    • 1
    • Medium

    Description

      Hello,

      A) Using following jars with Spark spark-2.4.4-bin-hadoop2.7 and Scala 2.11: 

      • couchbase-spark-connector_2.11-2.4.0.jar
      • core-io-1.7.6.jar
      • dcp-client-0.23.0.jar
      • java-client-2.7.6.jar
      • opentracing-api-0.31.0.jar
      • rxjava-1.3.8.jar
      • rxscala_2.11-0.27.0.jar

      B) Steps:

      • I start the spark streaming job to read from couchbase (from "now")
      • The job starts fine but after some time it stops accepting changed data and then eventually throws following exception. 
      • The spark streaming job still keeps running so then I need to manually find out if it throwing errors and then bounce that spark couchbase streaming job.
      • Please see the attached document for full stack trace.

       

      java.lang.IllegalArgumentException: Frame length should be positive: -2071937303567066099java.lang.IllegalArgumentException: Frame length should be positive: -2071937303567066099 at org.spark_project.guava.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:134) at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:81) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748)

      Couchbase-error.txt Couchbase-error.txt

       

      Can you please take a look and let me know if there is any stop-gap solution.

       

      Thanks,

      Amar

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            graham.pople Graham Pople
            amarsladdha Amar Laddha
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty