Stopping and restarting of Spark-Couchbase-Streaming-job throws ClassCastException
Description
Environment
Gerrit Reviews
Release Notes Description
Attachments
Activity
Graham Pople October 4, 2021 at 3:49 PM
Closing out this stale bug as a batch task, please reopen if further action is required.
Amar Laddha April 27, 2020 at 6:11 PMEdited
Hi ,
Sorry for the delayed response. Thanks again for the clarification.
I would need more time to review CLA and register for an account.
Instead can you please submit the patch? Sorry for the confusion.
Thanks,
Amar
Graham Pople April 23, 2020 at 8:13 AM
Thank you (though just to note, it will need to be a Gerrit review rather than a Github PR for us to be able to take it - all instructions are in the CONTRIBUTING.md guide)
Amar Laddha April 20, 2020 at 9:58 PM
Hi ,
Thank you for this opportunity. I will follow the instructions and send our a PR link by the end of this week.
Thanks,
Amar
Graham Pople April 20, 2020 at 8:33 AM
Hi
Thank you for the patch. Would you like to submit it to us through our code review process, Gerrit, following the instructions here https://github.com/couchbase/couchbase-spark-connector/blob/master/CONTRIBUTING.md? It would recognise your contribution properly. If not, I can submit it to gerrit instead.
Details
Details
Assignee
Graham PopleReporter
Amar LaddhaCVSS/Severity
Story Points
Components
Affects versions
Priority
MajorInstabug
PagerDuty
PagerDuty Incident
PagerDuty
Sentry
Linked Issues
Sentry
Zendesk Support
Linked Tickets
Zendesk Support
Using following jars with Spark spark-2.4.3-bin-hadoop2.7 and Scala 2.11:
spark-connector_2.11-2.3.0.jar
opentracing-api-0.31.0.jar
rxjava-1.3.8.jar
rxscala_2.11-0.27.0.jar
dcp-client-0.23.0.jar
java-client-2.7.6.jar
core-io-1.7.6.jar
Exception:
ERROR MicroBatchExecution: Query [id = 5206cb7e-4260-4465-816f-82bf6b1429f6, runId = df2d54e7-c3f9-4075-810b-1ea7ebe0b9b6] terminated with error
java.lang.ClassCastException: org.apache.spark.sql.execution.streaming.SerializedOffset cannot be cast to com.couchbase.spark.sql.streaming.CouchbaseSourceOffset
at com.couchbase.spark.sql.streaming.CouchbaseSource$$anonfun$7.apply(CouchbaseSource.scala:172)
at com.couchbase.spark.sql.streaming.CouchbaseSource$$anonfun$7.apply(CouchbaseSource.scala:172)
at scala.Option.map(Option.scala:146)
at com.couchbase.spark.sql.streaming.CouchbaseSource.getBatch(CouchbaseSource.scala:172)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$3$$anonfun$apply$10.apply(MicroBatchExecution.scala:438)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$3$$anonfun$apply$10.apply(MicroBatchExecution.scala:434)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at org.apache.spark.sql.execution.streaming.StreamProgress.foreach(StreamProgress.scala:25)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at org.apache.spark.sql.execution.streaming.StreamProgress.flatMap(StreamProgress.scala:25)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$3.apply(MicroBatchExecution.scala:434)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$3.apply(MicroBatchExecution.scala:434)
at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch(MicroBatchExecution.scala:433)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:198)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1.apply$mcZ$sp(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:160)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:281)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)