Uploaded image for project: 'Couchbase Spark Connector'
  1. Couchbase Spark Connector
  2. SPARKC-33

Avoid failing on unsupported filters

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Fixed
    • Major
    • 1.1.0
    • None
    • None
    • None

    Description

      From an email conversation with an engineer from Databricks:

      (Also, scanning through the source of the connector I think that I can already spot a few places where you'll hit potential trouble in 1.5.x upgrading. For instance, https://github.com/couchbaselabs/couchbase-spark-connector/blob/04a3e1be1119c5042075623b2a2e79357264316c/src/main/scala/com/couchbase/spark/sql/N1QLRelation.scala#L160 is probably going to be an issue if we ever add pushdown support for new filter types; you would be better off just ignoring unsupported filters, since Spark will perform its own layer of filtering on top anyways (just to be safe / to enable you to skip unsupported filters / filters that don't map to your query language))

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            daschl Michael Nitschinger
            daschl Michael Nitschinger
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty