Uploaded image for project: 'Couchbase Elasticsearch Connector'
  1. Couchbase Elasticsearch Connector
  2. CBES-117

Connector exits on values that fail to parse

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.0.0
    • Fix Version/s: 4.0.1
    • Labels:
      None

      Description

      This issue was encountered with a field in a document with a very large numeric value - 17626319910530664276, which caused an error when attempting to replicate to ElasticSearch.

      Seen on version 4.0. I'm unsure if this also affects previous versions.

      This value appears to have been read as a BigInteger, which has caused an error in the ElasticSearch client:
       

      10:01:55.285 [es-worker-0] WARN  c.c.c.e.ElasticsearchWorker - Error in Elasticsearch worker thread
      java.lang.IllegalArgumentException: cannot write xcontent for unknown value of type class java.math.BigInteger
              at org.elasticsearch.common.xcontent.XContentBuilder.unknownValue(XContentBuilder.java:755) ~[elasticsearch-x-content-6.3.2.jar:6.3.2]
              at org.elasticsearch.common.xcontent.XContentBuilder.map(XContentBuilder.java:810) ~[elasticsearch-x-content-6.3.2.jar:6.3.2]
              at org.elasticsearch.common.xcontent.XContentBuilder.map(XContentBuilder.java:792) ~[elasticsearch-x-content-6.3.2.jar:6.3.2]
              at org.elasticsearch.action.index.IndexRequest.source(IndexRequest.java:313) ~[elasticsearch-6.3.2.jar:6.3.2]
              at com.couchbase.connector.elasticsearch.io.DefaultDocumentTransformer.setSourceFromEventContent(DefaultDocumentTransformer.java:121) ~[couchbase-elasticsearch-connector-4.0.0.jar:?]
              at com.couchbase.connector.elasticsearch.io.RequestFactory.newIndexRequest(RequestFactory.java:87) ~[couchbase-elasticsearch-connector-4.0.0.jar:?]
              at com.couchbase.connector.elasticsearch.io.RequestFactory.newDocWriteRequest(RequestFactory.java:72) ~[couchbase-elasticsearch-connector-4.0.0.jar:?]
              at com.couchbase.connector.elasticsearch.io.ElasticsearchWriter.write(ElasticsearchWriter.java:151) ~[couchbase-elasticsearch-connector-4.0.0.jar:?]
              at com.couchbase.connector.elasticsearch.ElasticsearchWorker.lambda$doRun$0(ElasticsearchWorker.java:76) ~[couchbase-elasticsearch-connector-4.0.0.jar:?]
              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
      10:01:55.289 [es-worker-0] INFO  c.c.c.e.ElasticsearchWorker - Thread[es-worker-0,5,main] stopped.
      10:01:55.289 [main] ERROR c.c.c.e.ElasticsearchConnector - Terminating due to fatal error.
      java.lang.IllegalArgumentException: cannot write xcontent for unknown value of type class java.math.BigInteger
              at org.elasticsearch.common.xcontent.XContentBuilder.unknownValue(XContentBuilder.java:755) ~[elasticsearch-x-content-6.3.2.jar:6.3.2]
              at org.elasticsearch.common.xcontent.XContentBuilder.map(XContentBuilder.java:810) ~[elasticsearch-x-content-6.3.2.jar:6.3.2]
              at org.elasticsearch.common.xcontent.XContentBuilder.map(XContentBuilder.java:792) ~[elasticsearch-x-content-6.3.2.jar:6.3.2]
              at org.elasticsearch.action.index.IndexRequest.source(IndexRequest.java:313) ~[elasticsearch-6.3.2.jar:6.3.2]
              at com.couchbase.connector.elasticsearch.io.DefaultDocumentTransformer.setSourceFromEventContent(DefaultDocumentTransformer.java:121) ~[couchbase-elasticsearch-connector-4.0.0.jar:?]
              at com.couchbase.connector.elasticsearch.io.RequestFactory.newIndexRequest(RequestFactory.java:87) ~[couchbase-elasticsearch-connector-4.0.0.jar:?]
              at com.couchbase.connector.elasticsearch.io.RequestFactory.newDocWriteRequest(RequestFactory.java:72) ~[couchbase-elasticsearch-connector-4.0.0.jar:?]
              at com.couchbase.connector.elasticsearch.io.ElasticsearchWriter.write(ElasticsearchWriter.java:151) ~[couchbase-elasticsearch-connector-4.0.0.jar:?]
              at com.couchbase.connector.elasticsearch.ElasticsearchWorker.lambda$doRun$0(ElasticsearchWorker.java:76) ~[couchbase-elasticsearch-connector-4.0.0.jar:?]
              at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_201]
      
      

      There are two issues here:
      1. In this specific case, very large values are interpreted as a BigInteger by the connector, which is causing errors.
      2. More generally, and more importantly, the connector process exits when parsing errors such as these are hit, which can be quite disruptive.

      Is the process exiting the intended approach to handling errors such as these? In the general case, should documents that fail be logged and skipped, to allow the replication to continue?

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            david.nault David Nault added a comment -

            Toby Wilds 

            1. In this specific case, very large values are interpreted as a BigInteger by the connector, which is causing errors.

            Elasticsearch cannot store integers larger than 2^63-1. The document must either be rejected or somehow transformed. If a transformation is desired, let's track that as a separate issue.

            2. More generally, and more importantly, the connector process exits when parsing errors such as these are hit, which can be quite disruptive.

            Is the process exiting the intended approach to handling errors such as these? In the general case, should documents that fail be logged and skipped, to allow the replication to continue?

            Agreed, definitely a bug. The intent in these situations is to log the rejection and continue replicating. This particular case slipped through the cracks.

            Show
            david.nault David Nault added a comment - Toby Wilds   1. In this specific case, very large values are interpreted as a BigInteger by the connector, which is causing errors. Elasticsearch cannot store integers larger than 2^63-1. The document must either be rejected or somehow transformed. If a transformation is desired, let's track that as a separate issue. 2. More generally, and more importantly, the connector process exits when parsing errors such as these are hit, which can be quite disruptive. Is the process exiting the intended approach to handling errors such as these? In the general case, should documents that fail be logged and skipped, to allow the replication to continue? Agreed, definitely a bug. The intent in these situations is to log the rejection and continue replicating. This particular case slipped through the cracks.

              People

              • Assignee:
                david.nault David Nault
                Reporter:
                toby.wilds Toby Wilds
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.