Uploaded image for project: 'Couchbase Elasticsearch Connector'
  1. Couchbase Elasticsearch Connector
  2. CBES-10

Data Loss Loading 1 Million Docs to ES Cluster

    XMLWordPrintable

Details

    Description

      Used mcsoda to load 1 million documents.
      stop mcsoda at end of doc loading,
      664 items are missing from elastic search. It's been 2 hours since last data mutation and the missing documents have not been pushed to elastic search.

      Still I'm not sure what cause may be. In the couchbase logs there was an error replicating vbucket 802:

      2013-01-17 10:47:15 - Error replicating vbucket 802: {badarg, [

      {erlang, list_to_integer, ["004340123bf52ca200000HJ5IjoiM2UiLAogInJlYWxtIjoiZWYiLAogImNvaW5zIjozMDcuMjgsCiAiY2F0ZWdvcnkiOjEsCiAiYWNoaWV2ZW1lbnRzIjpbNjAsIDE4NiwgNDAsIDE5Nl0sImJvZHkiOiJ4YzRjYTQyMzhhMGI5MjM4MjBkY2M1MDlhNmY3NTg0OWIifQ==", 16]}

      ,

      {couch_doc, parse_revid,1}

      ,

      {couch_doc,parse_rev,1}

      ,

      {couch_api_wrap, '-get_missing_revs/2-fun-0-', 1}

      ,

      {lists,map,2}, {lists,map,2}

      ,

      {couch_api_wrap, '-get_missing_revs/2-fun-1-', 3}

      ,

      {xdc_vbucket_rep_worker, find_missing,2}

      ]}

      I did a couchdb_dump on this vbucket to see if all it's keys were in the index and they were. I've attached this stat log with this error and will continue to investigate, Marty perhaps you can repro. No errors reported in couchbase logs.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mschoch Marty Schoch [X] (Inactive)
            tommie Tommie McAfee (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty