Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-13694

XDCR fails for unknown reason

    XMLWordPrintable

Details

    • Bug
    • Resolution: Incomplete
    • Blocker
    • 4.0.0
    • 3.0.1
    • XDCR
    • Security Level: Public
    • None
    • AWS AMI - 2 clusters of 3 machines, each having m3.large
    • Untriaged
    • Centos 64-bit
    • Hide
      [root@ip-10-121-200-100 ec2-user]# tail -f /opt/couchbase/var/lib/couchbase/logs/xdcr_errors.log
      [xdcr:error,2015-02-26T16:18:55.949,ns_1@10.121.200.100:<0.6843.33>:xdc_vbucket_rep_worker:do_flush_docs_xmem:186]replication mode: "xmem", unable to replicate 195 docs to target "http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f77/"
      [xdcr:error,2015-02-26T16:18:55.949,ns_1@10.121.200.100:<0.28560.32>:xdc_vbucket_rep:terminate:499]Replication (XMem mode) `604f111b19d51891aed790af0ddd18c7/proxy_logs/proxy_logs` (`proxy_logs/77` -> `http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f77`) failed.Please see ns_server debug log for complete state dump
      [xdcr:error,2015-02-26T16:19:01.504,ns_1@10.121.200.100:<0.11132.33>:xdc_vbucket_rep_xmem:do_flush_docs:116]out of 188 docs, succ to send 187 docs, fail to send others (by error type, enoent: 1, not-my-vb: 0, einval: 0, tmp fail: 0, enomem: 0, other errors: 0
      [xdcr:error,2015-02-26T16:19:01.504,ns_1@10.121.200.100:<0.11132.33>:xdc_vbucket_rep_worker:do_flush_docs_xmem:186]replication mode: "xmem", unable to replicate 188 docs to target "http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f767/"
      [xdcr:error,2015-02-26T16:19:01.505,ns_1@10.121.200.100:<0.28415.32>:xdc_vbucket_rep:terminate:499]Replication (XMem mode) `604f111b19d51891aed790af0ddd18c7/proxy_logs/proxy_logs` (`proxy_logs/767` -> `http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f767`) failed.Please see ns_server debug log for complete state dump
      [xdcr:error,2015-02-26T16:19:05.118,ns_1@10.121.200.100:<0.15847.33>:xdc_vbucket_rep_xmem:do_flush_docs:116]out of 102 docs, succ to send 101 docs, fail to send others (by error type, enoent: 1, not-my-vb: 0, einval: 0, tmp fail: 0, enomem: 0, other errors: 0
      [xdcr:error,2015-02-26T16:19:05.119,ns_1@10.121.200.100:<0.15847.33>:xdc_vbucket_rep_worker:do_flush_docs_xmem:186]replication mode: "xmem", unable to replicate 102 docs to target "http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f6/"
      [xdcr:error,2015-02-26T16:19:05.119,ns_1@10.121.200.100:<0.28731.32>:xdc_vbucket_rep:terminate:499]Replication (XMem mode) `604f111b19d51891aed790af0ddd18c7/proxy_logs/proxy_logs` (`proxy_logs/6` -> `http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f6`) failed.Please see ns_server debug log for complete state dump
      [xdcr:error,2015-03-02T17:38:17.326,ns_1@10.121.200.100:<0.1902.479>:xdc_vbucket_rep:terminate:499]Replication (XMem mode) `8f97c21845154726d73f0c99ecb6647e/fldb/fldb` (`fldb/501` -> `http://*****@10.121.211.100:8092/fldb%2Ba9de84edc9032e8a2e0503dc3a414d4c%2f501`) failed.Please see ns_server debug log for complete state dump
      [xdcr:error,2015-03-02T17:47:20.428,ns_1@10.121.200.100:<0.31585.480>:xdc_vbucket_rep:terminate:499]Replication (XMem mode) `8f97c21845154726d73f0c99ecb6647e/fldb/fldb` (`fldb/456` -> `http://*****@10.121.211.100:8092/fldb%2Ba9de84edc9032e8a2e0503dc3a414d4c%2f456`) failed.Please see ns_server debug log for complete state dump
      Show
      [ root@ip-10-121-200-100 ec2-user]# tail -f /opt/couchbase/var/lib/couchbase/logs/xdcr_errors.log [xdcr:error,2015-02-26T16:18:55.949, ns_1@10.121.200.100 :<0.6843.33>:xdc_vbucket_rep_worker:do_flush_docs_xmem:186]replication mode: "xmem", unable to replicate 195 docs to target " http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f77/ " [xdcr:error,2015-02-26T16:18:55.949, ns_1@10.121.200.100 :<0.28560.32>:xdc_vbucket_rep:terminate:499]Replication (XMem mode) `604f111b19d51891aed790af0ddd18c7/proxy_logs/proxy_logs` (`proxy_logs/77` -> ` http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f77 `) failed.Please see ns_server debug log for complete state dump [xdcr:error,2015-02-26T16:19:01.504, ns_1@10.121.200.100 :<0.11132.33>:xdc_vbucket_rep_xmem:do_flush_docs:116]out of 188 docs, succ to send 187 docs, fail to send others (by error type, enoent: 1, not-my-vb: 0, einval: 0, tmp fail: 0, enomem: 0, other errors: 0 [xdcr:error,2015-02-26T16:19:01.504, ns_1@10.121.200.100 :<0.11132.33>:xdc_vbucket_rep_worker:do_flush_docs_xmem:186]replication mode: "xmem", unable to replicate 188 docs to target " http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f767/ " [xdcr:error,2015-02-26T16:19:01.505, ns_1@10.121.200.100 :<0.28415.32>:xdc_vbucket_rep:terminate:499]Replication (XMem mode) `604f111b19d51891aed790af0ddd18c7/proxy_logs/proxy_logs` (`proxy_logs/767` -> ` http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f767 `) failed.Please see ns_server debug log for complete state dump [xdcr:error,2015-02-26T16:19:05.118, ns_1@10.121.200.100 :<0.15847.33>:xdc_vbucket_rep_xmem:do_flush_docs:116]out of 102 docs, succ to send 101 docs, fail to send others (by error type, enoent: 1, not-my-vb: 0, einval: 0, tmp fail: 0, enomem: 0, other errors: 0 [xdcr:error,2015-02-26T16:19:05.119, ns_1@10.121.200.100 :<0.15847.33>:xdc_vbucket_rep_worker:do_flush_docs_xmem:186]replication mode: "xmem", unable to replicate 102 docs to target " http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f6/ " [xdcr:error,2015-02-26T16:19:05.119, ns_1@10.121.200.100 :<0.28731.32>:xdc_vbucket_rep:terminate:499]Replication (XMem mode) `604f111b19d51891aed790af0ddd18c7/proxy_logs/proxy_logs` (`proxy_logs/6` -> ` http://*****@10.121.211.100:8092/proxy_logs%2B645855ffde9aca14f8d347d5eded4625%2f6 `) failed.Please see ns_server debug log for complete state dump [xdcr:error,2015-03-02T17:38:17.326, ns_1@10.121.200.100 :<0.1902.479>:xdc_vbucket_rep:terminate:499]Replication (XMem mode) `8f97c21845154726d73f0c99ecb6647e/fldb/fldb` (`fldb/501` -> ` http://*****@10.121.211.100:8092/fldb%2Ba9de84edc9032e8a2e0503dc3a414d4c%2f501 `) failed.Please see ns_server debug log for complete state dump [xdcr:error,2015-03-02T17:47:20.428, ns_1@10.121.200.100 :<0.31585.480>:xdc_vbucket_rep:terminate:499]Replication (XMem mode) `8f97c21845154726d73f0c99ecb6647e/fldb/fldb` (`fldb/456` -> ` http://*****@10.121.211.100:8092/fldb%2Ba9de84edc9032e8a2e0503dc3a414d4c%2f456 `) failed.Please see ns_server debug log for complete state dump
    • Unknown

    Description

      While XDCR is running from one of the clusters, we merge a few buckets into one (fldb + flow + nforce buckets turn into "fldb" only in the target). All replications work except those three.

      Tried to move to version 1 XDCR / restarting / recreating and nothing helped...

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Aliaksey Artamonau Aliaksey Artamonau (Inactive)
            arthur.gimpel arthur.gimpel
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty