Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11605

XDCR replication (source to destination cluster) hangs when a hard OOM is hit.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Blocker
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • Untriaged
    • Ubuntu 64-bit
    • Yes
    • June 30 - July 18

    Description

      Uni-directional XDCR replication gets stuck if the source cluster hits hard out-of-memory (OOM) prior to the start of XDCR replication.
      Workaround: It is recoverable by increasing the bucket RAM quota on the source cluster.

      Steps to reproduce:

      1)Setup uni-directional xdcr with two buckets in source and destination clusters. The buckets are named "high"(high priority bucket) and "low"(low priority bucket).
      2)Both source buckets have ~600k items.
      3)XDCR is stuck for both the buckets and I see the following messages in babysitter.log in one case
      start seqno< end seqno and in another, end seqno is 0.

      stream created with start seqno 0 and end seqno 0
      memcached<0.77.0>: Tue Jul 1 11:47:57.540429 PDT 3: (high) UPR (Notifier) eq_uprq:xdcr:notifier:ns_1@172.23.105.239:high - (vb 93) stream created with start seqno 0 and end seqno 0
      memcached<0.77.0>: Tue Jul 1 11:47:57.560008 PDT 3: (high) UPR (Notifier) eq_uprq:xdcr:notifier:ns_1@172.23.105.239:high - (vb 440) stream created with start seqno 0 and end seqno 0

      memcached<0.77.0>: Tue Jul 1 11:56:04.169444 PDT 3: (low) UPR (Notifier) eq_uprq:xdcr:notifier:ns_1@172.23.105.239:low - (vb 88) stream created with start seqno 691 and end seqno 0

      Attachments

        1. babysitter.log
          5.13 MB
        2. node1_1
          47.68 MB
        3. node1_2
          46.33 MB
        4. node2_1
          47.68 MB
        5. node2_2
          41.29 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mikew Mike Wiederhold [X] (Inactive)
            venu Venu Uppalapati (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty