Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-15082

DCP enters deadlock, no inter/intra cluster replication from one node in a cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 4.0.0
    • 4.0.0
    • couchbase-bucket
    • Security Level: Public
    • None

    Description

      Build


      4.0.0-2206

      Observed the following during an xdcr test between 2 clusters:
      1. While comparing standard_bucket: same size (5GB per node) on two clusters, one with value and other with full eviction, saw that
      a. the bucket with value eviction had lesser resident ratio(~13%), more keys and lesser memory usage than the bucket with full eviction (see screenshot)
      b. A large number of tmp OOMs were seen from the cluster with full eviction but no tmp-OOMs were seen from the other cluster(with value eviction)
      c. When I further tried digging into memory usage on the bucket with full eviction, saw temp-OOMs from just one node: 172.23.105.57. The memory usage was beyond high watermark for just this node in cluster. See screenshot.
      d. Active resident ratio was ~82% on this node while rest of the nodes in the cluster were around ~38%
      e. The node was basically emitting tmp-OOMs to replica node, XDCR was not active either due to high memory usage of memcached. See screenshot.

      Setup

      C1: 8 node cluster - http://172.23.105.44:8091/index.html (live)
      C2: 8 node cluster - http://172.23.105.54:8091/index.html (live) <-- contains .57

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              apiravi Aruna Piravi (Inactive)
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty