Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-15082

DCP enters deadlock, no inter/intra cluster replication from one node in a cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 4.0.0
    • 4.0.0
    • couchbase-bucket
    • Security Level: Public
    • None

    Description

      Build

      4.0.0-2206

      Observed the following during an xdcr test between 2 clusters:
      1. While comparing standard_bucket: same size (5GB per node) on two clusters, one with value and other with full eviction, saw that
      a. the bucket with value eviction had lesser resident ratio(~13%), more keys and lesser memory usage than the bucket with full eviction (see screenshot)
      b. A large number of tmp OOMs were seen from the cluster with full eviction but no tmp-OOMs were seen from the other cluster(with value eviction)
      c. When I further tried digging into memory usage on the bucket with full eviction, saw temp-OOMs from just one node: 172.23.105.57. The memory usage was beyond high watermark for just this node in cluster. See screenshot.
      d. Active resident ratio was ~82% on this node while rest of the nodes in the cluster were around ~38%
      e. The node was basically emitting tmp-OOMs to replica node, XDCR was not active either due to high memory usage of memcached. See screenshot.

      Setup

      C1: 8 node cluster - http://172.23.105.44:8091/index.html (live)
      C2: 8 node cluster - http://172.23.105.54:8091/index.html (live) <-- contains .57

      Attachments

        1. screenshot_xdcr_changes_left.png
          195 kB
          Abhi Dangeti
        2. Screen Shot 2015-05-20 at 1.55.32 PM.png
          588 kB
          Aruna Piravi
        3. Screen Shot 2015-05-20 at 1.58.58 PM.png
          131 kB
          Aruna Piravi
        4. Screen Shot 2015-05-20 at 2.00.02 PM.png
          335 kB
          Aruna Piravi
        5. Screen Shot 2015-05-20 at 2.23.43 PM.png
          206 kB
          Aruna Piravi
        6. Screen Shot 2015-05-20 at 3.00.28 PM.png
          88 kB
          Aruna Piravi

        Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. MB-14187 GoXDCR: No errors recorded in goxdcr.log/UI when replication is incomplete

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MB-9897 Implement DCP cursor dropping

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is triggering

          Bug - A problem which impairs or prevents the functions of the product. MB-15085 Active resident ratio% doesn't go down after full eviction on one of the nodes

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Task - A task that needs to be done. MB-9897 Implement DCP cursor dropping

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-15706 [system-test] GoXDCR: DCP stuck for > 13 mins

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              apiravi Aruna Piravi (Inactive)
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty