Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10908

beam.smp RSS grows to 50GB during delta recovery causing OOM killer invocation and rebalance failure

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 3.0
    • 3.0
    • ns_server, view-engine
    • Security Level: Public
    • Builds 3.0.0-585+

      Platform = Physical
      OS = CentOS 6.5
      CPU = Intel Xeon E5-2630
      Memory = 64 GB
      Disk = 2 x SSD

    Description

      Delta rebalance after failover, 3 -> 4 nodes, 1 bucket x 100M x 2KB, DGM, 1 ddoc with 1 view, 10K mixed ops/sec, 400 qps

      Steps:
      1. "Failover" one node.
      2. Add it back.
      3. Enable delta recovery mode.
      4. Wait predefined time (20 minutes).
      5. Trigger cluster rebalance, wait for rebalance to finish.

      Attachments

        1. beam.smp_rss_594.png
          38 kB
          Pavel Paulau
        2. beam.smp_rss.png
          39 kB
          Pavel Paulau

        Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. MB-10472 Handle multiple snapshots properly in view engine

          • Test Blocker - QE-related tests can't proceed
          • Resolved
          is triggered by

          Bug - A problem which impairs or prevents the functions of the product. MB-10514 During rebalance, UPR stream gets stuck after sending a snapshot marker and does not send any further mutations for that stream.

          • Test Blocker - QE-related tests can't proceed
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavelpaulau Pavel Paulau (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty