Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5433

Rebalance hang in large cluster test with "mem_used" stat underflow issue

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 1.8.1
    • 1.8.1
    • couchbase-bucket
    • Security Level: Public
    • None
    • centos 5.4 64bit

    Description

      Install couchbase server 1.8.1-847 on 12 node in ec2 with 7.5 GB RAM and 100 GB disk
      Create a 10 nodes cluster and load 40 million data into default bucket with size from 256 to 512 bytes
      Maintain load around 7 K ops
      Do rebalance in and out in large cluster test
      In one rebalance test, rebalance hang near complete with error:
      Hard Out Of Memory Error. Bucket "default"

      Diags of cluster:

      https://s3.amazonaws.com/packages.couchbase/sandbox/ec2-23-20-53-38.compute-1.amazonaws.com-8091-diag.txt.gz
      https://s3.amazonaws.com/packages.couchbase/sandbox/ec2-23-22-45-111.compute-1.amazonaws.com-8091-diag.txt.gz
      https://s3.amazonaws.com/packages.couchbase/sandbox/ec2-50-16-17-17.compute-1.amazonaws.com-8091-diag.txt.gz
      https://s3.amazonaws.com/packages.couchbase/sandbox/ec2-67-202-49-238.compute-1.amazonaws.com-8091-diag.txt.gz
      https://s3.amazonaws.com/packages.couchbase/sandbox/ec2-72-44-47-164.compute-1.amazonaws.com-8091-diag.txt.gz
      https://s3.amazonaws.com/packages.couchbase/sandbox/ec2-107-21-88-105.compute-1.amazonaws.com-8091-diag.txt.gz
      https://s3.amazonaws.com/packages.couchbase/sandbox/ec2-174-129-130-202.compute-1.amazonaws.com-8091-diag.txt.gz
      https://s3.amazonaws.com/packages.couchbase/sandbox/ec2-184-72-67-54.compute-1.amazonaws.com-8091-diag.txt.gz
      https://s3.amazonaws.com/packages.couchbase/sandbox/ec2-184-73-111-82.compute-1.amazonaws.com-8091-diag.txt.gz
      https://s3.amazonaws.com/packages.couchbase/sandbox/ec2-184-73-112-165.compute-1.amazonaws.com-8091-diag.txt.gz
      https://s3.amazonaws.com/packages.couchbase/sandbox/ec2-23-21-17-242.compute-1.amazonaws.com-8091-diag.txt.gz

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mikew Mike Wiederhold [X] (Inactive)
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty