Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7382

rebalance froze when node failed over and added back (observed mem used > high water mark for bucket)

    Details

      Description

      • 2 node cluster
      • 2 buckets
      • Bucket 'bkt' had a very high percentage of sets in its front end load.
      • Failed over ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com, added back, rebalance.
      • Rebalance froze at around 98%.
      • Stopped front end loads, disk write queue drained.
      • Mem used for both nodes, greater than higher water mark.
      • Restarted couchbase server, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.
      • Rebooted nodes, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.

      Cluster diags:
      1 https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

      2 https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

      Attached the cbstats of all, raw memory for both nodes.

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            • Assignee:
              abhinav Abhinav Dangeti
              Reporter:
              abhinav Abhinav Dangeti
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes