Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-16447

Graceful failover failed in a highly scalable scenario of 128 data nodes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 5.0.0
    • 4.0.0
    • couchbase-bucket, DCP
    • Security Level: Public
    • 128 data nodes, 1 index node 1 query node
      640M 1K docs, 2Million writes / second (~1K per VPC)
      data stored on local SSD
      index (not used) also stored on local SSD

    Description

      Selected data node and clicked Failover button (graceful failover).

      Failed to complete. Left with following message

      "Fail Over Warning: Rebalance recommended, some data does not have the desired replicas configuration!"

      And in a"pending rebalance" state. However on clicking the "pending rebalance" button - it states "There are no servers pending rebalance."

      Have uploaded the logs of all nodes to S3
      https://s3.amazonaws.com/cb-support/MB-16447/

      I believe the following is the master node

      https://s3.amazonaws.com/cb-support/MB-16447/collectinfo-2015-10-02T180015-ns_1%40ip-10-0-44-197.eu-west-1.compute.internal.zip

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. MB-12116 KV Scalability

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ajay.bhullar Ajay Bhullar
              owend Daniel Owen
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty