Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29692

Unexpected warning from attempted failover for failed node

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • Morpheus
    • 5.1.0
    • ns_server
    • None
    • Triaged
    • Yes
    • UI - 2021 - Nov, UI 2021-Dec, UI 2022-Jan

    Description

      One of the VMs we use for our build_database nodes happened to be on a Xen host that had a fatal disk failure.  The configuration has 3 nodes and 2 replicas.  Attempting to do a failover from the failed node gave this warning:

      Attention: There are not replica (backup) copies of all data on this node! Failing over the node now will irrecoverably lose that data when the incomplete replica is activated and this node is removed from the cluster. If the node might come back online, it is recommended to wait. Check this box if you want to failover the node, despite the resulting data loss.

      Before replacing the node with a newly built one and rebalancing, cbcollect info was gathered from the two remaining nodes.  The data for each can be found here:

      http://latestbuilds.service.couchbase.com/builds/latestbuilds/cbcollect/collectinfo-2018-05-15T021555-ns_1@172.23.106.53.zip

      http://latestbuilds.service.couchbase.com/builds/latestbuilds/cbcollect/collectinfo-2018-05-15T021555-ns_1@172.23.120.12.zip

      After putting a new 3rd node in place and rebalancing, there seemed to be no data loss (still verifying this by building out a new database from our manifests to see a document count), so the warning seems a bit surprising and may require some analysis for validity?

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Abhijeeth.Nuthan Abhijeeth Nuthan
            kenneth.lareau Kenneth Lareau (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty