Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39220

Couchbase-cli needs a way to perform hard failover without passing the unsafe flag

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.5.0, 6.0.0, 5.5.0, 5.5.1, 5.5.2, 5.5.3, 6.0.1, 6.0.2, 5.5.4, 6.0.3, 5.5.5, 5.5.6, 6.0.4, 6.5.1
    • Fix Version/s: 6.6.0
    • Component/s: tools
    • Triage:
      Triaged
    • Story Points:
      1

      Description

      Currently, to perform a hard failover using couchbase-cli failover, you must pass the --force flag.
      If you do not pass this flag, then the failover will be a graceful failover, which will obviously not work in situations where the node is down.

      As well as specifying hard failover, when you pass the --force flag it also passes the allowUnsafe=true as part of the REST request.
      Since Couchbase Server 5.5.0 and the introduction of a quorum based orchestration mechanism (via leases), there needs to be a majority quorum to be able to perform failovers by default.
      There are obviously cases where this is not possible (maybe the majority of nodes are irrecoverably down in the cluster), so ns_server has a mechanism to specify when performing a failover that you do not need it to wait for a quorum.
      Due to its nature this is inherently unsafe and may cause 'split brain' effects if the exact nature of the issue is not well understood.
      Most users will not want to use this allowUnsafe option and its use should generally be avoided unless absolutely necessary.

      From the perspective of using couchbase-cli failover though, the two concepts of hard failover and unsafe failover are both merged together with the --force flag.
      I think it would be useful to reconsider this API to be safer by default, while still allowing users to perform unsafe failovers if they are genuinely required.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-2223 contains cbas-core commit feab8e7 with commit message:
            MB-39220: update tests

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2223 contains cbas-core commit feab8e7 with commit message: MB-39220 : update tests
            Hide
            arunkumar Arunkumar Senthilnathan added a comment -

            Verified in 6.6.0-7795:

            [root@node1-mad-hatter-testing-centos7 bin]# ./couchbase-cli failover -c 10.112.194.101 -u Administrator -p password --server-failover 10.112.194.102 --hard
            SUCCESS: Server failed over
            [root@node1-mad-hatter-testing-centos7 bin]#

            Verified failover actually happened

            Show
            arunkumar Arunkumar Senthilnathan added a comment - Verified in 6.6.0-7795: [root@node1-mad-hatter-testing-centos7 bin] # ./couchbase-cli failover -c 10.112.194.101 -u Administrator -p password --server-failover 10.112.194.102 --hard SUCCESS: Server failed over [root@node1-mad-hatter-testing-centos7 bin] # Verified failover actually happened
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-2503 contains couchbase-cli commit ef82caf with commit message:
            MB-39220 Failover separates hard and force options

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2503 contains couchbase-cli commit ef82caf with commit message: MB-39220 Failover separates hard and force options
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-2506 contains cbas-core commit 6fbaad8 with commit message:
            MB-39220: update tests

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2506 contains cbas-core commit 6fbaad8 with commit message: MB-39220 : update tests
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9599 contains cbas-core commit feab8e7 with commit message:
            MB-39220: update tests

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9599 contains cbas-core commit feab8e7 with commit message: MB-39220 : update tests

              People

              Assignee:
              pvarley Patrick Varley
              Reporter:
              matt.carabine Matt Carabine
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty