Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51482

[BP 7.0.4] Node is not upgraded and stuck trying to register with CC

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes
    • CX Sprint 284

    Description

      During upgrading nodes, a node could get into a state where the monitor for the cluster compatibility is not started because the coordinator on the node is not started. If the monitor for the cluster compatibility is not started, the node won't detect that the cluster compatibility has changed and act on it.

      The attached logs show that the node could not finish booting and registering with the CC as it was still running an older version than the CC, and therefore, the coordinator was not started.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.0.4-7222 contains cbas commit f4d74f1 with commit message:
            MB-51482: [BP 7.0.4] monitor cluster compat / DP changes before driver bootstrap

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.4-7222 contains cbas commit f4d74f1 with commit message: MB-51482 : [BP 7.0.4] monitor cluster compat / DP changes before driver bootstrap

            Build couchbase-server-7.0.4-7222 contains cbas-core commit 5658938 with commit message:
            MB-51482: [BP 7.0.4] update test results

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.4-7222 contains cbas-core commit 5658938 with commit message: MB-51482 : [BP 7.0.4] update test results
            michael.blow Michael Blow added a comment -

            This causes non-upgraded binaries on the NC(s) to be used with upgraded binaries on the CC upon node upgrade - leading to the driver restart loop. The only way to stop the loop is to restart Couchbase Server or killall -9 cbas on all affected nodes. SSH access to the node is needed to do this.

            michael.blow Michael Blow added a comment - This causes non-upgraded binaries on the NC(s) to be used with upgraded binaries on the CC upon node upgrade - leading to the driver restart loop. The only way to stop the loop is to restart Couchbase Server or killall -9 cbas on all affected nodes. SSH access to the node is needed to do this.
            umang.agrawal Umang added a comment -

            Ali AlsulimanCan you provide steps to verify this?

            umang.agrawal Umang added a comment - Ali Alsuliman Can you provide steps to verify this?

            This is covered by the dev tests (as was the original Neo issue).

            till Till Westmann added a comment - This is covered by the dev tests (as was the original Neo issue).

            Build couchbase-server-7.1.1-3024 contains cbas-core commit 5658938 with commit message:
            MB-51482: [BP 7.0.4] update test results

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.1-3024 contains cbas-core commit 5658938 with commit message: MB-51482 : [BP 7.0.4] update test results

            Build couchbase-server-7.2.0-1100 contains cbas-core commit 5658938 with commit message:
            MB-51482: [BP 7.0.4] update test results

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-1100 contains cbas-core commit 5658938 with commit message: MB-51482 : [BP 7.0.4] update test results

            People

              ali.alsuliman Ali Alsuliman
              ali.alsuliman Ali Alsuliman
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty