Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45698

ns-server won't start on some nodes after upgrade

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • ns_server
    • CB EE 6.0.0-9591 & 7.0.0-4947

    Description

      Summary:
      Not sure what I'm missing here. So was doing the following:
      1. .215, .217, .219 on 6.6.2 (3 node cluster). 
      2. .220, .237, .90 on 7.0
      3. On 6.6 cluster, load travel-sample
      4. Add .220 and remove .219 and rebalance from 6.6 UI (on .215)
      5. Add .237 and remove .217 and rebalance from 6.6 UI (on .215)
      6. Add .90 and remove .215 and rebalance from .220 node's UI

      Rebalance goes fine. But just after rebalance, observed that ns-server won't start on .237, .90
      on .237 debug.log

      =========================CRASH REPORT=========================
        crasher:
          initial call: leader_quorum_nodes_manager:init/1
          pid: <0.664.0>
          registered_name: leader_quorum_nodes_manager
          exception exit: {quorum_nodes_update_conflict,
                              ['ns_1@cb.local'],
                              ['ns_1@172.23.106.237']}

      fgrep exception * on .237

      debug.log:    exception exit: noproc
      debug.log:    exception exit: noproc
      debug.log:    exception exit: {failed_to_start_ns_server,
      debug.log:    exception exit: noproc
      debug.log:    exception exit: noproc

      on .220 (possibly due to consequence of nodes .237, 90 not reachable)

      [ns_server:error,2021-04-15T10:41:40.799-07:00,ns_1@172.23.105.220:<0.1525.8>:leader_activities:report_error:992]Activity {default,{service_janitor,kv,maybe_complete_pending_failover}} failed with error {no_quorum,
                                                                                                 [{required_quorum,
                                                                                                   majority},
                                                                                                  {leases,
                                                                                                   ['ns_1@172.23.105.220']}]}

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              artem Artem Stemkovski
              sumedh.basarkod Sumedh Basarkod (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty