Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55191

Online upgrade using graceful failover/delta recovery/rebalance fails during rebalance of eventing service

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • Morpheus
    • 6.6.6
    • eventing
    • 6.5.2-6634 —>  6.6.6-10576
    • Untriaged
    • Centos 64-bit
    • 0
    • No

    Description

      Steps to Repro
      1. Create a 3 node 6.5.2-6634 cluster(172.23.107.222,172.23.107.223,172.23.107.217)
      2. do a swap rebalance of 6.6.6-10576 node(172.23.107.99) with 6.5.2-6634(172.23.107.223) node. This succeeds.
      3. Graceful failover a 6.5.2-6634(172.23.107.217) node. Upgrade the build to 6.6.6-10576. Do a delta recovery and click on rebalance. This rebalance fails continuously on retry as shown below.

      172.23.107.99 11:00:26 PM 19 Jan, 2023

      Rebalance exited with reason {service_rebalance_failed,eventing,
      {worker_died,
      {'EXIT',<0.2411.13>,
      {{badmatch,
      {error,
      {bad_nodes,eventing,prepare_rebalance,
      [{'ns_1@172.23.107.99',
      {error,
      {unknown_error,
      <<"Eventing Rebalance or Failover processing ongoing on nodeId: f368ef74e474bcd6adbb0a9d48b1102a">>}}}]}}},
      [{service_rebalancer,rebalance_worker,1,
      [{file,"src/service_rebalancer.erl"},
      {line,164}]},
      {proc_lib,init_p,3,
      [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Rebalance Operation Id = c7870e66deb80aa1c72d64fe66321e5a
      

      Retried failed rebalance multiple times. Keeps failing.
      cbcollect_info attached.

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-55191
          # Subject Branch Project Status CR V

          Activity

            People

              ankit.prabhu Ankit Prabhu
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty