Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59467

[Rebalance] [30 buckets] Rebalance exited with reason {mover_crashed,{unexpected_exit,{'EXIT',<0.1338.56>,{{bulk_set_vbucket_state_failed, [{'ns_1@172.23.104.181',{'EXIT', {{badmatch,wrong_rebalancer_pid}

    XMLWordPrintable

Details

    Description

      Steps to reproduce

      1. Created a 4 node cluster

      2. Created 30 buckets with 3 scopes and 11 collections in each scope

      3. Rebalancing in a node with services kv, index

      4. Deleted 5 buckets, changed rank of all buckets and created 5 new buckets. The buckets after all these operations

      5.  Rebalancing in a node with services kv, n1ql

      6. Deleted 5 buckets, changed rank of all buckets and created 5 new buckets. The buckets after all these operations

      7. Rebalanced in a node with services kv, fts

      8. Deleted 5 buckets, changed rank of all buckets and created 5 new buckets. The buckets after all these operations

      9. Rebalanced in a node with services kv, cbas

      10. Deleted 5 buckets, changed rank of all buckets and created 5 new buckets. The buckets after all these operations

      11. Rebalanced in a kv node

      Rebalance exited with reason {mover_crashed, {unexpected_exit, {'EXIT',<0.1338.56>, 

      2023-11-05T12:59:17.661-08:00, ns_vbucket_mover:0:critical:message(ns_1@172.23.122.93) - Worker <0.1253.56> (for action {move,{635,                                      ['ns_1@172.23.122.91',                                       'ns_1@172.23.122.90',                                       'ns_1@172.23.104.226',                                       'ns_1@172.23.104.181'],                                      ['ns_1@172.23.122.91',                                       'ns_1@172.23.104.181',                                       'ns_1@172.23.104.226',                                       'ns_1@172.23.122.90'],                                      []}}) exited with reason {unexpected_exit,                                                                {'EXIT',                                                                 <0.1338.56>,                                                                 {{bulk_set_vbucket_state_failed,                                                                   [{'ns_1@172.23.104.181',                                                                     {'EXIT',                                                                      {{badmatch,                                                                        wrong_rebalancer_pid},                                                                       [{janitor_agent,                                                                         set_vbucket_state_inner,                                                                         5,                                                                         [{file,                                                                           "src/janitor_agent.erl"},                                                                          {line,                                                                           426}]},                                                                        {janitor_agent,                                                                         '-bulk_set_vbucket_state/4-fun-0-',                                                                         4,                                                                         [{file,                                                                           "src/janitor_agent.erl"},                                                                          {line,                                                                           396}]},                                                                        {async,                                                                         '-async_init/4-fun-1-',                                                                         3,                                                                         [{file,                                                                           "src/async.erl"},                                                                          {line,                                                                           199}]}]}}}]},                                                                  [{janitor_agent,                                                                    bulk_set_vbucket_state,                                                                    4,                                                                    [{file,                                                                      "src/janitor_agent.erl"},                                                                     {line,                                                                      408}]},                                                                   {proc_lib,                                                                    init_p,3,                                                                    [{file,                                                                      "proc_lib.erl"},                                                                     {line,                                                                      225}]}]}}}2023-11-05T12:59:17.722-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.122.93) - Rebalance exited with reason {mover_crashed,                              {unexpected_exit,                               {'EXIT',<0.1338.56>,                                {{bulk_set_vbucket_state_failed,                                  [{'ns_1@172.23.104.181',                                    {'EXIT',                                     {{badmatch,wrong_rebalancer_pid},                                      [{janitor_agent,                                        set_vbucket_state_inner,5,                                        [{file,"src/janitor_agent.erl"},                                         {line,426}]},                                       {janitor_agent,                                        '-bulk_set_vbucket_state/4-fun-0-',4,                                        [{file,"src/janitor_agent.erl"},                                         {line,396}]},                                       {async,'-async_init/4-fun-1-',3,                                        [{file,"src/async.erl"},                                         {line,199}]}]}}}]},                                 [{janitor_agent,bulk_set_vbucket_state,4,                                   [{file,"src/janitor_agent.erl"},                                    {line,408}]},                                  {proc_lib,init_p,3,                                   [{file,"proc_lib.erl"},{line,225}]}]}}}}.Rebalance Operation Id = 6712241f1e90641b4495ad7865ca9ff9 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              raghav.sk Raghav S K
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty