Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60535

Rebalance exited with reason mover_crashed, unexpected_exit, 'EXIT',<0.31278.134>, wait_seqno_persisted_failed,"default",996

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • 7.6.0
    • test-execution
    •  Enterprise Edition 7.6.0 build 2032

    Description

      Steps:
      1. create a 4 nodes cluster

      Starting rebalance, KeepNodes = ['ns_1@172.23.104.236','ns_1@172.23.104.242',
      		 
      'ns_1@172.23.104.97','ns_1@172.23.105.123'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = b64d52a893b7d62b1a6673af1fff615b

      2. create an ephemeral bucket "default"
      3. load some data into the bucket
      4. rebalance out a node 

      Starting rebalance, KeepNodes = ['ns_1@172.23.104.236','ns_1@172.23.104.97', 'ns_1@172.23.105.123'], EjectNodes = ['ns_1@172.23.104.242'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 07752412dba0ec454e058efd3b9703d0

      5. during rebalance update half of the items and delete other half
      6. once rebalance completes , recreate all deleted items 
      7. rebalance out another node 

      Starting rebalance, KeepNodes = ['ns_1@172.23.104.236','ns_1@172.23.105.123'], EjectNodes = ['ns_1@172.23.104.97'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = f30938c219f83d62b96558f0a1184e10

      Observation

      [user:error,2024-01-25T02:16:59.855-08:00,ns_1@172.23.105.123:<0.6988.133>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {mover_crashed,
      		 
                                    {unexpected_exit,
      		 
                                     {'EXIT',<0.31278.134>,
      		 
                                      {{wait_seqno_persisted_failed,"default",996,
      		 
                                        965,
      		 
                                        [{'ns_1@172.23.104.236',
      		 
                                          {'EXIT',
      		 
                                           {noproc,
      		 
                                            {gen_server,call,
      		 
                                             [{'janitor_agent-default',
      		 
                                               'ns_1@172.23.104.236'},
      		 
                                              {if_rebalance,<0.23427.134>,
      		 
                                               {wait_seqno_persisted,996,965}},
      		 
                                              infinity]}}}}]},
      		 
                                       [{ns_single_vbucket_mover,
      		 
                                         '-wait_seqno_persisted_many/5-fun-2-',5,
      		 
                                         [{file,"src/ns_single_vbucket_mover.erl"},
      		 
                                          {line,476}]},
      		 
                                        {proc_lib,init_p,3,
      		 
                                         [{file,"proc_lib.erl"},{line,225}]}]}}}}.
      		 
      Rebalance Operation Id = f30938c219f83d62b96558f0a1184e10

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-59721 [Rebalance] Rebalance failed with reason {mover_crashed,{{{{{badmatch,[{<0.29601.0>,{done,exit,{socket_closed,{gen_server,call,[<0.19270.0>,{setup_streams,

          • Major - Major loss of function.
          • Closed

          Activity

            People

              pulkit.matta Pulkit Matta
              pulkit.matta Pulkit Matta
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty