Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60535

Rebalance exited with reason mover_crashed, unexpected_exit, 'EXIT',<0.31278.134>, wait_seqno_persisted_failed,"default",996

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • 7.6.0
    • test-execution
    •  Enterprise Edition 7.6.0 build 2032

    Description

      Steps:
      1. create a 4 nodes cluster

      Starting rebalance, KeepNodes = ['ns_1@172.23.104.236','ns_1@172.23.104.242',
      'ns_1@172.23.104.97','ns_1@172.23.105.123'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = b64d52a893b7d62b1a6673af1fff615b

      2. create an ephemeral bucket "default"
      3. load some data into the bucket
      4. rebalance out a node 

      Starting rebalance, KeepNodes = ['ns_1@172.23.104.236','ns_1@172.23.104.97', 'ns_1@172.23.105.123'], EjectNodes = ['ns_1@172.23.104.242'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 07752412dba0ec454e058efd3b9703d0

      5. during rebalance update half of the items and delete other half
      6. once rebalance completes , recreate all deleted items 
      7. rebalance out another node 

      Starting rebalance, KeepNodes = ['ns_1@172.23.104.236','ns_1@172.23.105.123'], EjectNodes = ['ns_1@172.23.104.97'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = f30938c219f83d62b96558f0a1184e10

      Observation

      [user:error,2024-01-25T02:16:59.855-08:00,ns_1@172.23.105.123:<0.6988.133>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {mover_crashed,
                                    {unexpected_exit,
                                     {'EXIT',<0.31278.134>,
                                      {{wait_seqno_persisted_failed,"default",996,
                                        965,
                                        [{'ns_1@172.23.104.236',
                                          {'EXIT',
                                           {noproc,
                                            {gen_server,call,
                                             [{'janitor_agent-default',
                                               'ns_1@172.23.104.236'},
                                              {if_rebalance,<0.23427.134>,
                                               {wait_seqno_persisted,996,965}},
                                              infinity]}}}}]},
                                       [{ns_single_vbucket_mover,
                                         '-wait_seqno_persisted_many/5-fun-2-',5,
                                         [{file,"src/ns_single_vbucket_mover.erl"},
                                          {line,476}]},
                                        {proc_lib,init_p,3,
                                         [{file,"proc_lib.erl"},{line,225}]}]}}}}.
      Rebalance Operation Id = f30938c219f83d62b96558f0a1184e10
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pulkit.matta Pulkit Matta
              pulkit.matta Pulkit Matta
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty