Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51355

Backup: Rebalance failed with reason "could not confirm self removed: could not remove self: exhausted retry count after 5 attempts: node status is not out"

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • None
    • 7.1.0
    • tools
    • Enterprise Edition 7.1.0 build 2434

    Description

      Build: 7.1.0 - 2434

      Unable to rebalance out nodes involving backup service

      Rebalance exited with reason {service_rebalance_failed,backup,
      		 
      {worker_died,
      		 
      {'EXIT',<0.3048.33>,
      		 
      {rebalance_failed,
      		 
      {service_error,
      		 
      <<"could not confirm self removed: could not remove self: exhausted retry count after 5 attempts: node status is not out">>}}}}}.
      		 
      Rebalance Operation Id = e315afe1695428616eeded3e557e0f17

      Initially rebalance failed with the reason

      Rebalance exited with reason {service_rebalance_failed,backup,
      		 
      {{badmatch, {error, {bad_nodes,backup,get_agent,
      		 
      [{'ns_1@172.23.136.111', {exit,
      		 
      {{timeout, {gen_server,call,
      		 
      [<35063.7400.0>, {call,"ServiceAPI.CancelTask",
      		 
      #Fun<json_rpc_connection.0.86436583>,
      		 
      #{timeout => 60000}}, 60000]}},
      		 
      {gen_server,call,
      		 
      [{'service_agent-backup', 'ns_1@172.23.136.111'},
      		 
      get_agent,infinity]}}}}]}}},
      		 
      [{service_rebalancer,wait_for_agents,1,
      		 
      [{file,"src/service_rebalancer.erl"}, {line,80}]},
      		 
      {service_rebalancer,run_rebalance,1,
      		 
      [{file,"src/service_rebalancer.erl"}, {line,59}]},
      		 
      {proc_lib,init_p,3, {file,"proc_lib.erl"},{line,211}]}]}}.
      		 
      Rebalance Operation Id = cfdfd36843cfb2e7ecd58e0be21366da

      Following this all rebalances are failing.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-53559 [CBBS] Rebalance fails if previous leader was failed over

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty