Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5592

Rebalance fails with "mover_failed,{wrong_number_takeovers,0,1}}" on issuing a rebalance after failing over a Master node.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Blocker
    • 1.8.1
    • 1.8.1
    • ns_server
    • Security Level: Public
    • None
    • Large Cluster - Centos, - 16 node cluster
      Build 181-916rel
      3 buckets - bucket1(3G), bucket2(2.8G), bucket3(200M)

    Description

      Setup:
      1.Setup a 18 node cluster. Enable Auto-failover
      2.Load data on all 3 buckets [around 50M, 22M, 500k] items.
      3. Continue loading data..
      4. Failover orchestrator node [104]
      5. Issue rebalance on this cluster.

      Output
      1. Node 104 is failed over . As expected
      2. Rebalance operation fails withe error" mover_failed,{wrong_number_takeovers,0,1}}

      Output from web-log

      error_logger:error] [2012-06-15 15:30:32] [ns_1@10.3.2.89:error_logger:ale_error_logger_handler:log_report:72]
      =========================CRASH REPORT=========================
      crasher:
      initial call: ebucketmigrator_srv:init/1
      pid: <0.23836.1>
      registered_name: []
      exception exit: downstream_closed
      in function gen_server:terminate/6
      ancestors: ['ns_vbm_sup-bucket1','single_bucket_sup-bucket1',<0.470.0>]
      messages: []
      links: [<0.519.0>,<0.23837.1>]
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 2584
      stack_size: 24
      reductions: 13752727
      neighbours:

      [error_logger:error] [2012-06-15 15:30:32] [ns_1@10.3.2.89:error_logger:ale_error_logger_handler:log_report:72]
      =========================SUPERVISOR REPORT=========================
      Supervisor:

      {local,'ns_vbm_sup-bucket1'}

      Context: child_terminated
      Reason: downstream_closed
      Offender: [

      {pid,<0.23836.1>}

      ,
      {name,
      {child_id,
      [144,145,146,147,148,149,851],
      'ns_1@10.3.2.104'}},
      {mfargs,
      {ebucketmigrator_srv,start_link,
      [

      {"10.3.2.89",11209}

      ,

      {"10.3.2.104",11209}

      ,
      [

      {username,"bucket1"}

      ,

      {password,[]}

      ,

      {vbuckets,[144,145,146,147,148,149,851]}

      ,

      {takeover,false}

      ,

      {suffix,"ns_1@10.3.2.104"}

      ,

      {passed_downstream_retriever, #Fun<ns_vbm_sup.12.25558387>}

      ]]}},

      {restart_type,permanent}

      ,

      {shutdown,60000}

      ,

      {child_type,worker}

      ]

      Logs at https://s3.amazonaws.com/bugdb/jira/bug-rebalance-2/bug2.tar

      Checked existing bugs -Bug 5343 and this one, well it looked different to me.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            ketaki Ketaki Gangal (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty