Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10398

Rebalance exited while adding nodes in cluster with reason {mover_failed,{badmatch,{error,timeout}}}

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • Triaged
    • Centos 64-bit
    • 2
    • Yes
    • June 30 - July 18

    Description

      Build : 3.0.0-416-rel

      Test Failed:
      ./testrunner -i /tmp/new-viewtests-all.ini get-cbcollect-info=True,get-delays=True -t view.createdeleteview.CreateDeleteViewTests.pending_removal_with_ddoc_ops,ddoc_ops=delete,test_with_view=False,num_ddocs=4,num_views_per_ddoc=3,items=200000,nodes_out=1

      • Note : this test cannot deterministically produce this error but observed this failure twice

      Jenkins Job:
      http://qa.hq.northscale.net/view/3.0.0/job/centos_x64--29_01--new_view_all-P1/48/consoleFull

      Logs:
      [ns_server:error,2014-03-07T7:48:07.193,ns_1@10.1.3.74:<0.10255.22>:misc:try_with_maybe_ignorant_after:1535]Eating exception from ignorant after-block:
      {error,
      {badmatch,
      [{<0.10264.22>,
      {{badmatch,[

      {<18152.21537.5>,noproc}]},
      [{misc,sync_shutdown_many_i_am_trapping_exits,1},
      {misc,try_with_maybe_ignorant_after,2},
      {gen_server,terminate,6},
      {proc_lib,init_p_do_apply,3}]}},
      {<0.10267.22>,
      {badmatch,
      [{'EXIT',
      {normal,
      {gen_server,call, [<18152.21537.5>,had_backfill,infinity]}}},
      {'EXIT',
      {shutdown,
      {gen_server,call, [<18597.4662.5>,had_backfill,infinity]}}}]}}]},
      [{misc,sync_shutdown_many_i_am_trapping_exits,1},
      {misc,try_with_maybe_ignorant_after,2},
      {ns_single_vbucket_mover,mover,6},
      {proc_lib,init_p_do_apply,3}]}
      [error_logger:error,2014-03-07T7:48:07.194,ns_1@10.1.3.74:error_logger<0.6.0>:ale_error_logger_handler:log_report:115]
      =========================CRASH REPORT=========================
      crasher:
      initial call: new_ns_replicas_builder:init/1
      pid: <0.10264.22>
      registered_name: []
      exception exit: {{badmatch,[{<18152.21537.5>,noproc}

      ]},
      [

      {misc,sync_shutdown_many_i_am_trapping_exits,1}

      ,

      {misc,try_with_maybe_ignorant_after,2}

      ,

      {gen_server,terminate,6}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}
      in function gen_server:terminate/6
      ancestors: [<0.10255.22>,<0.10108.22>,<0.10052.22>,<0.4818.0>,
      mb_master_sup,mb_master,ns_server_sup,ns_server_cluster_sup,
      <0.59.0>]
      messages: [

      {'EXIT',<0.10255.22>,shutdown}

      ,

      {'EXIT',<0.11398.22>,normal}

      ]
      links: <0.10255.22>,#Port<0.469502>
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 317811
      stack_size: 24
      reductions: 30548
      neighbours:

      [ns_server:debug,2014-03-07T7:48:07.202,ns_1@10.1.3.74:<0.10072.22>:ns_pubsub:do_subscribe_link:136]Parent process of subscription

      {master_activity_events,<0.10071.22>}

      exited with reason {mover_failed,
      {badmatch,

      {error, timeout}

      }}
      [user:info,2014-03-07T7:48:07.202,ns_1@10.1.3.74:<0.4818.0>:ns_orchestrator:handle_info:469]Rebalance exited with reason {mover_failed,{badmatch,

      {error,timeout}

      }}

      [ns_server:debug,2014-03-07T7:48:07.203,ns_1@10.1.3.74:compaction_daemon<0.482.0>:compaction_daemon:handle_info:456]Looks like vbucket mover inhibiting view compaction for for bucket "default" is dead. Canceling inhibition
      [ns_server:info,2014-03-07T7:48:07.203,ns_1@10.1.3.74:janitor_agent-default<0.3732.22>:janitor_agent:handle_info:750]Undoing temporary vbucket states caused by rebalance
      [ns_server:debug,2014-03-07T7:48:07.203,ns_1@10.1.3.74:<0.10117.22>:ns_pubsub:do_subscribe_link:136]Parent process of subscription

      {ns_node_disco_events,<0.10108.22>}

      exited with reason {mover_failed,
      {badmatch,

      {error, timeout}

      }}
      [error_logger:error,2014-03-07T7:48:07.204,ns_1@10.1.3.74:error_logger<0.6.0>:ale_error_logger_handler:log_report:115]
      =========================CRASH REPORT=========================
      crasher:
      initial call: ns_single_vbucket_mover:mover/6
      pid: <0.10255.22>
      registered_name: []
      exception exit: {unexpected_exit,
      {'EXIT',<0.10267.22>,
      {badmatch,
      [{'EXIT',
      {normal,

      {gen_server,call, [<18152.21537.5>,had_backfill,infinity]}

      }},
      {'EXIT',
      {shutdown,

      {gen_server,call, [<18597.4662.5>,had_backfill,infinity]}

      }}]}}}
      in function ns_single_vbucket_mover:spawn_and_wait/1
      in call from ns_single_vbucket_mover:mover_inner/5
      in call from misc:try_with_maybe_ignorant_after/2
      in call from ns_single_vbucket_mover:mover/6
      ancestors: [<0.10108.22>,<0.10052.22>,<0.4818.0>,mb_master_sup,
      mb_master,ns_server_sup,ns_server_cluster_sup,<0.59.0>]
      messages: [{'EXIT',<0.10108.22>,
      {mover_failed,{badmatch,{error,timeout}}}}]
      links: [<0.10108.22>]
      dictionary: [

      {cleanup_list,[<0.10264.22>,<0.10267.22>]}

      ]
      trap_exit: true
      status: running
      heap_size: 2584
      stack_size: 24
      reductions: 7991
      neighbours:

      Uploading Logs.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Meenakshi Meenakshi Goel
              Meenakshi Meenakshi Goel
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty