Rebalance exiting with reason {{badmatch,{error,timeout}} during start/stop rebalance operation

Build : 3.0.0-419-rel

*Note: Rerun of the same tests might node reproduce same error always, Observed the same failure in two different tests where in one node is getting added to cluster while in other master node is being removed from cluster.

Jenkins Job for reference:
http://qa.hq.northscale.net/view/3.0.0/job/ubuntu_x64--35_02--view_query_extended-P1/34/consoleFull
http://qa.hq.northscale.net/view/3.0.0/job/centos_x64--29_01--new_view_all-P1/56/consoleFull

Tests Failed:
./testrunner -i /tmp/ubuntu-rebalance.ini get-cbcollect-info=True,num-tries=100,get-delays=True,GROUP=P1 -t view.viewquerytests.ViewQueryTests.test_employee_dataset_alldocs_queries_start_stop_rebalance_in_incremental,limit=10000,skip_rebalance=true,GROUP=P1
./testrunner -i /tmp/new-viewtests-all.ini get-cbcollect-info=True,get-delays=True -t view.createdeleteview.CreateDeleteViewTests.ddoc_ops_removing_master,ddoc_ops=create,test_with_view=True,num_ddocs=4,num_views_per_ddoc=3,items=200000

Logs:
[error_logger:error,2014-03-10T23:49:34.171,ns_1@10.3.3.28:error_logger<0.6.0>:ale_error_logger_handler:log_msg:119]** Generic server <0.3693.53> terminating

• • Last message in was {'EXIT',<0.3687.53>,shutdown}
  • When Server state ==
    Unknown macro: {state,"default",59,'ns_1@10.3.3.28', [{'ns_1@10.3.3.34',<19063.30939.32>}]}
  • Reason for termination ==
  • badmatch,{error,timeout,
    [ {ns_replicas_builder_utils,kill_a_bunch_of_tap_names,3},
    {misc,try_with_maybe_ignorant_after,2},
    {gen_server,terminate,6},
    {proc_lib,init_p_do_apply,3}]}
    
    [ns_server:error,2014-03-10T23:49:34.171,ns_1@10.3.3.28:<0.3687.53>:misc:sync_shutdown_many_i_am_trapping_exits:1499]Shutdown of the following failed: [{<0.3693.53>,
    badmatch,{error,timeout,
    [{ns_replicas_builder_utils, kill_a_bunch_of_tap_names,3},
    {misc,try_with_maybe_ignorant_after,2},
    {gen_server,terminate,6},
    {proc_lib,init_p_do_apply,3}]}}]
    [ns_server:error,2014-03-10T23:49:34.171,ns_1@10.3.3.28:<0.3687.53>:misc:try_with_maybe_ignorant_after:1535]Eating exception from ignorant after-block:
    {error,{badmatch,[{<0.3693.53>,
    badmatch,{error,timeout,
    [{ns_replicas_builder_utils,kill_a_bunch_of_tap_names,3}

    ,
    

    {misc,try_with_maybe_ignorant_after,2},
    {gen_server,terminate,6},
    {proc_lib,init_p_do_apply,3}]}}]},
    [{misc,sync_shutdown_many_i_am_trapping_exits,1},
    {misc,try_with_maybe_ignorant_after,2}

    ,
    

    {ns_single_vbucket_mover,mover,6}

    ,
    

    {proc_lib,init_p_do_apply,3}]}
    [error_logger:error,2014-03-10T23:49:34.172,ns_1@10.3.3.28:error_logger<0.6.0>:ale_error_logger_handler:log_report:115]
    =========================CRASH REPORT=========================
    crasher:
    initial call: new_ns_replicas_builder:init/1
    pid: <0.3693.53>
    registered_name: []
    exception exit: badmatch,{error,timeout,
    [{ns_replicas_builder_utils,kill_a_bunch_of_tap_names,3},
    {misc,try_with_maybe_ignorant_after,2},
    {gen_server,terminate,6},
    {proc_lib,init_p_do_apply,3}

    ]}
    in function gen_server:terminate/6
    ancestors: [<0.3687.53>,<0.23581.52>,<0.23542.52>,<0.8452.1>,
    mb_master_sup,mb_master,ns_server_sup,ns_server_cluster_sup,
    <0.59.0>]
    messages: [

    {'EXIT',<0.3929.53>,normal}

    ]
    links: <0.3687.53>,#Port<0.3751096>
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 317811
    stack_size: 24
    reductions: 21293
    neighbours:

[ns_server:debug,2014-03-10T23:49:34.185,ns_1@10.3.3.28:<0.23588.52>:ns_pubsub:do_subscribe_link:136]Parent process of subscription

{ns_node_disco_events,<0.23581.52>}

exited with reason {badmatch,
[{<0.3670.53>,
{{badmatch,
{error,
timeout}},
[

{ns_replicas_builder_utils, kill_a_bunch_of_tap_names, 3}

,

{misc, try_with_maybe_ignorant_after, 2}

,

{gen_server, terminate, 6}

,

{proc_lib, init_p_do_apply, 3}

]}}]}
[ns_server:info,2014-03-10T23:49:34.186,ns_1@10.3.3.28:janitor_agent-default<0.19454.50>:janitor_agent:handle_info:750]Undoing temporary vbucket states caused by rebalance
[ns_server:debug,2014-03-10T23:49:34.186,ns_1@10.3.3.28:compaction_daemon<0.508.0>:compaction_daemon:handle_info:456]Looks like vbucket mover inhibiting view compaction for for bucket "default" is dead. Canceling inhibition
[{<0.3670.53>,
badmatch,{error,timeout,
[

{ns_replicas_builder_utils, kill_a_bunch_of_tap_names,3}

,

{misc,try_with_maybe_ignorant_after,2},
{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}}]}

[ns_server:debug,2014-03-10T23:49:34.186,ns_1@10.3.3.28:<0.23558.52>:ns_pubsub:do_subscribe_link:136]Parent process of subscription {master_activity_events,<0.23557.52>} exited with reason {badmatch,
[{<0.3670.53>,
{{badmatch,
{error,
timeout}},
[{ns_replicas_builder_utils, kill_a_bunch_of_tap_names, 3},
{misc, try_with_maybe_ignorant_after, 2},
{gen_server, terminate, 6},
{proc_lib, init_p_do_apply, 3}]}}]}
[error_logger:error,2014-03-10T23:49:34.188,ns_1@10.3.3.28:error_logger<0.6.0>:ale_error_logger_handler:log_report:115]
=========================CRASH REPORT=========================
crasher:
initial call: ns_single_vbucket_mover:mover/6
pid: <0.3687.53>
registered_name: []
exception exit: {unexpected_exit,
{'EXIT',<0.23581.52>,
{badmatch,
[{<0.3670.53>,
badmatch,{error,timeout,
[{ns_replicas_builder_utils, kill_a_bunch_of_tap_names,3},
{misc,try_with_maybe_ignorant_after,2}

,

{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}}]}}}
in function ns_single_vbucket_mover:spawn_and_wait/1
in call from ns_single_vbucket_mover:mover_inner/5
in call from misc:try_with_maybe_ignorant_after/2
in call from ns_single_vbucket_mover:mover/6
ancestors: [<0.23581.52>,<0.23542.52>,<0.8452.1>,mb_master_sup,
mb_master,ns_server_sup,ns_server_cluster_sup,<0.59.0>]
messages: [{'EXIT',<0.23581.52>,
{badmatch,
[{<0.3670.53>,
badmatch,{error,timeout,
[{ns_replicas_builder_utils, kill_a_bunch_of_tap_names,3},
{misc,try_with_maybe_ignorant_after,2},
{gen_server,terminate,6}

,

{proc_lib,init_p_do_apply,3}

]}}]}}]
links: [<0.23581.52>]
dictionary: [

{cleanup_list,[<0.3693.53>,<0.3694.53>]}

]
trap_exit: true
status: running
heap_size: 2584
stack_size: 24
reductions: 6662
neighbours:

Uploading Logs.