rebalance failed after crasher: initial call: ebucketmigrator_srv and then crashes one by one : erlang:apply, ns_vbucket_mover, ns_memcached, ns_port_server...

2.0.0-1672-rel
http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions/28/consoleFull
/testrunner -i /tmp/rebalance_regression.ini get-logs=True -t rebalancetests.IncrementalRebalanceInTests.test_load,replica=1,delete-ratio=0.6,expiry-ratio=0.2,keys-count=1000000

rebalance in 1+1 node with ddoc ops

[error_logger:error,2012-09-05T2:41:07.896,ns_1@10.3.121.92:error_logger:ale_error_logger_handler:log_msg:76]** Generic server <19640.7042.32> terminating

• • Last message in was {tcp_closed,#Port<19640.1529268>}
  • When Server state == {state,#Port<19640.1529266>,#Port<19640.1529261>,
    #Port<19640.1529268>,#Port<19640.1529262>,
    <19640.7045.32>,<<>>,<<>>,
    {set,1,16,16,8,80,48,
    {[],[],[],[],[],[],[],[],[],[],[],[],[], [],[],[]}

    ,
    {{[311],
    [],[],[],[],[],[],[],[],[],[],[],[],[],
    [],[]}}},
    1160,false,false,0,
    

    {1346,838066,920600}

    ,
    not_started,undefined,
    <<"replication_building_311_'ns_1@10.3.121.94'">>,
    <19640.7042.32>,
    

    Unknown macro: {had_backfill,true,undefined,[]}

    }

  • Reason for termination ==
  • downstream_closed

[couchdb:info,2012-09-05T2:41:07.899,ns_1@10.3.121.92:<0.19708.264>:couch_log:info:39]Compaction for db "bucket-0/997" completed.
[ns_server:info,2012-09-05T2:41:07.901,ns_1@10.3.121.92:<0.2738.289>:compaction_daemon:spawn_vbucket_compactor:644]Compacting <<"bucket-0/998">>
[couchdb:info,2012-09-05T2:41:07.902,ns_1@10.3.121.92:<0.19701.264>:couch_log:info:39]Starting compaction for db "bucket-0/998"
[error_logger:error,2012-09-05T2:41:07.906,ns_1@10.3.121.92:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: ebucketmigrator_srv:init/1
pid: <19640.7042.32>
registered_name: []
exception exit: downstream_closed
in function gen_server:terminate/6
ancestors: [<0.2407.289>,<0.2404.289>,<0.4477.288>,<0.4434.288>]
messages: []
links: [#Port<19640.1529262>,<0.2407.289>,<19640.7045.32>,
#Port<19640.1529261>]
dictionary: []
trap_exit: true
status: running
heap_size: 987
stack_size: 24
reductions: 132041
neighbours:

[ns_server:error,2012-09-05T2:41:25.182,ns_1@10.3.121.92:<0.2404.289>:misc:sync_shutdown_many_i_am_trapping_exits:1373]Shutdown of the following failed: [{<0.2407.289>,
{{badmatch,[

{<19640.7042.32>,noproc}]},
[{misc, sync_shutdown_many_i_am_trapping_exits, 1},
{misc,try_with_maybe_ignorant_after,2},
{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}}]
[rebalance:error,2012-09-05T2:41:25.184,ns_1@10.3.121.92:<0.4477.288>:ns_vbucket_mover:handle_info:250]<0.2404.289> exited with {badmatch,
[{<0.2407.289>,
{{badmatch,[{<19640.7042.32>,noproc}

]},
[

{misc, sync_shutdown_many_i_am_trapping_exits,1}

,

{misc,try_with_maybe_ignorant_after,2},
{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}}]}
[ns_server:info,2012-09-05T2:41:25.187,ns_1@10.3.121.92:<0.23432.179>:mc_connection:run_loop:190]mccouch connection was normally closed
[ns_server:info,2012-09-05T2:41:25.187,ns_1@10.3.121.92:<0.5870.115>:mc_connection:run_loop:190]mccouch connection was normally closed
[ns_server:info,2012-09-05T2:41:25.191,ns_1@10.3.121.92:'janitor_agent-bucket-0':janitor_agent:handle_info:640]Undoing temporary vbucket states caused by rebalance
[ns_server:debug,2012-09-05T2:41:25.191,ns_1@10.3.121.92:<0.4481.288>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_node_disco_events,<0.4477.288>} exited with reason {badmatch,
[{<0.2407.289>,
{{badmatch,
[{<19640.7042.32>, noproc}]},
[{misc, sync_shutdown_many_i_am_trapping_exits, 1},
{misc, try_with_maybe_ignorant_after, 2},
{gen_server, terminate, 6},
{proc_lib, init_p_do_apply, 3}]}}]}
[ns_server:debug,2012-09-05T2:41:25.192,ns_1@10.3.121.92:<0.4481.288>:ns_pubsub:do_subscribe_link:149]Deleting {ns_node_disco_events,<0.4477.288>} event handler: ok
[user:info,2012-09-05T2:41:25.198,ns_1@10.3.121.92:<0.5344.115>:ns_orchestrator:handle_info:295]Rebalance exited with reason {badmatch,
[{<0.2407.289>,
{{badmatch,[{<19640.7042.32>,noproc}]},
[{misc, sync_shutdown_many_i_am_trapping_exits,1},
{misc,try_with_maybe_ignorant_after,2}

,

{gen_server,terminate,6}

,

{proc_lib,init_p_do_apply,3}

]}}]}