Rebalance exited with reason nodedown(crasher: initial call: new_ns_replicas_builder:): vm is alive, but node didn't respond. test startkey_endkey_queries_rebalance_incrementaly

2.0.0-1698-rel
http://qa.hq.northscale.net/job/centos-64-2.0-view-query-extended-tests/51/consoleFull
./testrunner -i resources/jenkins/centos-64-7node-viewquery.ini get-logs=True -t viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_incrementaly,skip_rebalance=true

uptime from node that was 'down'

[root@localhost ~]# uptime
02:46:53 up 18 days, 4:19, 2 users, load average: 1.69, 1.72, 1.51
[root@localhost ~]# ifconfig
eth0 Link encap:Ethernet HWaddr AE:EF:F1:B1:FA:B0
inet addr:10.2.2.108

node was not up, all the tests have fallen.

Unfortunately there is no way to get the logs from the server, because we get it using API http://

{0}

:

{1}

/diag:
grabbing diags from
http://10.2.2.108:8091/diag
unable to obtain diags from http://10.2.2.108:8091/diag

Alex, I do not know how this information will be useful in the event of such an error, but we cannot even say whether there was the erlang core dump

[views:info,2012-09-08T14:58:05.459,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:apply_index_states:467]
couch_set_view:remove_replica_partitions([<<"default">>,
<<"_design/test_view-ca3c82f">>,
"VWXY"]) returned ok in 30ms
[views:info,2012-09-08T14:58:05.460,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:apply_index_states:476]
Calling couch_set_view:mark_partitions_unindexable([<<"default">>,
<<"_design/test_view-ca3c82f">>,
"Z"])
[views:info,2012-09-08T14:58:05.460,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:apply_index_states:477]
couch_set_view:mark_partitions_unindexable([<<"default">>,
<<"_design/test_view-ca3c82f">>,
"Z"]) returned ok in 0ms
[ns_server:warn,2012-09-08T14:58:05.460,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:handle_info:333]Remote server node

{'capi_ddoc_replication_srv-default','ns_1@10.2.2.108'} process down: noconnection
[ns_server:debug,2012-09-08T14:58:05.461,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:handle_info:337]doing replicate_newnodes_docs
[ns_server:debug,2012-09-08T14:58:05.463,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:replicate_change_to_node:545]Sending _design/test_view-49553ca to ns_1@10.2.2.108
[ns_server:debug,2012-09-08T14:58:05.463,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:replicate_change_to_node:545]Sending _design/test_view-c5556ce to ns_1@10.2.2.108
[ns_server:debug,2012-09-08T14:58:05.464,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:replicate_change_to_node:545]Sending _design/test_view-9dc02b5 to ns_1@10.2.2.108
[ns_server:debug,2012-09-08T14:58:05.464,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:replicate_change_to_node:545]Sending _design/test_view-87aa970 to ns_1@10.2.2.108
[ns_server:debug,2012-09-08T14:58:05.464,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:replicate_change_to_node:545]Sending _design/test_view-ae5566b to ns_1@10.2.2.108
[ns_server:debug,2012-09-08T14:58:05.464,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:replicate_change_to_node:545]Sending _design/test_view-ca3c82f to ns_1@10.2.2.108
[ns_server:warn,2012-09-08T14:58:05.465,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:handle_info:333]Remote server node {'capi_ddoc_replication_srv-default','ns_1@10.2.2.108'}

process down: noconnection
[ns_server:info,2012-09-08T14:58:09.605,ns_1@10.2.2.60:<0.5382.0>:ns_orchestrator:handle_info:258]Skipping janitor in state rebalancing: {rebalancing_state,<0.12860.7>,
{dict,3,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[], [],[],[],[]}

,
{{[['ns_1@10.2.2.60'|
0.11111111111111116]],
[],[],
[['ns_1@10.2.2.63'|0.0]],
[],[],[],[],
[['ns_1@10.2.2.108'|
0.11111111111111116]],
[],[],[],[],[],[],[]}}},
['ns_1@10.2.2.63','ns_1@10.2.2.60',
'ns_1@10.2.2.108'],
[],[]}
[ns_server:error,2012-09-08T14:58:09.709,ns_1@10.2.2.60:<0.13647.7>:misc:inner_wait_shutdown:1391]Expected exit signal from <20739.11092.2> but could not get it in 5 seconds. This is a bug, but process we're waiting for is dead (noproc), so trying to ignore...
[ns_server:debug,2012-09-08T14:58:09.709,ns_1@10.2.2.60:<0.13647.7>:misc:inner_wait_shutdown:1392]Here's messages:
{messages,[

{'EXIT',<0.13606.7>,shutdown}]}
[ns_server:error,2012-09-08T14:58:09.710,ns_1@10.2.2.60:<0.13647.7>:misc:sync_shutdown_many_i_am_trapping_exits:1373]Shutdown of the following failed: [{<20739.11092.2>,noproc}]
[ns_server:error,2012-09-08T14:58:09.712,ns_1@10.2.2.60:<0.13647.7>:misc:try_with_maybe_ignorant_after:1409]Eating exception from ignorant after-block:
{error,{badmatch,{error,econnrefused}},
[{ns_replicas_builder_utils,kill_a_bunch_of_tap_names,3},
{misc,try_with_maybe_ignorant_after,2},
{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}
[ns_server:error,2012-09-08T14:58:09.713,ns_1@10.2.2.60:<0.13606.7>:misc:sync_shutdown_many_i_am_trapping_exits:1373]Shutdown of the following failed: [{<0.13647.7>,
{{badmatch,[{<20739.11092.2>,noproc}]},
[{misc, sync_shutdown_many_i_am_trapping_exits, 1},
{misc,try_with_maybe_ignorant_after,2},
{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}}]
[error_logger:error,2012-09-08T14:58:09.713,ns_1@10.2.2.60:error_logger:ale_error_logger_handler:log_msg:76]** Generic server <0.13647.7> terminating
** Last message in was {'EXIT',<20739.11092.2>,normal}
** When Server state == {state,"default",24,'ns_1@10.2.2.108',
[{'ns_1@10.2.2.63',<20739.11092.2>}]}
** Reason for termination ==
** {{badmatch,[{<20739.11092.2>,noproc}]},
[{misc,sync_shutdown_many_i_am_trapping_exits,1},
{misc,try_with_maybe_ignorant_after,2},
{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}

[ns_server:info,2012-09-08T14:58:09.713,ns_1@10.2.2.60:'janitor_agent-default':janitor_agent:handle_info:646]Undoing temporary vbucket states caused by rebalance
[ns_server:debug,2012-09-08T14:58:09.713,ns_1@10.2.2.60:<0.12919.7>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_node_disco_events,<0.12915.7>} exited with reason {{nodedown,
'ns_1@10.2.2.108'},
{gen_server,
call,
[{'janitor_agent-default', 'ns_1@10.2.2.108'},
{if_rebalance,
<0.12915.7>,
{update_vbucket_state,
90,
active,
undefined,
undefined}},
infinity]}}
[user:info,2012-09-08T14:58:09.713,ns_1@10.2.2.60:<0.5382.0>:ns_orchestrator:handle_info:295]Rebalance exited with reason {{nodedown,'ns_1@10.2.2.108'},
{gen_server,call,
[{'janitor_agent-default','ns_1@10.2.2.108'},
{if_rebalance,<0.12915.7>,
{update_vbucket_state,90,active,undefined,
undefined}},
infinity]}}

[ns_server:debug,2012-09-08T14:58:09.714,ns_1@10.2.2.60:<0.12919.7>:ns_pubsub:do_subscribe_link:149]Deleting {ns_node_disco_events,<0.12915.7>} event handler: ok
[error_logger:error,2012-09-08T14:58:09.714,ns_1@10.2.2.60:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: new_ns_replicas_builder:init/1
pid: <0.13647.7>
registered_name: []
exception exit: {{badmatch,[{<20739.11092.2>,noproc}]},
[{misc,sync_shutdown_many_i_am_trapping_exits,1},
{misc,try_with_maybe_ignorant_after,2},
{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}
in function gen_server:terminate/6
ancestors: [<0.13606.7>,<0.12915.7>,<0.12860.7>]
messages: [{'EXIT',<0.13606.7>,shutdown}

]
links: [<0.13606.7>]
dictionary: []
trap_exit: true
status: running
heap_size: 121393
stack_size: 24
reductions: 19436
neighbours:

[views:info,2012-09-08T14:58:09.714,ns_1@10.2.2.60:'capi_set_view_manager-default':capi_set_view_manager:apply_index_states:453]
Calling couch_set_view:mark_partitions_indexable([<<"default">>,
<<"_design/test_view-49553ca">>,
[0,1,2,3,4,5,6,7,8,9,10,11,
12,13,14,15,16,17,18,19,20,
21,22,23,24,25,26,27,28,29,
30,31,32,33,34,35,36,37,38,
39,40,41,42,43,44,45,46,47,
48,49,50,51,52,53,54,55,56,
57,58,59,60,61,62,63,64,65,
66,67,68,69,70,71,72,73,74,
75,76,77,78,79,80,81,82,83,
84,85,86,87,88,89,90,91,92,
93,94,95,96,97,98,99,100,
101,102,103,104,105,106,
107,108,109,110,111,112,
113,114,115,116,117,118,
119,120,121,122,123,124,
125,126,127]])
[ns_server:info,2012-09-08T14:58:09.716,ns_1@10.2.2.60:<0.14222.7>:diag_handler:log_all_tap_and_checkpoint_stats:126]logging tap & checkpoint stats
[error_logger:error,2012-09-08T14:58:09.717,ns_1@10.2.2.60:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: ns_single_vbucket_mover:mover/6
pid: <0.13606.7>
registered_name: []
exception error: no match of right hand side value
[{<0.13647.7>,
{{badmatch,[

{<20739.11092.2>,noproc}]},
[{misc,sync_shutdown_many_i_am_trapping_exits,1},
{misc,try_with_maybe_ignorant_after,2},
{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}}]
in function misc:sync_shutdown_many_i_am_trapping_exits/1
in call from misc:try_with_maybe_ignorant_after/2
in call from ns_single_vbucket_mover:mover/6
ancestors: [<0.12915.7>,<0.12860.7>]
messages: [{'EXIT',<0.12915.7>,
{{nodedown,'ns_1@10.2.2.108'},
{gen_server,call,
[{'janitor_agent-default','ns_1@10.2.2.108'},
{if_rebalance,<0.12915.7>,
{update_vbucket_state,90,active,undefined,
undefined}},
infinity]}}}]
links: [<0.12915.7>,<0.14164.7>]
dictionary: [{cleanup_list,[<0.13647.7>]}]
trap_exit: true
status: running
heap_size: 75025
stack_size: 24
reductions: 15055
neighbours:

[ns_server:debug,2012-09-08T14:58:09.717,ns_1@10.2.2.60:ns_config_rep:ns_config_rep:do_push_keys:317]Replicating some config keys ([counters,rebalance_status,rebalancer_pid]..)
[error_logger:error,2012-09-08T14:58:09.717,ns_1@10.2.2.60:error_logger:ale_error_logger_handler:log_msg:76]** Generic server <0.12915.7> terminating
** Last message in was {'EXIT',<0.13849.7>,
{{nodedown,'ns_1@10.2.2.108'},
{gen_server,call,
[{'janitor_agent-default','ns_1@10.2.2.108'},
{if_rebalance,<0.12915.7>,
{update_vbucket_state,90,active,undefined,
undefined}},
infinity]}}}
** When Server state == {state,"default",<0.12919.7>,
{dict,3,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[], [],[],[]},
{{[['ns_1@10.2.2.60'|63]],
[],[],
[['ns_1@10.2.2.63'|42]],
[],[],[],[],
[['ns_1@10.2.2.108'|63]],
[],[],[],[],[],[],[]}}},
1,
{array,128,0,undefined,
{{{['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60']},
{['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60']},
{['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63'], ['ns_1@10.2.2.60','ns_1@10.2.2.63'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60']},
{['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60']},
{['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60']},
{['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60']},
{['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108']},
{['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108']},
{['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108','ns_1@10.2.2.63'], ['ns_1@10.2.2.108','ns_1@10.2.2.63'], ['ns_1@10.2.2.108','ns_1@10.2.2.63'], ['ns_1@10.2.2.108','ns_1@10.2.2.63']},
{['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108']},
10},
{{['ns_1@10.2.2.60','ns_1@10.2.2.108'],
['ns_1@10.2.2.60','ns_1@10.2.2.108'],
['ns_1@10.2.2.60','ns_1@10.2.2.108'],
['ns_1@10.2.2.60','ns_1@10.2.2.108'],
['ns_1@10.2.2.60','ns_1@10.2.2.108'],
['ns_1@10.2.2.60','ns_1@10.2.2.108'],
['ns_1@10.2.2.60','ns_1@10.2.2.108'],
['ns_1@10.2.2.60','ns_1@10.2.2.108'],
['ns_1@10.2.2.60','ns_1@10.2.2.108'],
['ns_1@10.2.2.60','ns_1@10.2.2.108']},
{['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108']},
{['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.60','ns_1@10.2.2.108'], undefined,undefined},
10,10,10,10,10,10,10,10},
100,100,100,100,100,100,100,100,100}},
{dict,2,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[], [],[],[]},
{{[['ns_1@10.2.2.60',
{91, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{92, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{93, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{94, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{95, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{96, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{97, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{98, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{99, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{100, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{101, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{102, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{103, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{104, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{105, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{106, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.108', 'ns_1@10.2.2.63']},
{107, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{108, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{109, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{110, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{111, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{112, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{113, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{114, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{115, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{116, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{117, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{118, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{119, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{120, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{121, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{122, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{123, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{124, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{125, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{126, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']},
{127, ['ns_1@10.2.2.60','ns_1@10.2.2.108'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.108']}]],
[],[],[],[],[],[],[],
[['ns_1@10.2.2.108',
{25, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{26, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{27, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{28, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{29, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{30, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{31, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{32, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{33, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{34, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{35, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{36, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{37, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{38, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{39, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{40, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{41, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{42, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
{43, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{44, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{45, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{46, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{47, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{48, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{49, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{50, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{51, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{52, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{53, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{54, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{55, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{56, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{57, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{58, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{59, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{60, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{61, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{62, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63','ns_1@10.2.2.60']},
{63, ['ns_1@10.2.2.108','ns_1@10.2.2.60'], ['ns_1@10.2.2.63', 'ns_1@10.2.2.60']}]],
[],[],[],[],[],[],[]}}},
{dict,2,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[], [],[],[]},
{{[['ns_1@10.2.2.60'|1]],
[],[],[],[],[],[],[],
[['ns_1@10.2.2.108'|1]],
[],[],[],[],[],[],[]}}},
#Fun<ns_rebalancer.3.93630518>}
** Reason for termination ==
** {{nodedown,'ns_1@10.2.2.108'},
{gen_server,call,
[{'janitor_agent-default','ns_1@10.2.2.108'},
{if_rebalance,<0.12915.7>,
{update_vbucket_state,90,active,undefined,undefined}},
infinity]}}

[ns_server:debug,2012-09-08T14:58:09.732,ns_1@10.2.2.60:ns_config_log:ns_config_log:log_common:111]config change:
counters ->
[{rebalance_fail,1},
{rebalance_start,31},
{rebalance_success,30},
{failover_node,2}]
[ns_server:debug,2012-09-08T14:58:09.732,ns_1@10.2.2.60:ns_config_log:ns_config_log:log_common:111]config change:
rebalancer_pid ->
undefined
[ns_server:debug,2012-09-08T14:58:09.732,ns_1@10.2.2.60:ns_config_log:ns_config_log:log_common:111]config change:
rebalance_status -> {none,<<"Rebalance failed. See logs for detailed reason. You can try rebalance again.">>}
[error_logger:error,2012-09-08T14:58:09.736,ns_1@10.2.2.60:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: ns_vbucket_mover:init/1
pid: <0.12915.7>
registered_name: []
exception exit: {{nodedown,'ns_1@10.2.2.108'},
{gen_server,call,
[{'janitor_agent-default','ns_1@10.2.2.108'},
{if_rebalance,<0.12915.7>,
{update_vbucket_state,90,active,undefined,
undefined}},
infinity]}}
in function gen_server:terminate/6
ancestors: [<0.12860.7>]
messages: [{ns_node_disco_events, ['ns_1@10.2.2.108','ns_1@10.2.2.60','ns_1@10.2.2.63'], ['ns_1@10.2.2.60','ns_1@10.2.2.63']},
log_tap_stats,
{'EXIT',<0.13606.7>,
{badmatch,
[{<0.13647.7>,
{{badmatch,[{<20739.11092.2>,noproc}

]},
[

{misc,sync_shutdown_many_i_am_trapping_exits,1}

,

{misc,try_with_maybe_ignorant_after,2}

,

{gen_server,terminate,6}

,

{proc_lib,init_p_do_apply,3}

]}}]}}]
links: [<0.12860.7>,<0.12919.7>,<0.62.0>]
dictionary: [

{bucket_name,"default"}

,

{i_am_master_mover,true}

,

{child_processes,[<0.13849.7>,<0.13648.7>,<0.13606.7>, <0.13354.7>,<0.13110.7>,<0.13088.7>, <0.12922.7>,<0.12921.7>]}

]
trap_exit: true
status: running
heap_size: 2584
stack_size: 24
reductions: 108891
neighbours:
neighbour: [

{pid,<0.12860.7>}

,

{registered_name,[]}

,
{initial_call,{erlang,apply,2}},
{current_function,{ns_rebalancer,wait_for_mover_tail,2}},

{ancestors,[]}

,

{messages,[]}

,

{links,[<0.5382.0>,<0.12915.7>]}

,
{dictionary,[{random_seed,{3688,3451,10969}}]},

{trap_exit,false}

,

{status,waiting}

,

{heap_size,46368}

,

{stack_size,28}

,

{reductions,188106}

]