Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
2.0
-
Security Level: Public
-
build 10.3.3.59
Description
1 node went down while loading data on 22 node cluster. (possibly related to xen-hypervisor as it could not ping gateway and network interface needed to be restarted)
While node was down I tried to fail it over and rebalance.
However, rebalance never completes and looks like there is no rebalance activity occuring on tap.
Some activity seen in logs at time of node down:
10.3.3.59 sees .60 nodedown :
[user:warn,2012-10-22T11:06:38.896,ns_1@10.3.3.59:ns_node_disco:ns_node_disco:handle_info:168]Node 'ns_1@10.3.3.59' saw that node 'ns_1@10.3.3.60' went down.
at the same time stamp node .60 shows:
[ns_server:error,2012-10-22T11:06:00.350,ns_1@10.3.3.60:<0.12281.36>:ns_janitor:cleanup_with_states:84]Bucket "default" not yet ready on ['ns_1@10.3.2.84','ns_1@10.3.2.
85',
'ns_1@10.3.2.110','ns_1@10.3.2.111',
'ns_1@10.3.2.112','ns_1@10.3.2.113',
'ns_1@10.3.2.114','ns_1@10.3.2.115',
'ns_1@10.3.3.59','ns_1@10.3.3.62',
'ns_1@10.3.3.65','ns_1@10.3.3.66',
'ns_1@10.3.3.69','ns_1@10.3.3.70',
'ns_1@10.3.121.90','ns_1@10.3.121.91',
'ns_1@10.3.2.107','ns_1@10.3.2.108',
'ns_1@10.3.2.109']
[ns_server:debug,2012-10-22T11:06:07.388,ns_1@10.3.3.60:<0.12508.36>:janitor_agent:new_style_query_vbucket_states_loop:116]Exception from query_vbucket_states of "defau
lt":'ns_1@10.3.2.85'
{'EXIT',{{nodedown,'ns_1@10.3.2.85'},
{gen_server,call,
[
,
query_vbucket_states,infinity]}}}