Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
2.0-developer-preview-4
-
Security Level: Public
-
None
-
Windows 2008 R2 64Bit, Ubuntu 11.04.
Description
Given:
- A cluster with 3 boxes (2 Windows 2008 R2, 1 Ubuntu 11.04) and 5 data buckets with no replication.
When:
- Ubuntu´s box goes down.
- Restart the couchbase-server service in that box.
Then:
- Can´t execute any view.
- The admin console shows:
Subset of nodes failed with the following error:
[
]
Logs:
[ns_server:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.17585.189>:ns_vbm_sup:spawn_mover:198] Spawned mover "dev" 125 'ns_1@10.230.58.221' -> 'ns_1@10.230.58.238': <0.17586.189>
[ns_server:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.738.0>:ns_port_server:log:161] memcached<0.738.0>: Vbucket <121> is going dead.
memcached<0.738.0>: Vbucket <122> is going dead.
memcached<0.738.0>: Vbucket <123> is going dead.
memcached<0.738.0>: Vbucket <124> is going dead.
[rebalance:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.17586.189>:ebucketmigrator_srv:init:135] CheckpointIdsDict:
{dict,128,26,32,16,130,78,
,
{{[[0|1],[32|1],[64|1],[96|1]],
[[19|1],[51|1],[83|1],[115|1]],
[[6|1],[38|1],[70|1],[102|1]],
[[25|1],[57|1],[89|1],[121|1]],
[[12|1],[44|1],[76|1],[108|1]],
[[31|1],[63|1],[95|1],[127|1]],
[[18|1],[50|1],[82|1],[114|1]],
[[5|2],[37|1],[69|1],[101|1]],
[[24|1],[56|1],[88|1],[120|1]],
[[11|1],[43|1],[75|1],[107|1]],
[[14|1],[30|1],[46|1],[62|1],[78|1],[94|1],[110|1],[126|1]],
[[1|1],[17|1],[33|1],[49|1],[65|1],[81|1],[97|1],[113|1]],
[[4|1],[20|1],[36|1],[52|1],[68|1],[84|1],[100|1],[116|1]],
[[7|1],[23|1],[39|1],[55|1],[71|1],[87|1],[103|1],[119|1]],
[[10|1],[26|1],[42|1],[58|1],[74|1],[90|1],[106|1],[122|1]],
[[13|1],[29|1],[45|1],[61|1],[77|1],[93|1],[109|1],[125|1]]},
}}
[rebalance:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.17586.189>:ebucketmigrator_srv:init:166] Starting tap stream:
[
"},
{checkpoints,[
]},
,
]
[error_logger:error] [2012-03-28 17:11:53] [ns_1@10.230.58.221:error_logger:ale_error_logger_handler:log_msg:76] ** Generic server auto_failover terminating
-
- Last message in was tick
- When Server state == {state,
Unknown macro: {state, [{node_state,'ns_1@10.230.58.221',0,up,false}, {node_state,'ns_1@10.230.58.238',1, nearly_down,false}, {node_state,'ns_1@10.230.58.37',0,up,false}], 0,3},
{interval,#Ref<0.0.3282.124225>}
,
30,0} - Reason for termination ==
- {{badmatch,rebalancing},
[ {ns_cluster_membership,failover,1},
{auto_failover,'-handle_info/2-fun-0-',2},
{lists,foldl,3},
{auto_failover,handle_info,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
[ns_server:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.17586.189>:ebucketmigrator_srv:init:175] upstream_sender pid: <0.17589.189>
[error_logger:error] [2012-03-28 17:11:53] [ns_1@10.230.58.221:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: auto_failover:init/1
pid: <0.14760.180>
registered_name: []
exception exit: {{badmatch,rebalancing},
[{ns_cluster_membership,failover,1},
{auto_failover,'-handle_info/2-fun-0-',2},
{lists,foldl,3},
{auto_failover,handle_info,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
in function gen_server:terminate/6
ancestors: [mb_master_sup,mb_master,ns_server_sup,ns_server_cluster_sup,
<0.51.0>]
messages: []
links: [<0.696.0>,<0.160.0>]
dictionary: [{random_seed,{6571,6892,26285}}]
trap_exit: false
status: running
heap_size: 75025
stack_size: 24
reductions: 4294650
neighbours:
[error_logger:error] [2012-03-28 17:11:53] [ns_1@10.230.58.221:error_logger:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,mb_master_sup}
Context: child_terminated
Reason: {{badmatch,rebalancing},
[{ns_cluster_membership,failover,1},
{auto_failover,'-handle_info/2-fun-0-',2},
{lists,foldl,3}
,
{auto_failover,handle_info,2}
,
{gen_server,handle_msg,5}
,
{proc_lib,init_p_do_apply,3}
]}
{pid,<0.14760.180>}
Offender: [,
{name,auto_failover}
,
{restart_type,permanent}
{mfargs,{auto_failover,start_link,[]}},
,
{shutdown,10}
,
{child_type,worker}
]
[rebalance:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.17586.189>:ebucketmigrator_srv:terminate:202] Skipping close ack for successfull takover