Undefined set view error when running views after rebalancing out a node which was restarted

Given:

• A cluster with 3 boxes (2 Windows 2008 R2, 1 Ubuntu 11.04) and 5 data buckets with no replication.

When:

• Ubuntu´s box goes down.
• Restart the couchbase-server service in that box.

Then:

• Can´t execute any view.
• The admin console shows:

Subset of nodes failed with the following error:
[

{ "from": "http://10.230.58.238:8092/_view_merge/", "reason": "Undefined set view `test` for `_design/dev_CategoryTree` design document." }

]

Logs:
[ns_server:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.17585.189>:ns_vbm_sup:spawn_mover:198] Spawned mover "dev" 125 'ns_1@10.230.58.221' -> 'ns_1@10.230.58.238': <0.17586.189>
[ns_server:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.738.0>:ns_port_server:log:161] memcached<0.738.0>: Vbucket <121> is going dead.
memcached<0.738.0>: Vbucket <122> is going dead.
memcached<0.738.0>: Vbucket <123> is going dead.
memcached<0.738.0>: Vbucket <124> is going dead.

[rebalance:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.17586.189>:ebucketmigrator_srv:init:135] CheckpointIdsDict:
{dict,128,26,32,16,130,78,

{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}

,
{{[[0|1],[32|1],[64|1],[96|1]],
[[19|1],[51|1],[83|1],[115|1]],
[[6|1],[38|1],[70|1],[102|1]],
[[25|1],[57|1],[89|1],[121|1]],
[[12|1],[44|1],[76|1],[108|1]],
[[31|1],[63|1],[95|1],[127|1]],
[[18|1],[50|1],[82|1],[114|1]],
[[5|2],[37|1],[69|1],[101|1]],
[[24|1],[56|1],[88|1],[120|1]],
[[11|1],[43|1],[75|1],[107|1]],
[[14|1],[30|1],[46|1],[62|1],[78|1],[94|1],[110|1],[126|1]],
[[1|1],[17|1],[33|1],[49|1],[65|1],[81|1],[97|1],[113|1]],
[[4|1],[20|1],[36|1],[52|1],[68|1],[84|1],[100|1],[116|1]],
[[7|1],[23|1],[39|1],[55|1],[71|1],[87|1],[103|1],[119|1]],
[[10|1],[26|1],[42|1],[58|1],[74|1],[90|1],[106|1],[122|1]],
[[13|1],[29|1],[45|1],[61|1],[77|1],[93|1],[109|1],[125|1]]},

{[[16|1],[48|1],[80|1],[112|1]], [[3|1],[35|1],[67|1],[99|1]], [[22|1],[54|1],[86|1],[118|1]], [[9|1],[41|1],[73|1],[105|1]], [[28|1],[60|1],[92|1],[124|1]], [[15|1],[47|1],[79|1],[111|1]], [[2|1],[34|1],[66|1],[98|1]], [[21|1],[53|1],[85|1],[117|1]], [[8|1],[40|1],[72|1],[104|1]], [[27|1],[59|1],[91|1],[123|1]], [],[],[],[],[],[]}

}}

[rebalance:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.17586.189>:ebucketmigrator_srv:init:166] Starting tap stream:
[

{vbuckets,"}

"},
{checkpoints,[

{125,0}

]},

{name,"rebalance_125"}

,

{takeover,true}

]

[error_logger:error] [2012-03-28 17:11:53] [ns_1@10.230.58.221:error_logger:ale_error_logger_handler:log_msg:76] ** Generic server auto_failover terminating

• • Last message in was tick
  • When Server state == {state,
    
    Unknown macro: {state, [{node_state,'ns_1@10.230.58.221',0,up,false}, {node_state,'ns_1@10.230.58.238',1, nearly_down,false}, {node_state,'ns_1@10.230.58.37',0,up,false}], 0,3}

    ,
    

    {interval,#Ref<0.0.3282.124225>}

    ,
    30,0}

  • Reason for termination ==
  • {{badmatch,rebalancing},
    [ {ns_cluster_membership,failover,1},
    {auto_failover,'-handle_info/2-fun-0-',2},
    {lists,foldl,3},
    {auto_failover,handle_info,2},
    {gen_server,handle_msg,5},
    {proc_lib,init_p_do_apply,3}]}
    
    [ns_server:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.17586.189>:ebucketmigrator_srv:init:175] upstream_sender pid: <0.17589.189>
    [error_logger:error] [2012-03-28 17:11:53] [ns_1@10.230.58.221:error_logger:ale_error_logger_handler:log_report:72]
    =========================CRASH REPORT=========================
    crasher:
    initial call: auto_failover:init/1
    pid: <0.14760.180>
    registered_name: []
    exception exit: {{badmatch,rebalancing},
    [{ns_cluster_membership,failover,1}

    ,
    

    {auto_failover,'-handle_info/2-fun-0-',2},
    {lists,foldl,3},
    {auto_failover,handle_info,2},
    {gen_server,handle_msg,5},
    {proc_lib,init_p_do_apply,3}]}
    in function gen_server:terminate/6
    ancestors: [mb_master_sup,mb_master,ns_server_sup,ns_server_cluster_sup,
    <0.51.0>]
    messages: []
    links: [<0.696.0>,<0.160.0>]
    dictionary: [{random_seed,{6571,6892,26285}}]
    trap_exit: false
    status: running
    heap_size: 75025
    stack_size: 24
    reductions: 4294650
    neighbours:
    
    [error_logger:error] [2012-03-28 17:11:53] [ns_1@10.230.58.221:error_logger:ale_error_logger_handler:log_report:72]
    =========================SUPERVISOR REPORT=========================
    Supervisor: {local,mb_master_sup}
    Context: child_terminated
    Reason: {{badmatch,rebalancing},
    [{ns_cluster_membership,failover,1},
    {auto_failover,'-handle_info/2-fun-0-',2}

    ,
    

    {lists,foldl,3}

    ,
    

    {auto_failover,handle_info,2}

    ,
    

    {gen_server,handle_msg,5}

    ,
    

    {proc_lib,init_p_do_apply,3}

    ]}
    Offender: [

    {pid,<0.14760.180>}

    ,
    

    {name,auto_failover}

    ,
    {mfargs,{auto_failover,start_link,[]}},
    

    {restart_type,permanent}

    ,
    

    {shutdown,10}

    ,
    

    {child_type,worker}

    ]

[rebalance:info] [2012-03-28 17:11:53] [ns_1@10.230.58.221:<0.17586.189>:ebucketmigrator_srv:terminate:202] Skipping close ack for successfull takover