View request may return 500 when node is leaving cluster (was: Rebalance out of two nodes complete and then Http errors retrieved (Uncaught error in HTTP request: {error, {node_vbuckets_dict_failed,not_present}}) )

Ran the test scenario - Rebalance out two nodes from a four node cluster. (2.5.1-1083)

Corresponding to the following errors in the client log -

[84.52 INFO] (SDKD log:137) ESC[39mFINE: Operation returned, but needs to be retried because of: HTTP/1.1 500 Internal Server ErrorESC[0;39m
[84.52 INFO] (SDKD log:137) ESC[39mApr 3, 2014 5:48:59 AM com.couchbase.client.http.HttpResponseCallback retryOperationESC[0;39m

Getting the following errors in the server logs of different nodes in the cluster -

[couchdb:error,2014-04-03T5:48:59.778,ns_1@10.3.4.6:<0.13363.0>:couch_log:error:42]Uncaught error in HTTP request:

{error,function_clause}

Stacktrace: [{base64,b64d_ok,[eq]},
{base64,decode,2},
{menelaus_auth,extract_auth,1},
{menelaus_auth,is_bucket_accessible,3},
{capi_frontend,verify_bucket_auth,2},
{capi_frontend,continue_do_db_req,2},
{request_throttler,do_request,3},
{couch_httpd,handle_request,6}]
[couchdb:info,2014-04-03T5:48:59.778,ns_1@10.3.4.6:<0.13363.0>:couch_log:info:39]10.3.4.8 - - GET /protected/_design/test_design/_view/test_view?limit=0 500
[couchdb:info,2014-04-03T5:48:59.779,ns_1@10.3.4.6:<0.2981.0>:couch_log:info:39]10.3.3.245 - - POST /_view_merge/?limit=0 200
[couchdb:error,2014-04-03T5:48:59.779,ns_1@10.3.4.6:<0.2949.0>:couch_log:error:42]Uncaught error in HTTP request: {error,function_clause}

Stacktrace: [

{base64,b64d_ok,[eq]}

,

{base64,decode,2}

,

{menelaus_auth,extract_auth,1}

,

{menelaus_auth,is_bucket_accessible,3}

,

{capi_frontend,verify_bucket_auth,2}

,

{capi_frontend,continue_do_db_req,2}

,

{request_throttler,do_request,3},
{couch_httpd,handle_request,6}]
[couchdb:info,2014-04-03T5:48:59.780,ns_1@10.3.4.6:<0.2949.0>:couch_log:info:39]10.3.4.8 - - GET /protected/_design/test_design/_view/test_view?limit=0 500



**********************************************************************************


[couchdb:error,2014-04-03T5:48:44.489,ns_1@10.3.4.11:<0.247.0>:couch_log:error:42]Uncaught error in HTTP request: {throw,
{error,
<<"A view spec can not consist of merges exclusively.">>}}

Stacktrace: [{couch_index_merger,get_first_ddoc,3},
{couch_index_merger,query_index,3},
{capi_view,design_doc_view_loop,6},
{request_throttler,do_request,3}

,

{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[couchdb:info,2014-04-03T5:48:44.489,ns_1@10.3.4.11:<0.247.0>:couch_log:info:39]10.3.4.8 - - GET /protected/_design/test_design/_view/test_view?limit=0 500
[couchdb:info,2014-04-03T5:48:46.193,ns_1@10.3.4.11:couch_server<0.212.0>:couch_log:info:39]Deleting database protected/162
[couchdb:info,2014-04-03T5:48:46.196,ns_1@10.3.4.11:couch_server<0.212.0>:couch_log:info:39]Deleting file /opt/couchbase/var/lib/couchbase/data/protected/162.couch.1
[couchdb:info,2014-04-03T5:48:47.301,ns_1@10.3.4.11:couch_server<0.212.0>:couch_log:info:39]Deleting database protected/160
[couchdb:info,2014-04-03T5:48:47.302,ns_1@10.3.4.11:couch_server<0.212.0>:couch_log:info:39]Deleting file /opt/couchbase/var/lib/couchbase/data/protected/160.couch.1
[couchdb:info,2014-04-03T5:48:57.532,ns_1@10.3.4.11:couch_server<0.212.0>:couch_log:info:39]Deleting database _replicator
[couchdb:info,2014-04-03T5:48:57.533,ns_1@10.3.4.11:couch_server<0.212.0>:couch_log:info:39]Deleting file /opt/couchbase/var/lib/couchbase/data/_replicator.couch.1
[couchdb:error,2014-04-03T5:49:27.971,ns_1@10.3.4.11:<0.3598.0>:couch_log:error:42]Uncaught error in HTTP request: {error,
{node_vbuckets_dict_failed,not_present}}

Stacktrace: [{vbucket_map_mirror,node_vbuckets_dict,1},
{capi_view,design_doc_view_loop,6},
{request_throttler,do_request,3},
{couch_httpd,handle_request,6}

,

{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[couchdb:info,2014-04-03T5:49:27.971,ns_1@10.3.4.11:<0.3598.0>:couch_log:info:39]10.3.4.8 - - GET /protected/_design/test_design/_view/test_view?limit=0 500
[couchdb:error,2014-04-03T5:49:27.972,ns_1@10.3.4.11:<0.2938.0>:couch_log:error:42]Uncaught error in HTTP request: {error,
{node_vbuckets_dict_failed,not_present}}

Stacktrace: [{vbucket_map_mirror,node_vbuckets_dict,1},
{capi_view,design_doc_view_loop,6},
{request_throttler,do_request,3},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5}

,

{proc_lib,init_p_do_apply,3}]
[couchdb:info,2014-04-03T5:49:27.972,ns_1@10.3.4.11:<0.2938.0>:couch_log:info:39]10.3.4.8 - - GET /protected/_design/test_design/_view/test_view?limit=0 500
[couchdb:error,2014-04-03T5:49:44.492,ns_1@10.3.4.11:<0.2949.0>:couch_log:error:42]Uncaught error in HTTP request: {error,
{node_vbuckets_dict_failed,not_present}}

Stacktrace: [{vbucket_map_mirror,node_vbuckets_dict,1},
{capi_view,design_doc_view_loop,6},
{request_throttler,do_request,3},
{couch_httpd,handle_request,6},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}

]
[couchdb:info,2014-04-03T5:49:44.492,ns_1@10.3.4.11:<0.2949.0>:couch_log:info:39]10.3.4.8 - - GET /protected/_design/test_design/_view/test_view?limit=0 500

**********************************************************************************

[error_logger:error,2014-04-03T5:47:40.158,ns_1@10.3.3.245:error_logger<0.6.0>:ale_error_logger_handler:log_report:115]
=========================CRASH REPORT=========================
crasher:
initial call: ns_log:start_link_crash_consumer/0-fun-0/0
pid: <0.295.0>
registered_name: []
exception exit: {{nodedown,'babysitter_of_ns_1@127.0.0.1'},
{gen_server,call,
[

{ns_crash_log,'babysitter_of_ns_1@127.0.0.1'}

,
consume,infinity]}}
in function gen_server:call/3
in call from ns_log:crash_consumption_loop/0
in call from misc:delaying_crash/2
ancestors: [ns_server_sup,ns_server_cluster_sup,<0.58.0>]
messages: [

{#Ref<0.0.0.1159>,superseded}

]
links: [<0.289.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 1597
stack_size: 24
reductions: 1221
neighbours:

[error_logger:info,2014-04-03T5:47:40.159,ns_1@10.3.3.245:error_logger<0.6.0>:ale_error_logger_handler:log_report:115]
=========================PROGRESS REPORT=========================
supervisor:

{local,ns_server_sup}

started: [

{pid,<0.958.0>}

,

{name,ns_crash_log_consumer}

,
{mfa,{ns_log,start_link_crash_consumer,[]}},
{restart_type,{permanent,4}},

{shutdown,1000}

,

{child_type,worker}

]

[error_logger:error,2014-04-03T5:47:40.160,ns_1@10.3.3.245:error_logger<0.6.0>:ale_error_logger_handler:log_report:115]
=========================CRASH REPORT=========================
crasher:
initial call: ns_ports_setup:setup_body_tramp/0
pid: <0.820.0>
registered_name: ns_ports_setup
exception error: no match of right hand side value false
in function ns_ports_setup:dynamic_children/0
in call from ns_ports_setup:setup_body/0
in call from misc:delaying_crash/2
ancestors: [ns_server_sup,ns_server_cluster_sup,<0.58.0>]
messages: [check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update,
check_childs_update,check_childs_update,check_childs_update]
links: [<0.289.0>,<0.821.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 10946
stack_size: 24
reductions: 801
neighbours:

[error_logger:info,2014-04-03T5:47:40.160,ns_1@10.3.3.245:error_logger<0.6.0>:ale_error_logger_handler:log_report:115]