Details
-
Bug
-
Resolution: Duplicate
-
Major
-
2.0
-
None
-
Security Level: Public
-
None
Description
Build 1974:
verified bug MB-7263 Service memcached constantly exited on dest master node after certain steps in XDCR + rebalance scenarious: Port server memcached on node 'ns_1@10.3.121.63' exited with status 71. failed to listen on TCP port 11210: Address already in use - was not reproduced
but rebalance failed on both cluster when I add replication
steps: 2 clusters with 5 nodes in each, 10 buckets* 100K
10.3.121.63* and 10.3.121.11* on
1. swap rebalance 1 node on each cluster
2. during rebalance on both clusters create replications
Resetting rebalance status since it's not really running ns_janitor000 ns_1@10.3.121.64 17:02:45 - Wed Dec 5, 2012
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.3.121.64 17:02:44 - Wed Dec 5, 2012
Created remote cluster reference "http://10.3.121.112:8091/" via 10.3.121.112. menelaus_web_remote_clusters000 ns_1@10.3.121.63 17:02:42 - Wed Dec 5, 2012
Server error during processing: ["web request failed",
{what,
{timeout, {gen_server,call, [ns_node_disco,nodes_wanted]}}},
{trace,
[{gen_server,call,2}, {menelaus_web,handle_tasks,2}, {menelaus_web,loop,3}, {mochiweb_http,headers,5}, {proc_lib,init_p_do_apply,3}]}] menelaus_web019 ns_1@10.3.121.63 17:02:42 - Wed Dec 5, 2012
Got exit from node disco events subscription ns_vbucket_mover000 ns_1@10.3.121.63 17:02:35 - Wed Dec 5, 2012
Server error during processing: ["web request failed",{path,"/pools/default/tasks"}
,
{type,exit},{what,
{timeout, {gen_server,call,[ns_doctor,get_nodes]}}},
{trace,
[{gen_server,call,2}, {ns_doctor,build_tasks_list,2}, {menelaus_web,handle_tasks,2}, {menelaus_web,loop,3}, {mochiweb_http,headers,5}, {proc_lib,init_p_do_apply,3}]}] menelaus_web019 ns_1@10.3.121.63 17:02:29 - Wed Dec 5, 2012
Server error during processing: ["web request failed", {path,"/pools/default"},{type,exit}
,
{what,
{timeout,
}},
{trace,
[
,
{menelaus_web,build_nodes_info_fun,3},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info,2},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] menelaus_web019 ns_1@10.3.121.63 17:02:01 - Wed Dec 5, 2012
Bucket "standard_bucket3" rebalance appears to be swap rebalance ns_vbucket_mover000 ns_1@10.3.121.63 16:49:59 - Wed Dec 5, 2012
Bucket "standard_bucket3" loaded on node 'ns_1@10.3.121.69' in 0 seconds. ns_memcached001 ns_1@10.3.121.69 16:49:58 - Wed Dec 5, 2012
Shutting down bucket "standard_bucket4" on 'ns_1@10.3.121.68' for deletion ns_memcached002 ns_1@10.3.121.68 16:49:56 - Wed Dec 5, 2012
Started rebalancing bucket standard_bucket3 ns_rebalancer000 ns_1@10.3.121.63 16:49:56 - Wed Dec 5, 2012
Bucket "standard_bucket4" rebalance appears to be swap rebalance ns_vbucket_mover000 ns_1@10.3.121.63 16:37:59 - Wed Dec 5, 2012
Bucket "standard_bucket4" loaded on node 'ns_1@10.3.121.69' in 0 seconds. ns_memcached001 ns_1@10.3.121.69 16:37:58 - Wed Dec 5, 2012
Started rebalancing bucket standard_bucket4 ns_rebalancer000 ns_1@10.3.121.63 16:37:57 - Wed Dec 5, 2012
Starting rebalance, KeepNodes = ['ns_1@10.3.121.63','ns_1@10.3.121.69',
'ns_1@10.3.121.67','ns_1@10.3.121.66',
'ns_1@10.3.121.64'], EjectNodes = ['ns_1@10.3.121.68']
ns_orchestrator004 ns_1@10.3.121.63 16:37:56 - Wed Dec 5, 2012