Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
4.0.0
-
Security Level: Public
-
4.0.0-3402
-
Untriaged
-
Windows 64-bit
-
Unknown
Description
steps:
1. 3 nodes in cluster, 4 buckets. run data loader more then a day
2. setup replication from SRC to cluster DEST for all buckets.
3. rebalance in at SRC cluster
rebalance in 2+2 nodes at DEST cluster
result on dest cluster:
Control connection to memcached on 'ns_1@172.23.105.167' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
cmd_vocal_recv,
5,
[
{mc_client_binary,
select_bucket,
2,
[{file, "src/mc_client_binary.erl"}
,
{line, 351}]},{ns_memcached,
ensure_bucket,
2,
[{file, "src/ns_memcached.erl"}, {line, 1292}]},
{ns_memcached,
handle_info,
2,
[{file, "src/ns_memcached.erl"}, {line, 745}]},
{gen_server,
handle_msg,
5,
[{file, "gen_server.erl"}, {line, 604}]},
{ns_memcached,
init,
1,
[{file, "src/ns_memcached.erl"}, {line, 174}]},
{gen_server,
init_it,
6,
[{file, "gen_server.erl"}, {line, 304}]},
{proc_lib,
init_p_do_apply,
3,
[{file, "proc_lib.erl"}, {line, 239}]}]} (repeated 3 times) ns_memcached000 ns_1@172.23.105.167 07:45:00 - Mon Jul 13, 2015
Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.11817.17>,
{{badmatch,{error,timeout,
[{mc_client_binary,cmd_vocal_recv,5,
[{file,"src/mc_client_binary.erl"}, {line,156}]},
{mc_client_binary,select_bucket,2,
[{file,"src/mc_client_binary.erl"}, {line,351}]},
{ns_memcached,ensure_bucket,2,
[{file,"src/ns_memcached.erl"}, {line,1292}]},
{ns_memcached,handle_info,2,
[{file,"src/ns_memcached.erl"}, {line,745}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,604}]},
{ns_memcached,init,1,
[{file,"src/ns_memcached.erl"}, {line,174}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]},
{gen_server,call,
['ns_memcached-UserInfo', {delete_vbucket,437},
360000]}},
{gen_server,call,
[{'janitor_agent-UserInfo', 'ns_1@172.23.105.167'},
{if_rebalance,<0.30859.16>,
{update_vbucket_state,433,active,paused,
undefined}},
infinity]}}}}
ns_orchestrator002 ns_1@172.23.105.167 07:43:48 - Mon Jul 13, 2015
<0.11616.17> exited with {unexpected_exit,
{'EXIT',<0.11817.17>,
{{badmatch,{error,timeout,
[{mc_client_binary,cmd_vocal_recv,5,
[{file,"src/mc_client_binary.erl"}, {line,156}]},
{mc_client_binary,select_bucket,2,
[{file,"src/mc_client_binary.erl"},{line,351}]},
{ns_memcached,ensure_bucket,2,
[{file,"src/ns_memcached.erl"},{line,1292}]},
{ns_memcached,handle_info,2,
[{file,"src/ns_memcached.erl"},{line,745}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"},{line,604}]},
{ns_memcached,init,1,
[{file,"src/ns_memcached.erl"},{line,174}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]},
{gen_server,call,
['ns_memcached-UserInfo', {delete_vbucket,437},
360000]}},
{gen_server,call,
[{'janitor_agent-UserInfo','ns_1@172.23.105.167'},
{if_rebalance,<0.30859.16>,
{update_vbucket_state,433,active,paused,
undefined}},
infinity]}}}} ns_vbucket_mover000 ns_1@172.23.105.167 07:43:48 - Mon Jul 13, 2015
Control connection to memcached on 'ns_1@172.23.105.167' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
cmd_vocal_recv,
5,
[{file, "src/mc_client_binary.erl"}, {line, 156}]},
{mc_client_binary,
select_bucket,
2,
[{file, "src/mc_client_binary.erl"},{line, 351}
]},
{ns_memcached,
ensure_bucket,
2,
[
{ns_memcached,
handle_info,
2,
[{file, "src/ns_memcached.erl"}
,
{line, 745}]},
{gen_server,
handle_msg,
5,
[
{ns_memcached,
init,
1,
[{file, "src/ns_memcached.erl"}, {line, 174}]},
{gen_server,
init_it,
6,
[{file, "gen_server.erl"}
,
{line, 304}]},
{proc_lib,
init_p_do_apply,
3,
[
,
{line, 239}]}]} ns_memcached000 ns_1@172.23.105.167 07:43:47 - Mon Jul 13, 2015
Bucket "UserInfo" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@172.23.105.167 07:39:43 - Mon Jul 13, 2015
Bucket "UserInfo" loaded on node 'ns_1@172.23.105.169' in 0 seconds. ns_memcached000 ns_1@172.23.105.169 07:39:41 - Mon Jul 13, 2015
Bucket "UserInfo" loaded on node 'ns_1@172.23.105.168' in 0 seconds. ns_memcached000 ns_1@172.23.105.168 07:39:41 - Mon Jul 13, 2015
Started rebalancing bucket UserInfo ns_rebalancer000 ns_1@172.23.105.167 07:39:40 - Mon Jul 13, 2015
Starting rebalance, KeepNodes = ['ns_1@172.23.105.167','ns_1@172.23.105.170',
'ns_1@172.23.105.168','ns_1@172.23.105.169'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
ns_orchestrator004 ns_1@172.23.105.167 07:39:38 - Mon Jul 13, 2015
Node ns_1@172.23.105.169 joined cluster ns_cluster003 ns_1@172.23.105.169 07:36:45 - Mon Jul 13, 2015
Couchbase Server has started on web port 8091 on node 'ns_1@172.23.105.169'. Version: "4.0.0-3402-enterprise". menelaus_sup001 ns_1@172.23.105.169 07:36:45 - Mon Jul 13, 2015
Node 'ns_1@172.23.105.170' saw that node 'ns_1@172.23.105.169' came up. Tags: [] ns_node_disco004 ns_1@172.23.105.170 07:36:37 - Mon Jul 13, 2015
Node 'ns_1@172.23.105.168' saw that node 'ns_1@172.23.105.169' came up. Tags: [] ns_node_disco004 ns_1@172.23.105.168 07:36:37 - Mon Jul 13, 2015
Node 'ns_1@172.23.105.167' saw that node 'ns_1@172.23.105.169' came up. Tags: [] ns_node_disco004 ns_1@172.23.105.167 07:36:36 - Mon Jul 13, 2015
Started node add transaction by adding node 'ns_1@172.23.105.169' to nodes_wanted (group: 0) ns_cluster000 ns_1@172.23.105.167 07:36:34 - Mon Jul 13, 2015
Node ns_1@172.23.105.168 joined cluster ns_cluster003 ns_1@172.23.105.168 07:36:24 - Mon Jul 13, 2015
Couchbase Server has started on web port 8091 on node 'ns_1@172.23.105.168'. Version: "4.0.0-3402-enterprise". menelaus_sup001 ns_1@172.23.105.168 07:36:23 - Mon Jul 13, 2015
Node 'ns_1@172.23.105.170' saw that node 'ns_1@172.23.105.168' came up. Tags: [] ns_node_disco004 ns_1@172.23.105.170 07:36:14 - Mon Jul 13, 2015
Node 'ns_1@172.23.105.167' saw that node 'ns_1@172.23.105.168' came up. Tags: [] ns_node_disco004 ns_1@172.23.105.167 07:36:14 - Mon Jul 13, 2015
Started node add transaction by adding node 'ns_1@172.23.105.168' to nodes_wanted (group: 0) ns_cluster000 ns_1@172.23.105.167 07:36:10 - Mon Jul 13, 2015
Reset auto-failover count auto_failover000 ns_1@172.23.105.167 07:23:00 - Mon Jul 13, 2015
Rebalance completed successfully.
ns_orchestrator001 ns_1@172.23.105.167 07:22:59 - Mon Jul 13, 2015