Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.5.0
-
Enterprise Edition 6.5.0 build 4558
-
Untriaged
-
Centos 64-bit
-
-
Unknown
-
KV-Engine Mad-Hatter GA
Description
Steps to Reproduce:
- Create a 3-node cluster.(10.112.193.101, 10.112.193.102, 10.112.193.103)
- Create an ephemeral bucket with replicas=1, compression=OFF, eviction policy = NoEviction.(GleamBookUsers)
- Load 300k docs in the bucket with doc_size=1KB, durability=MAJORITY.
- Step 3 completes successfully.
- Graceful Failover a node (10.112.193.103 in this case).
- Rebalance Out the node failed over in Step 5.
Rebalance fails with mover_crashed.
Cluster Nodes after failover: 10.112.193.101, 10.112.193.102
Failed Over Node: 10.112.193.103
Rebalance Out failed in these Builds also: 6.5.0-4380, 6.5.0-4471
Error Messages:
Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.19238.1>, |
{{{child_interrupted,
|
{'EXIT',<23374.3623.0>,socket_closed}}, |
[{dcp_replicator,spawn_and_wait,1, |
[{file,"src/dcp_replicator.erl"}, |
{line,266}]}, |
{dcp_replicator,handle_call,3, |
[{file,"src/dcp_replicator.erl"}, |
{line,121}]}, |
{gen_server,try_handle_call,4, |
[{file,"gen_server.erl"},{line,636}]}, |
{gen_server,handle_msg,6, |
[{file,"gen_server.erl"},{line,665}]}, |
{proc_lib,init_p_do_apply,3, |
[{file,"proc_lib.erl"},{line,247}]}]}, |
{gen_server,call,
|
[{'janitor_agent-GleamBookUsers', |
'ns_1@10.112.193.102'}, |
{if_rebalance,<0.1234.1>, |
{wait_dcp_data_move,
|
['ns_1@10.112.193.101'], |
678}}, |
infinity]}}}}}.
|
Rebalance Operation Id = 3b5f0eef3435aa5be6ebf4557cbb53b1
|
Worker <0.18779.1> (for action {move,{678, |
['ns_1@10.112.193.102',undefined], |
['ns_1@10.112.193.102', |
'ns_1@10.112.193.101'], |
[]}}) exited with reason {unexpected_exit,
|
{'EXIT', |
<0.19238.1>, |
{{{child_interrupted,
|
{'EXIT', |
<23374.3623.0>, |
socket_closed}},
|
[{dcp_replicator,
|
spawn_and_wait,
|
1, |
[{file,
|
"src/dcp_replicator.erl"}, |
{line,
|
266}]}, |
{dcp_replicator,
|
handle_call,
|
3, |
[{file,
|
"src/dcp_replicator.erl"}, |
{line,
|
121}]}, |
{gen_server,
|
try_handle_call,
|
4, |
[{file,
|
"gen_server.erl"}, |
{line,
|
636}]}, |
{gen_server,
|
handle_msg,
|
6, |
[{file,
|
"gen_server.erl"}, |
{line,
|
665}]}, |
{proc_lib,
|
init_p_do_apply,
|
3, |
[{file,
|
"proc_lib.erl"}, |
{line,
|
247}]}]}, |
{gen_server,
|
call,
|
[{'janitor_agent-GleamBookUsers', |
'ns_1@10.112.193.102'}, |
{if_rebalance,
|
<0.1234.1>, |
{wait_dcp_data_move,
|
['ns_1@10.112.193.101'], |
678}}, |
infinity]}}}}
|