Details
-
Bug
-
Resolution: Not a Bug
-
Critical
-
None
-
6.5.0
-
Enterprise Edition 6.5.0 build 4558
-
Triaged
-
Centos 64-bit
-
-
Unknown
Description
Steps to Reproduce:
- Create a 9 node cluster.
------------------------------------
| Nodes | Services | Status |
------------------------------------
| 172.23.106.22 | [u'kv'] | Cluster node |
| 172.23.106.23 | None | <--- IN — |
| 172.23.106.28 | None | <--- IN — |
| 172.23.105.229 | None | <--- IN — |
| 172.23.105.230 | None | <--- IN — |
| 172.23.106.21 | None | <--- IN — |
| 172.23.105.223 | None | <--- IN — |
| 172.23.105.225 | None | <--- IN — |
| 172.23.105.226 | None | <--- IN — |
------------------------------------
2. Create an ephemeral bucket with compression = Off, eviction policy=NoEviction, replicas = 1
3. Load 8M docs in the bucket with doc size = 1.06KB and durability=MAJORITY. This step was successful.
4. Rebalance In 1 node with 4M upserts, 1.6M creates, 1.6M deletes in parallel.
Rebalance fails with mover_crashed.
Node Rebalanced In : 172.23.105.227
Error Messages:
Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.23304.2>, |
{{{{{badmatch,
|
[{<25044.392.1>, |
{done,exit,
|
{socket_closed,
|
{gen_server,call,
|
[<25044.25340.0>, |
{setup_streams,
|
[68,69,70,183,184,185,297,298,299, |
412,413,653,654]}, |
infinity]}},
|
[{gen_server,call,3, |
[{file,"gen_server.erl"}, |
{line,214}]}, |
{dcp_replicator,
|
'-spawn_and_wait/1-fun-0-',1, |
[{file,"src/dcp_replicator.erl"}, |
{line,243}]}]}}]}, |
[{misc,
|
sync_shutdown_many_i_am_trapping_exits,
|
1, |
[{file,"src/misc.erl"},{line,1368}]}, |
{dcp_replicator,spawn_and_wait,1, |
[{file,"src/dcp_replicator.erl"}, |
{line,265}]}, |
{dcp_replicator,handle_call,3, |
[{file,"src/dcp_replicator.erl"}, |
{line,121}]}, |
{gen_server,try_handle_call,4, |
[{file,"gen_server.erl"},{line,636}]}, |
{gen_server,handle_msg,6, |
[{file,"gen_server.erl"},{line,665}]}, |
{proc_lib,init_p_do_apply,3, |
[{file,"proc_lib.erl"},{line,247}]}]}, |
{gen_server,call,
|
[<25044.25339.0>, |
{setup_replication,
|
[68,69,70,183,184,185,297,298,299,412, |
413,653,654]}, |
infinity]}},
|
{gen_server,call,
|
['replication_manager-GleamBookUsers', |
{change_vbucket_replication,653, |
'ns_1@172.23.106.21'}, |
infinity]}},
|
{gen_server,call,
|
[{'janitor_agent-GleamBookUsers', |
'ns_1@172.23.105.227'}, |
{if_rebalance,<0.3491.2>, |
{update_vbucket_state,752,active, |
undefined,undefined,undefined}},
|
infinity]}}}}}.
|
Rebalance Operation Id = df7ee7d4bb586666adc3c18feb62339d
|
Worker <0.21979.2> (for action {move,{752, |
['ns_1@172.23.106.22', |
'ns_1@172.23.105.230'], |
['ns_1@172.23.105.227', |
'ns_1@172.23.105.230'], |
[]}}) exited with reason {unexpected_exit,
|
{'EXIT', |
<0.23304.2>, |
{{{{{badmatch,
|
[{<25044.392.1>, |
{done,
|
exit,
|
{socket_closed,
|
{gen_server,
|
call,
|
[<25044.25340.0>, |
{setup_streams,
|
[68, |
69, |
70, |
183, |
184, |
185, |
297, |
298, |
299, |
412, |
413, |
653, |
654]}, |
infinity]}},
|
[{gen_server,
|
call,
|
3, |
[{file,
|
"gen_server.erl"}, |
{line,
|
214}]}, |
{dcp_replicator,
|
'-spawn_and_wait/1-fun-0-', |
1, |
[{file,
|
"src/dcp_replicator.erl"}, |
{line,
|
243}]}]}}]}, |
[{misc,
|
sync_shutdown_many_i_am_trapping_exits,
|
1, |
[{file,
|
"src/misc.erl"}, |
{line,
|
1368}]}, |
{dcp_replicator,
|
spawn_and_wait,
|
1, |
[{file,
|
"src/dcp_replicator.erl"}, |
{line,
|
265}]}, |
{dcp_replicator,
|
handle_call,
|
3, |
[{file,
|
"src/dcp_replicator.erl"}, |
{line,
|
121}]}, |
{gen_server,
|
try_handle_call,
|
4, |
[{file,
|
"gen_server.erl"}, |
{line,
|
636}]}, |
{gen_server,
|
handle_msg,
|
6, |
[{file,
|
"gen_server.erl"}, |
{line,
|
665}]}, |
{proc_lib,
|
init_p_do_apply,
|
3, |
[{file,
|
"proc_lib.erl"}, |
{line,
|
247}]}]}, |
{gen_server,
|
call,
|
[<25044.25339.0>, |
{setup_replication,
|
[68,69, |
70, |
183, |
184, |
185, |
297, |
298, |
299, |
412, |
413, |
653, |
654]}, |
infinity]}},
|
{gen_server,
|
call,
|
['replication_manager-GleamBookUsers', |
{change_vbucket_replication,
|
653, |
'ns_1@172.23.106.21'}, |
infinity]}},
|
{gen_server,
|
call,
|
[{'janitor_agent-GleamBookUsers', |
'ns_1@172.23.105.227'}, |
{if_rebalance,
|
<0.3491.2>, |
{update_vbucket_state,
|
752, |
active,
|
undefined,
|
undefined,
|
undefined}},
|
infinity]}}}}
|
