Details
-
Bug
-
Resolution: Duplicate
-
Test Blocker
-
None
-
6.5.0
-
component cluster
Description
Build: 6.5.0-1441
Test : -test tests/eventing/test_eventing_rebalance_timers.yml -scope tests/eventing/scope_eventing_rebalance_timers.yml
Rebalance failed with mover_crashed when we rebalance doing failover -> delta recovery of KV node
[2018-10-17T05:22:00-07:00, sequoiatools/couchbase-cli:db4b1d] failover -c 172.23.104.16:8091 --server-failover 172.23.104.18:8091 -u Administrator -p password --force
|
[2018-10-17T05:22:13-07:00, sequoiatools/couchbase-cli:3fe451] recovery -c 172.23.104.16:8091 --server-recovery 172.23.104.18:8091 --recovery-type delta -u Administrator -p password
|
[2018-10-17T05:22:22-07:00, sequoiatools/couchbase-cli:6507a8] rebalance -c 172.23.104.16:8091 -u Administrator -p password
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:[rebalance -c 172.23.104.16:8091 -u Administrator -p password]
|
|
|
docker logs 6507a8
|
docker start 6507a8
|
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[user:error,2018-10-17T00:48:49.327-07:00,ns_1@172.23.104.16:<0.6135.0>:ns_orchestrator:do_log_rebalance_completion:1175]Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.23473.94>,
|
{{wait_seqno_persisted_failed,"metadata",730,
|
100956,
|
[{'ns_1@172.23.104.18',
|
{'EXIT',
|
{{{badmatch,{error,closed}},
|
[{mc_client_binary,cmd_vocal_recv,5,
|
[{file,"src/mc_client_binary.erl"},
|
{line,150}]},
|
{mc_client_binary,
|
wait_for_seqno_persistence,3,
|
[{file,"src/mc_client_binary.erl"},
|
{line,853}]},
|
{ns_memcached,
|
'-wait_for_seqno_persistence/3-fun-0-',
|
3,
|
[{file,"src/ns_memcached.erl"},
|
{line,1362}]},
|
{ns_memcached,
|
'-perform_very_long_call/3-fun-0-',2,
|
[{file,"src/ns_memcached.erl"},
|
{line,330}]},
|
{ns_memcached_sockets_pool,
|
'-executing_on_socket/3-fun-0-',3,
|
[{file,
|
"src/ns_memcached_sockets_pool.erl"},
|
{line,74}]},
|
{async,'-async_init/4-fun-2-',3,
|
[{file,"src/async.erl"},
|
{line,208}]}]},
|
{gen_server,call,
|
[{'janitor_agent-metadata',
|
'ns_1@172.23.104.18'},
|
{if_rebalance,<0.23143.94>,
|
{wait_seqno_persisted,730,100956}},
|
infinity]}}}}]},
|
[{ns_single_vbucket_mover,
|
'-wait_seqno_persisted_many/5-fun-2-',5,
|
[{file,"src/ns_single_vbucket_mover.erl"},
|
{line,462}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,232}]}]}}}}
|
[user:error,2018-10-17T05:22:42.291-07:00,ns_1@172.23.104.16:<0.6135.0>:ns_orchestrator:do_log_rebalance_completion:1175]Rebalance exited with reason {{badmatch,
|
{error,
|
{failed_nodes,['ns_1@172.23.104.18']}}},
|
[{ns_janitor,cleanup_apply_config_body,5,
|
[{file,"src/ns_janitor.erl"},{line,197}]},
|
{ns_janitor,'-cleanup_apply_config/5-fun-0-',
|
5,
|
[{file,"src/ns_janitor.erl"},{line,184}]},
|
{async,'-async_init/4-fun-2-',3,
|
[{file,"src/async.erl"},{line,208}]}]}
|
