Details
-
Bug
-
Resolution: Won't Fix
-
Critical
-
None
-
6.6.2
-
Untriaged
-
-
1
-
Yes
Description
Build : 7.0.0-5219 (RC2)
Test : -test tests/2i/cheshirecat/test_idx_clusterops_cheshire_cat_moi.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat_moi.yml (MOI storage)
Scale : 2
Iteration : 7th
Day : 3rd
In the 7th iteration, there were 7 rebalance operations that failed in quick succession due to the error - "indexer rebalance failure - move index in progress". This indicates there is a test issue as the test should have ensured all indexes are ready before initiating a rebalance.
However, the 3 subsequent rebalance operations failed due to the following error :
Protocol Conflict Error: Existing Rebalance Token Found
[ns_server:error,2021-05-24T06:50:03.547-07:00,ns_1@172.23.97.215:service_rebalancer-index<0.10474.1668>:service_rebalancer:run_rebalance_worker:119]Worker terminated abnormally: {'EXIT',<0.4906.1668>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Protocol Conflict Error: Existing Rebalance Token Found">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,164}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,234}]}]}}
|
[user:error,2021-05-24T06:50:03.550-07:00,ns_1@172.23.97.215:<0.8853.0>:ns_orchestrator:log_rebalance_completion:1416]Rebalance exited with reason {service_rebalance_failed,index,
|
{worker_died,
|
{'EXIT',<0.4906.1668>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Protocol Conflict Error: Existing Rebalance Token Found">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,164}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,234}]}]}}}}.
|
Rebalance Operation Id = a03708bdf8f80b0f3b2edaff38e32c0d
|
[ns_server:error,2021-05-24T07:17:18.697-07:00,ns_1@172.23.97.215:service_rebalancer-index<0.16251.1685>:service_rebalancer:run_rebalance_worker:119]Worker terminated abnormally: {'EXIT',<0.17913.1685>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Protocol Conflict Error: Existing Rebalance Token Found">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,164}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,234}]}]}}
|
[user:error,2021-05-24T07:17:18.699-07:00,ns_1@172.23.97.215:<0.8853.0>:ns_orchestrator:log_rebalance_completion:1416]Rebalance exited with reason {service_rebalance_failed,index,
|
{worker_died,
|
{'EXIT',<0.17913.1685>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Protocol Conflict Error: Existing Rebalance Token Found">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,164}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,234}]}]}}}}.
|
Rebalance Operation Id = 49d2f673493da1c05f08fd28c66aa0a2
|
[ns_server:error,2021-05-24T07:29:14.067-07:00,ns_1@172.23.97.215:service_rebalancer-index<0.5268.1693>:service_rebalancer:run_rebalance_worker:119]Worker terminated abnormally: {'EXIT',<0.11182.1693>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Protocol Conflict Error: Existing Rebalance Token Found">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,164}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,234}]}]}}
|
[user:error,2021-05-24T07:29:14.070-07:00,ns_1@172.23.97.215:<0.8853.0>:ns_orchestrator:log_rebalance_completion:1416]Rebalance exited with reason {service_rebalance_failed,index,
|
{worker_died,
|
{'EXIT',<0.11182.1693>,
|
{{badmatch,
|
{error,
|
{unknown_error,
|
<<"Protocol Conflict Error: Existing Rebalance Token Found">>}}},
|
[{service_rebalancer,rebalance_worker,1,
|
[{file,"src/service_rebalancer.erl"},
|
{line,164}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,234}]}]}}}}.
|
Rebalance Operation Id = 6186bf3cffb717ecc670e07a6f296662
|
On 172.23.107.5, the following can be seen in the indexer logs :
2021-05-24T06:50:03.515-07:00 [Info] ServiceMgr::StartTopologyChange {efd412ababc18a786301ab9605e66c95 [] topology-change-rebalance [{{5cd57e7e3bd134045a64d201e597eeb3 5 <nil>} recovery-full} {{c8f8e9eb7afb7781acb6269f4910760c 5 <nil>} recovery-full} {{bfefce3231a5c2f3f4e1bc0541532c00 5 <nil>} recovery-full} {{ddcd87eb9e0baf13d2277c336d4a1f58 5 <nil>} recovery-full} {{72e8ea877d50a980d1560bc0f59ba076 5 <nil>} recovery-full} {{d1bd466537b13847cb9434734566ae3f 5 <nil>} recovery-full}] []}
|
2021-05-24T06:50:03.524-07:00 [Info] ServiceMgr::cleanupOrphanTokens Found Rebalance Token &{bfefce3231a5c2f3f4e1bc0541532c00 fa:0:79:61:c7:49:16:3e MoveIndex Cannot Process Move Index - Rebalance In Progress }
|
2021-05-24T06:50:03.540-07:00 [Error] ServiceMgr::startRebalance Found Existing Global RToken &{bfefce3231a5c2f3f4e1bc0541532c00 fa:0:79:61:c7:49:16:3e MoveIndex Cannot Process Move Index - Rebalance In Progress }
|
2021-05-24T06:50:03.540-07:00 [Info] ServiceMgr::runCleanupPhase path /indexing/rebalance/RebalanceToken isMaster true
|
2021-05-24T06:50:03.543-07:00 [Info] ServiceMgr::cleanupLocalRToken Cleanup
|
2021-05-24T06:50:03.544-07:00 [Info] ClustMgr:handleDelLocalValue Key RebalanceToken
|
2021-05-24T06:50:03.544-07:00 [Info] ServiceMgr::cleanupRebalanceRunning Cleanup
|
2021-05-24T06:50:03.544-07:00 [Info] ClustMgr:handleDelLocalValue Key RebalanceRunning
|
2021-05-24T06:50:03.545-07:00 [Info] ServiceMgr::StartTopologyChange returns Error Protocol Conflict Error: Existing Rebalance Token Found. isBalanced false.
|
172.23.107.5 was hard failed over earlier. This rebalance operation is to add it back to the cluster after full recovery.
Index nodes : 172.23.107.2, 172.23.107.3, 172.23.107.4, 172.23.107.5, 172.23.97.216, 172.23.97.217