Details
-
Bug
-
Resolution: Unresolved
-
Major
-
5.5.0
-
5.5.0-2938
-
Untriaged
-
Centos 64-bit
-
-
No
Description
Script to Repro
./testrunner -i /tmp/win10-bucket-ops.ini -p upgrade_to=5.5.0-2938,initial_version=5.0.1-5003,get-cbcollect-info=False,get-coredumps=False -t 2i.upgrade_2i.UpgradeSecondaryIndex.test_online_upgrade_with_failover,nodes_init=3,services_init=kv:n1ql-index-index,dataset=default,scan_consistency=request_plus,groups=simple,init_nodes=False,gsi_type=forestdb,nodes_out=2,nodes_out_dist=index:2,nodes_in=1,services_in=index:1,before=create_index,disable_plasma_upgrade=True,toggle_disable_upgrade=True
|
The test basically tries to upgrade from 5.0.1 - 5.5.0 with failover/recovery , when we try to do rebalance it fails with
Rebalance exited with reason {service_rebalance_failed,index,
|
{lost_connection,shutdown}}
|
However it should be noted that retry of the failed rebalance succeeded. Logs attached.
Some log snippets from indexer.log
2018-06-28T00:48:15.889-07:00 [Error] DDLServiceMgr: notifyNewTopologyChange(): Failed to initialize metadata provider. Error=DDLServiceMgr: Failed to initialize metadata provider. Unknown host=map[0bb1d9883d77e5cacbcd33009e8a9dac:true].
|
2018-06-28T00:48:15.889-07:00 [Warn] DDLServiceMgr: Failed to clean delete index token upon rebalancing. Skip Cleanup. Internal Error = DDLServiceMgr: Failed to initialize metadata provider. Unknown host=map[0bb1d9883d77e5cacbcd33009e8a9dac:true]
|
Node failover and recovered :- 172.23.106.144
Node where rebalance failure occurred :- 172.23.106.152