[FTS] Rebalance fails during indexing with bad_nodes,fts,get_node_infos error

Index building has begun...

2017-06-21 11:31:40 | INFO | MainProcess | test_thread | [moving_topology_fts.rebalance_in_during_index_building] Index count for default_index_1: 5331

2017-06-21 11:31:40 | INFO | MainProcess | test_thread | [fts_base.async_rebalance_in] Starting rebalance-in nodes:[ip:172.23.105.54 port:8091 ssh_username:root] at C1 cluster 172.23.105.96

2017-06-21 11:31:40 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.54:8091 to cluster

2017-06-21 11:31:40 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.54:8091 to this cluster @172.23.105.96:8091

2017-06-21 11:31:46 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.54,ns_1@172.23.105.96,ns_1@172.23.105.190'}

2017-06-21 11:31:46 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started

2017-06-21 11:31:46 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 %

2017-06-21 11:31:46 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0

2017-06-21 11:31:56 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 10.50 %

2017-06-21 11:31:56 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 10.5034722222

2017-06-21 11:32:07 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 22.94 %

2017-06-21 11:32:07 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 22.9383680556

2017-06-21 11:32:17 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 36.85 %

2017-06-21 11:32:17 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 36.8489583333

2017-06-21 11:32:27 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 %

2017-06-21 11:32:27 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 50.0

2017-06-21 11:32:37 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 %

2017-06-21 11:32:37 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 50.0

2017-06-21 11:32:47 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed

2017-06-21 11:32:47 | INFO | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.105.96:

2017-06-21 11:32:47 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.96', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,fts,\n                              {badmatch,\n                               {error,\n                                {bad_nodes,fts,get_node_infos,\n                                 [{\'ns_1@172.23.105.190\',\n                                   {exit,\n                                    {{linked_process_died,<20245.8315.0>,\n                                      {no_connection,"fts-service_api"}},\n                                     {gen_server,call,\n                                      [{\'service_agent-fts\',\n                                        \'ns_1@172.23.105.190\'},\n                                       {if_rebalance,<0.11100.2>,\n                                        get_node_info},\n                                       infinity]}}}}]}}}}', u'shortText': u'message', u'serverTime': u'2017-06-21T11:32:41.178Z', u'module': u'ns_orchestrator', u'tstamp': 1498069961178, u'type': u'critical'}