Rebalance exited with reason service_rebalance_failed, index timeout

This is a showfast cluster which has 4 nodes and was running 5.1.1.

Two of the nodes, 172.23.104.212 and 172.23.96.147, have upgraded upgraded to Vulcan successfully.

While trying to rebalance out node 172.23.121.90, the process failed (but items seemed to have been transferred out).   FYI, this is the showfast live cluster.

Logs are attached:

 
https://s3.amazonaws.com/cb-customers/showfast/collectinfo-2018-07-17T045727-ns_1%40172.23.104.212.zip
https://s3.amazonaws.com/cb-customers/showfast/collectinfo-2018-07-17T045727-ns_1%40172.23.121.90.zip
https://s3.amazonaws.com/cb-customers/showfast/collectinfo-2018-07-17T045727-ns_1%40172.23.123.46.zip
https://s3.amazonaws.com/cb-customers/showfast/collectinfo-2018-07-17T045727-ns_1%40172.23.96.147.zip