Index rebalance can result in very large number of TIME_WAIT connections. There is a bug in waitForIndexBuild routine which causes it not to terminate at the end of the batch. Rather, it remains active till the end of rebalance. Due to the bug, it reaches a state where it will be in a for loop collection stats/status till next batch or end of rebalance. Each such collection creates new http.Client object which will lead to TIME_WAIT connections when freed by golang GC.
This can lead to rebalance failure with following error:
|For Gerrit Dashboard: MB-41722|
|137464,1||MB-41722 [BP 6.6.1] fix termination for waitForIndexBuild||mad-hatter||indexing||Status: MERGED||+2||+1|
|137631,1||MB-41722 [BP 6.6.1] fix termination for waitForIndexBuild||6.6.0||indexing||Status: MERGED||+2||+1|