connection may not recover until OS TCP timeout

In situations where the query node (or any HTTP service) goes offline without sending a TCP FIN, because we no longer pipeline (as of 1.4.2), the keepalive may not detect a stale TCP connection as intended.

To repro, set up continuous workload with 2 query nodes and pool size fixed. Forcibly take one down and restart it. This will cause the TCP connection to go half open, as no TCP FIN will be sent on "crash".

Observe that the workload is unbalanced after this.

Note that this may not be observable without fixed workload as new connecitons would be opened within the pool limit. Also, note that the client doesn't detect it as stale because it won't schedule other workload on the connection. Thus this requires a pretty specific set of circumstances to reproduce.

Workaround
Tune TCP timeout down to a shorter time interval or add TCP keep-alive at the OS level.