When connected to the EPT/streaming node and the node is "frozen" or dies silently otherwise (doesn't force the closing of the chunked socket), the connection stays established.
This can easily be reproduced outside of the client by connecting the browser to the streaming URL and then freezing a VM. The browser will still "spin" and wait for new chunks to come up.
The proposed solution is to have a netty handler in place that raises a exception when there is not traffic for N number of seconds (like 30) over the streaming connection. After this is detected, we have two possibilities:
- reconnect completely, but this involves lots of overhead every 30 seconds.
- send a HTTP HEAD packet and only if this doesnt work out reconnect. This means in the normal case we only have a HTTP HEAD request sent every 30 seconds, not much overhead.
If this fails, we then trigger the reconfigure.
Netty has a ReadTimeoutHandler to help with this. My POC already kinda works, I just need to find a way to properly distinguish the HEAD response on the ResponseHandler from regular chunks that arrive from the same channel.