Hostnames are not re-resolved in cases of IP address change

The Java SDK does not appear to attempt to re-resolve any hostnames past the point of the initial bootstrap, even if it loses connection to that node.

I have been able to reproduce this in a rather round about way with the help of Phil Stott. We created a simple test application (source attached) which connects to a 2 node cluster, and then tries to read all the documents in that cluster in a loop. After the first successful round of reading, the application is paused for 60 seconds during which time one of the nodes (timtest1) was rebooted and brought back up with a new IP address (same hostname). Once the node comes back online successfully with its new IP address, the application is still not able to connect to this node.

We tested this under two scenarios, one being with the default Java Security property for networkaddress.cache.ttl and the other with it being set to 0 so as to remove DNS caching from the equation on the JVM side of things. Logs for both scenarios are included.

Expected behaviour in the instance would be to attempt to re-resolve the IP address of the node if we are unable to connect to it, rather than using the IP address we got from the initial resolution. This is especially needed for use cases which involve the CB Operator, since this may change the IP addresses of nodes in the cluster during normal operation.