Major Description
We currently use a singleton CouchbaseHttpClient which inherits from HttpClient. Using a singleton HttpClient is common practice and known to A) improve performance by allowing HTTP keep-alives and reducing DNS requests and B) avoid problems with TCP port depletion.
However, the common practice is misunderstood. In fact, it is the HttpClientHandler within the HttpClient that should be reused, more so than the HttpClient itself. Additionally, it should only be reused for a limited period of time. Failure to replace the HttpClientHandler occasionally can cause unexpected network failures if DNS names are changed to refer to new IP addresses.
More importantly, our current pattern of a singleton CouchbaseHttpClient introduces side effects. There are multiple places in the codebase where we are setting properties on this client as if it was transient when it is in fact a singleton. Setting these properties is affecting other usages of the singleton by other internal services. Examples include setting the Timeout to infinite or setting a per-connection ID on DefaultRequestHeaders.