Preemptively 'auto-detecting' new nodes is not supported via CCCP (memcached) bootstrap; only via the HTTP 'streaming' config, which is disabled by default for the library.

You can specify `bootstrap_on=http` in the connection string to use old, 'streaming-style' config updates which will notify the client whenever there is a configuration change. Otherwise the client only checks if there is a new configuration in the case of an error.

Considering that most deployments of the library would be using a combination of KV and query operations, this is not considered an issue at the moment. For those running 'query-only' workloads, they can specify the extra option in the connection string.

Note that this same problem appears in 'view-only' connections, but in practice does not seem to be a common deployment configuration on the client.

Thanks Mark. I assume this is an issue across all SDKs. does the option exist in all SDKs or should I open a separate bug for each sdk?

Also, Can't we do this refresh on the topology for the default connection mode? I think this is basic for online elasticity. We'd need to trigger the topology updates without triggering with an error?

We update for both views and n1ql query via a regular polling interval. I remember we covered this once before for PHP and solved it there with a similar backstop, I thought. If not, I believe we need to. This is one of those 'behavioral' things we need to document better and verify across SDKs.

Java definitely has this 'backstop' and I believe .NET does as well. I'll ask Brett to comment here.

The Java client can use the streaming config (though it's off by default), in addition it might end up doing some intermittent polling for new configurations (this is an implementation detail; I'm not sure if it still does so). I'm unsure what .NET does, but it might use polling as well.

Regarding 'polling'. It's technically possible to do so in the C library as well, but has its own issues: consider the impact of having several hundred client instances (as would be the case with PHP) sending/receiving config information every few seconds.

We've had serious scalability issues in the past with having many open connections to port 8091, in addition to the fact that push-based config updates are inherently unreliable (what if the node sending you the updates becomes unresponsive). The issue is a bit specific to common deployments of the C client (many smaller application instances, each using a library instance; typically several hundred+) than the Java or .NET clients (one large application instance, one large client instance, multiple threads).

Polling shouldn't be difficult to add to the C library. Please file a bug if you think this is the correct solution (rather than just using the streaming config). This will add ~8k of traffic every 10 seconds or so per client instance.

The Java and .NET polling is via Carrier Publication, not HTTP. Thus, it does not bother port 8091 at all. The backstop is 10s IIRC on Java. I remember this first coming up on PHP where a user had a views only workload back in the 2.0 days.

IIRC the issue with PHP was that it wasn't detecting when a node was removed, and would return with errors (non-200 HTTP return codes) when contacting that node. This was fixed by having LCB taking any view API request with a non-200 return code as a cue to refresh the config.

The issue in this ticket however is that lcb is failing to take advantage of a new node added to the cluster: no errors are being returned, but existing instances aren't managing to take advantage of the newly added node.

I seem to remember solving for both cases though Mark. Let's see what Brett's comments are on expected behavior.

Resurrecting this for fast-failover.