Poll regularly for config updates

Description

start a cluster with 2 query nodes
start cbc-n1qlback with some query
add a new query node to the cluster and rebalance
observe the request/sec per node

expected: topology changes should eb automatically picked up by the clients. after rebalance the new query node needs to be part of the round robin requests being sent to the cluster. however new node does not start taking traffic even after a long wait.

if the load is stopped and restarted, the requests do go to the newly added node as well. However this means topology changes would require a restart of the app servers. that does cause admin overhead and possibly failed requests for the app and downtime.

Environment

build 3508 running cbc-n1qlback

Gerrit Reviews

None

Release Notes Description

None

relates to

Activity

Mark Nunberg April 10, 2017 at 10:43 PM

Resurrecting this for fast-failover.

Matt Ingenthron July 28, 2015 at 3:33 AM

I seem to remember solving for both cases though Mark. Let's see what Brett's comments are on expected behavior.

Mark Nunberg July 27, 2015 at 2:09 AM
Edited

IIRC the issue with PHP was that it wasn't detecting when a node was removed, and would return with errors (non-200 HTTP return codes) when contacting that node. This was fixed by having LCB taking any view API request with a non-200 return code as a cue to refresh the config.

The issue in this ticket however is that lcb is failing to take advantage of a new node added to the cluster: no errors are being returned, but existing instances aren't managing to take advantage of the newly added node.

Matt Ingenthron July 27, 2015 at 2:02 AM

The Java and .NET polling is via Carrier Publication, not HTTP. Thus, it does not bother port 8091 at all. The backstop is 10s IIRC on Java. I remember this first coming up on PHP where a user had a views only workload back in the 2.0 days.

Mark Nunberg July 27, 2015 at 1:30 AM
Edited

Polling shouldn't be difficult to add to the C library. Please file a bug if you think this is the correct solution (rather than just using the streaming config). This will add ~8k of traffic every 10 seconds or so per client instance.

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Fix versions

Labels

Reporter

Affects versions

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created July 26, 2015 at 8:22 PM
Updated April 19, 2017 at 11:08 PM
Resolved April 19, 2017 at 11:08 PM
Instabug