I believe there's a potential race condition when acquiring credentials using the 'AuthProvider' interface, the general steps would be as follows (when applied to 'cbbackupmgr'):
- SDK prepares to dispatch request
- SDK asks 'cbbackupmgr' the credentials for a given host:port
- 'cbbackupmgr' uses gRPC to ask for credentials from 'cbbs' (the backup service) via 'cbauth'
- Cluster node required for step 1 is removed from the cluster
- 'cbbs' fails to find credentials for the given host:port
- 'cbbackupmgr' falls back to the credentials provided via the CLI (which will be wrong due to being a system user)
'cbbackupmgr' should be able to inform the SDK that we can't provide credentials for the given host:port and that it should retry the request (preferably after a cluster config update) so that it's dispatched to a different node (or fails because no other nodes run the desired service).