[CBBS][System Test] Starting a backup can fail due to stale authentication

What is the problem?
In a QE test we got:

2023-10-26T08:00:15.849-07:00 ERROR (Dispatcher) Cannot verify target bucket exists {"bucket": {"name":"bucket5","uuid":"f84b319e4f76f70bf48edfe4eb62d495"}, "err": "could not create REST client: failed to bootstrap client: failed to connect to any host(s) from the connection string, check username and password"}

when trying to run a backup. After creating a pcap I found that our authentication was being rejected by ns_server. The backup service uses cbauth to get credentials, but it does it once on application startup. Killing the service and letting it restart fixed the issue.

Reproduction

1. Create a cluster with a backup service
2. Create a repository that backs up a specific bucket
3. Do curl -v -u Administrator:asdasd localhost:9000/node/controller/rotateInternalCredentials -XPOST
4. Try to run a backup from the backup service

What is the solution?
I believe we should be asking cbauth for the credentials every time we try to make a REST request.