cbbackupmgr should retry failed REST requests due to temprorary failures
Description
Components
Fix versions
Labels
Environment
None
Release Notes Description
None
clones
Activity
Show:
Arunkumar Senthilnathan January 27, 2022 at 7:23 PM
No impact to regression post this change
Arunkumar Senthilnathan September 21, 2021 at 3:42 PM
As per discussion with dev, we just need to ensure regression runs okay for this change
James Lee June 7, 2021 at 10:04 AM
Marking as resolved, retrying temporary failures was added while fixing https://couchbasecloud.atlassian.net/browse/MB-45152#icft=MB-45152.
James Lee January 25, 2021 at 5:10 PM
Hi @Daniel Owen,
I've updated the issue accordingly, thanks for the response.
Daniel Owen January 25, 2021 at 4:37 PM
Hi @James Lee, I think we should consider this an improvement to make use of the retry information - and so defer from CC.
Done
Pinned fields
Click on the next to a field label to start pinning.
Details
Assignee
Reporter
Perry Krug
Perry Krug(Deactivated)Story Points
1
Priority
Trivial
Instabug
Open Instabug
PagerDuty
PagerDuty Incident
PagerDuty Incident
Sentry
Linked Issues
Linked Issues
Zendesk Support
Linked Tickets
Linked Tickets
Created January 25, 2021 at 10:59 AM
Updated August 31, 2024 at 11:05 AM
Resolved June 7, 2021 at 10:04 AM
Instabug
What's the issue?
'
cbbackupmgr
' currently fails fast when receiving an unexpected response code from a HTTP REST request. This isn't ideal as it's clear that some services may respond with temporary failure status codes (for example the 503 status).What's the fix?
The CC codebase already supports generic retrying of REST requests which fail due to expected status code, we simply need to plug in known failure codes so that they're automatically retried.
Example of an Analytics 503 status code
Repo "c360_no_analytics":
[root@s60801-cnt7 backups]# /opt/couchbase/bin/cbbackupmgr backup -c localhost -u perry -p 'Krugster83!' -a /backups/ -r c360_config_customer_dashboard Backing up to '2021-01-25T02_13_13.992344145-08_00' Transferring Analytics metadata for 'config' 0 items / 0B [==================================================================================================================================================================] 100.00% Error backing up cluster: Rest client error (GET http://172.23.97.84:8095/analytics/backup?bucket=config): Received error 503 while executing "GET http://172.23.97.84:8095/analytics/backup?bucket=config"
Server logs:
https://s3.amazonaws.com/cb-engineering/perry/timers_lost/collectinfo-2021-01-25T091645-ns_1%40172.23.97.84.zip
https://s3.amazonaws.com/cb-engineering/perry/timers_lost/collectinfo-2021-01-25T091645-ns_1%40172.23.97.85.zip
https://s3.amazonaws.com/cb-engineering/perry/timers_lost/collectinfo-2021-01-25T091645-ns_1%40172.23.97.86.zip
and backup logs attached to this ticket