cbbackupmgr should retry failed REST requests due to temprorary failures

Description

What's the issue?
'cbbackupmgr' currently fails fast when receiving an unexpected response code from a HTTP REST request. This isn't ideal as it's clear that some services may respond with temporary failure status codes (for example the 503 status).

What's the fix?
The CC codebase already supports generic retrying of REST requests which fail due to expected status code, we simply need to plug in known failure codes so that they're automatically retried.

Example of an Analytics 503 status code
Repo "c360_no_analytics":

[root@s60801-cnt7 backups]# /opt/couchbase/bin/cbbackupmgr backup -c localhost -u perry -p 'Krugster83!' -a /backups/ -r c360_config_customer_dashboard Backing up to '2021-01-25T02_13_13.992344145-08_00' Transferring Analytics metadata for 'config' 0 items / 0B [==================================================================================================================================================================] 100.00% Error backing up cluster: Rest client error (GET http://172.23.97.84:8095/analytics/backup?bucket=config): Received error 503 while executing "GET http://172.23.97.84:8095/analytics/backup?bucket=config"

Server logs:
https://s3.amazonaws.com/cb-engineering/perry/timers_lost/collectinfo-2021-01-25T091645-ns_1%40172.23.97.84.zip
https://s3.amazonaws.com/cb-engineering/perry/timers_lost/collectinfo-2021-01-25T091645-ns_1%40172.23.97.85.zip
https://s3.amazonaws.com/cb-engineering/perry/timers_lost/collectinfo-2021-01-25T091645-ns_1%40172.23.97.86.zip

and backup logs attached to this ticket

Components

Fix versions

Environment

None

Release Notes Description

None

Activity

Show:

Arunkumar Senthilnathan January 27, 2022 at 7:23 PM

No impact to regression post this change

Arunkumar Senthilnathan September 21, 2021 at 3:42 PM

As per discussion with dev, we just need to ensure regression runs okay for this change

James Lee June 7, 2021 at 10:04 AM

Marking as resolved, retrying temporary failures was added while fixing https://couchbasecloud.atlassian.net/browse/MB-45152#icft=MB-45152.

James Lee January 25, 2021 at 5:10 PM

Hi ,

I've updated the issue accordingly, thanks for the response.

Daniel Owen January 25, 2021 at 4:37 PM

Hi , I think we should consider this an improvement to make use of the retry information - and so defer from CC.

Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

James Lee

Reporter

Story Points

1

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created January 25, 2021 at 10:59 AM
Updated August 31, 2024 at 11:05 AM
Resolved June 7, 2021 at 10:04 AM
Instabug