Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43846

cbbackupmgr should retry failed REST requests due to temprorary failures

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Done
    • Trivial
    • 7.1.0
    • 5.5.0, 5.5.1, 5.5.2, 5.5.3, 5.5.4, 5.5.5, 6.0.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.5.1, 6.6.0, 6.6.1, 6.5.0, Cheshire-Cat
    • tools
    • 1

    Description

      What's the issue?
      'cbbackupmgr' currently fails fast when receiving an unexpected response code from a HTTP REST request. This isn't ideal as it's clear that some services may respond with temporary failure status codes (for example the 503 status).

      What's the fix?
      The CC codebase already supports generic retrying of REST requests which fail due to expected status code, we simply need to plug in known failure codes so that they're automatically retried.

      Example of an Analytics 503 status code
      Repo "c360_no_analytics":

      [root@s60801-cnt7 backups]# /opt/couchbase/bin/cbbackupmgr backup -c localhost -u perry -p 'Krugster83!' -a /backups/ -r c360_config_customer_dashboard
      Backing up to '2021-01-25T02_13_13.992344145-08_00'
      Transferring Analytics metadata for 'config'                                                                                                                    0 items / 0B
      [==================================================================================================================================================================] 100.00%
      Error backing up cluster: Rest client error (GET http://172.23.97.84:8095/analytics/backup?bucket=config): Received error 503 while executing "GET http://172.23.97.84:8095/analytics/backup?bucket=config"
      

      Server logs:
      https://s3.amazonaws.com/cb-engineering/perry/timers_lost/collectinfo-2021-01-25T091645-ns_1%40172.23.97.84.zip
      https://s3.amazonaws.com/cb-engineering/perry/timers_lost/collectinfo-2021-01-25T091645-ns_1%40172.23.97.85.zip
      https://s3.amazonaws.com/cb-engineering/perry/timers_lost/collectinfo-2021-01-25T091645-ns_1%40172.23.97.86.zip

      and backup logs attached to this ticket

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              james.lee James Lee
              perry Perry Krug
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty