Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61280

Backup Service: Existing encrypted repository encryption config corrupted when km_credential_name used to create a new repository and error occurs

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • 7.6.2
    • 7.6.0
    • tools
    • Untriaged
    • 0
    • No
    • Tools 2024-Q1

    Description

      Was creating Backup Service encrypted archive repositories using the Backup Service when I ran into this issue.

      Repro using AWS KMS key:

      1) Create an encrypted archive repository.

       
      % curl -X POST http://localhost:8097/api/v1/cluster/self/repository/active/test_encrypted_repo -u Administrator:'password' -d '{"plan":"_daily_backups", "archive":"/Users/hyun-ju.vega/cb_backup_20240324", "encryption_key_url":"awskms://alias/hjvtestkey", "km_auth_type":"credentials", "km_credentials_id":"AKIAWIDGYGXXXXXXXXXX", "km_credentials_key":"sfTsIusZtmAyT1ORzm1RXNqRUATwOBXXXXXXXXXX", "km_region":"us-west-2"}'
      

      2) Run an on-demand full backup from the Backup Service UI in the test_encrypted_repo to be sure that the backup runs OK.

      3) GET the description for the archive repository created in 1) using the REST API.

      {
        "id": "test_encrypted_repo",
        "plan_name": "_daily_backups",
        "state": "active",
        "archive": "/Users/hyun-ju.vega/cb_backup_20240324",
        "repo": "b464e958-1705-4a27-b776-fa21e01b9775",
        "scheduled": {
      ...
        },
        "version": 1,
        "health": {
          "healthy": true
        },
        "creation_time": "2024-03-24T23:04:35.238561-07:00",
        "update_time": "2024-03-24T23:04:35.238561-07:00",
        "encryption_opts": {
          "key_url": "awskms://alias/hjvtestkey",
          "auth_type": "credentials",
          "region": "us-west-2",
          "creds_name": "615901c9-3970-4535-a753-3ac8a5aadfa9"
        }
      }
      

      4) Create a second backup repository using AWS KMS encryption using the same credentials as in 1) – use the "creds_name" value from 3) for km_credential_name. Also, don't include "km_region" so that the REST API command to create the repository returns an error.

      % curl -X POST http://localhost:8097/api/v1/cluster/self/repository/active/test_encrypted_repo2 -u Administrator:'password' -d '{"plan":"_daily_backups", "archive":"/Users/hyun-ju.vega/cb_backup_20240324_2", "encryption_key_url":"awskms://alias/hjvtestkey", "km_auth_type":"credentials", "km_credential_name":"615901c9-3970-4535-a753-3ac8a5aadfa9"}'
       
      {"status":500,"msg":"Could not create cbbackupmgr repository","extras":"exit status 1: Backup repository creation failed: a key region is required\n"
      

      5)
      Now try creating again, this time, including the "km_region":"us-west-2" – you will get an error saying that the "given credential set does not exits" (typo):

      % curl -X POST http://localhost:8097/api/v1/cluster/self/repository/active/test_encrypted_repo2 -u Administrator:'password' -d '{"plan":"_daily_backups", "archive":"/Users/hyun-ju.vega/cb_backup_20240324_2", "encryption_key_url":"awskms://alias/hjvtestkey", "km_auth_type":"credentials", "km_credential_name":"615901c9-3970-4535-a753-3ac8a5aadfa9", "km_region":"us-west-2"}'
      {"status":400,"msg":"Invalid encryption options","extras":"given credential set does not exits"}
      

      6)
      Now try running an on-demand full backup from the Backup Service UI in the test_encrypted_repo again. This time you'll get the below error message, and the backup will not run.

      *Could not trigger backup: Could not send task. could not dispatch task: rpc error: code = Internal desc = failed to set encryption config for repository test_encrypted_repo: could not get KM credentials: could not get cloud credentials: element not found
      *
      It appears that the error while creating the second repository with the km_credential_name from this repository may have removed the credentials or caused some issue so that this repository is no longer functional.

      Note that if the second repository had been created OK in step 4 (i.e. if the km_region had been included in the body of the POST), both repositories would be working OK.

      Attachments

        For Gerrit Dashboard: MB-61280
        # Subject Branch Project Status CR V

        Activity

          People

            Nour.Elmenyawi Nour Elmenyawi
            hyun-ju.vega Hyun-Ju Vega
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There is 1 open Gerrit change

                PagerDuty