Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-1691

Unable to find Restore Label/Name of Restore

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • None
    • kubernetes, operator
    • None
    • 1

    Description

      Steps to recreate:

      1) Using the image that is the fix for K8S-1679.

      Type Reason Age From Message
       ---- ------ ---- ---- -------
       Normal Scheduled 104s default-scheduler Successfully assigned default/my-backup-full-1601335800-cb8dg to minikube
       Normal Pulled 103s kubelet, minikube Container image "couchbase/operator-backup:6.5.1-111" already present on machine
       Normal Created 103s kubelet, minikube Created container cbbackupmgr-full
       Normal Started 103s kubelet, minikube Started container cbbackupmgr-full

      2) Back up was created successfully:

      2020-09-28 23:30:10,855 - root - INFO - epoch: 1601335810 
      2020-09-28 23:30:10,856 - root - INFO - timestamp: 2020-09-28T23_30_10 
      2020-09-28 23:30:10,856 - root - INFO - Namespace(backup_ret='24.00', cacert=None, cluster='cb-example', config=
      'true', end=None, log_ret='24.00', mode='backup', repo=None, start=None, strategy='full_incremental', verbosity=
      'INFO') 
      2020-09-28 23:30:10,856 - root - INFO - start logRetention check 
      2020-09-28 23:30:10,856 - root - INFO - removed 0 logs in /data/scriptlogs/incremental 
      2020-09-28 23:30:10,856 - root - INFO - mode: BACKUP 
      2020-09-28 23:30:10,856 - root - INFO - Perform CONFIG: new Repo to be created 
      2020-09-28 23:30:10,856 - root - INFO - Strategy: full_incremental 
      2020-09-28 23:30:10,856 - root - INFO - Perform FULL BACKUP 
      mkdir: cannot create directory ‘/data/backups’: File exists 
      rm: cannot remove '/data/backups/lock.lk': No such file or directory 
      2020-09-28 23:30:10,861 - root - INFO - backup archive /data/backups already exists 
      2020-09-28 23:30:10,861 - root - INFO - config true, config needs to be performed 
      2020-09-28 23:30:10,861 - root - INFO - attempting to create repo: cb-example-2020-09-28T23_30_10 
      2020-09-28 23:30:10,878 - root - INFO - b'Backup repository `cb-example-2020-09-28T23_30_10` created successfull
      y in archive `/data/backups`\n' 
      2020-09-28 23:30:10,879 - root - INFO - attempt to write status 
      2020-09-28 23:30:10,879 - root - INFO - status should have been hopefully written to 
      2020-09-28 23:30:10,879 - root - INFO - attempting to query K8S objects 
      2020-09-28 23:30:10,880 - root - INFO - k8s config loaded 
      2020-09-28 23:30:10,880 - root - INFO - working in namespace: default 
      2020-09-28 23:30:10,923 - root - INFO - BACKUP start 
      2020-09-28 23:30:10,923 - root - INFO - backing up to: cb-example-2020-09-28T23_30_10 
      2020-09-28 23:30:14,985 - root - INFO - Starting Backup cleanup 
      2020-09-28 23:30:14,985 - root - INFO - Time 2020-09-28 23:30:14.985481, Retention 1 day, 0:00:00, Threshold 202
      0-09-27 23:30:14.985481 
      2020-09-28 23:30:14,985 - root - INFO - Considering repository cb-example-2020-09-28T23_30_10 ... 
      2020-09-28 23:30:14,985 - root - INFO - Creation time within threshold, continuing 
      2020-09-28 23:30:15,010 - root - INFO - Exiting Script, success. Output: {"location": "2020-09-28T23_30_10.97088
      4413Z", "duration_seconds": "3.976066463", "avg_data_transfer_rate_bytes_sec": 762839, "total_items": 7303, "tot
      al_items_size_bytes": 3030384, "buckets": {"beer-sample": {"mutations_backedup": "7303", "mutations_failed": "0"
      , "deletions_backedup": "0", "deletions_failed": "0"}}} 
      (422) 
      Reason: Unprocessable Entity 
      HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json',
       'Date': 'Mon, 28 Sep 2020 23:30:10 GMT', 'Content-Length': '383'}) 
      HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"CouchbaseBack
      up.couchbase.com \"my-backup\" is invalid: status.backups.full: Required value","reason":"Invalid","details":{"n
      ame":"my-backup","group":"couchbase.com","kind":"CouchbaseBackup","causes":[{"reason":"FieldValueRequired","mess
      age":"Required value","field":"status.backups.full"}]},"code":422}

      3) We see the following error when creating a restore, this is with specifying a specific repo in the restore resource yaml:

      2020-09-28 23:32:55,423 - root - INFO - epoch: 1601335975
      2020-09-28 23:32:55,423 - root - INFO - timestamp: 2020-09-28T23_32_55
      2020-09-28 23:32:55,423 - root - INFO - Namespace(backup_ret=None, cacert=None, cluster='cb-example', config=False, end='oldest', log_ret='24.00', mode='restore', repo='cb-example-2020-09-28T23_30_10', start='oldest', strategy=None, verbosity='INFO')
      2020-09-28 23:32:55,423 - root - INFO - start logRetention check
      2020-09-28 23:32:55,423 - root - INFO - removed 0 logs in /data/scriptlogs/restore
      2020-09-28 23:32:55,424 - root - INFO - mode: RESTORE
      mkdir: cannot create directory ‘/data/backups’: File exists
      rm: cannot remove '/data/backups/lock.lk': No such file or directory
      2020-09-28 23:32:55,428 - root - INFO - attempting to query K8S objects
      2020-09-28 23:32:55,428 - root - INFO - k8s config loaded
      2020-09-28 23:32:55,429 - root - INFO - working in namespace: default
      Traceback (most recent call last):
       File "/opt/couchbase/bin/backup_script", line 1125, in <module>
       main()
       File "/opt/couchbase/bin/backup_script", line 166, in main
       k8s_setup(args)
       File "/opt/couchbase/bin/backup_script", line 455, in k8s_setup
       cbrestore = currentpod.metadata.labels[RESTORE_LABEL]
      KeyError: 'couchbase_restore'

       

      Strangely, repeating the process with the following image, restore was sucessful:

      couchbase/operator-backup:6.5.0

       

      2020-09-28 23:20:25,983 - root - INFO - epoch: 1601335225 
      2020-09-28 23:20:25,983 - root - INFO - timestamp: 2020-09-28T23_20_25 
      2020-09-28 23:20:25,983 - root - INFO - Namespace(backup_ret=None, cacert=None, cluster='cb-example', config=Fal
      se, end='oldest', log_ret='24.00', mode='restore', repo='cb-example-2020-09-28T23_15_41', start='oldest', strate
      gy=None, verbosity='INFO') 
      2020-09-28 23:20:25,983 - root - INFO - start logRetention check 
      2020-09-28 23:20:25,983 - root - INFO - removed 0 logs in /data/scriptlogs/restore 
      2020-09-28 23:20:25,983 - root - INFO - mode: RESTORE 
      mkdir: cannot create directory ‘/data/backups’: File exists 
      rm: cannot remove '/data/backups/lock.lk': No such file or directory 
      2020-09-28 23:20:25,988 - root - INFO - attempting to query K8S objects 
      NoneType: None 
      2020-09-28 23:20:25,989 - root - INFO - k8s config loaded 
      2020-09-28 23:20:25,989 - root - INFO - working in namespace: default 
      2020-09-28 23:20:26,014 - root - INFO - running on POD[my-restore-n5htm] for JOB[my-restore] for RESTORE[my-rest
      ore] 
      2020-09-28 23:20:26,014 - root - INFO - current CouchbaseBackupRestore object: my-restore 
      2020-09-28 23:20:26,101 - root - INFO - rfoldersepos  
      2020-09-28 23:20:26,101 - root - INFO - ['/data/backups/cb-example-2020-09-28T23_15_41/2020-09-28T23_15_41.68796
      0388Z/'] 
      2020-09-28 23:20:26,192 - root - INFO - RESTORE start 
      2020-09-28 23:20:26,192 - root - INFO - restoring from: cb-example-2020-09-28T23_15_41 
      2020-09-28 23:20:29,499 - root - INFO - rfoldersepos  
      2020-09-28 23:20:29,500 - root - INFO - ['/data/backups/cb-example-2020-09-28T23_15_41/2020-09-28T23_15_41.68796
      0388Z/'] 
      2020-09-28 23:20:29,545 - root - INFO - Exiting Script, success. Output: {"buckets": {"beer-sample": {"mutations
      ": {"succeeded": "0", "failed": "0", "skipped": "7303"}, "deletions": {"succeeded": "0", "failed": "0", "skipped
      ": "0"}}}, "backups_restored": "1", "duration_seconds": "3.235310705", "avg_data_transfer_rate_bytes_sec": 91471
      8, "total_items": "7303", "total_items_size_bytes": 2956984}

      Logs are attached

      Attachments

        1. backup.yaml
          0.4 kB
        2. cb-example.yaml
          2 kB
        3. cbopinfo-20200928T163603-0700.tar.gz
          395 kB
        4. restore.yaml
          0.3 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          tin.tran Tin Tran added a comment -

          hi Daniel Ma

          For the next image release can we match it with the latest Couchbase Release (6.6.0)? 

          Currently our image name is up to 6.5.1-xxx and customers are asking about the names as they are concerned about compatibility. 

          Thank you Daniel

          tin.tran Tin Tran added a comment - hi Daniel Ma For the next image release can we match it with the latest Couchbase Release (6.6.0)?  Currently our image name is up to 6.5.1-xxx and customers are asking about the names as they are concerned about compatibility.  Thank you Daniel

          Unfortunately, because of cbbackupmgr compatibility matrix, we have to ship multiple backup images and document/tell users to pick the right image for the right version of Couchbase Server they are deploying in the CouchbaseServer resource.

          For example, we will have…

          images (as an example):

          {server-base}

          -

          {buildno}

          6.5.1-112 (based on Couchbase Server 6.5.1)
          6.6.0-102 (based on Couchbase Server 6.6.0)

          This gives them the compatibility they seek. It does mean, of course, if a user upgrades they need to change the backup image they are using.

          It also means that "patched" backup images would be coming in a later build number.

          ingenthr Matt Ingenthron added a comment - Unfortunately, because of cbbackupmgr compatibility matrix, we have to ship multiple backup images and document/tell users to pick the right image for the right version of Couchbase Server they are deploying in the CouchbaseServer resource. For example, we will have… images (as an example): {server-base} - {buildno} 6.5.1-112 (based on Couchbase Server 6.5.1) 6.6.0-102 (based on Couchbase Server 6.6.0) This gives them the compatibility they seek. It does mean, of course, if a user upgrades they need to change the backup image they are using. It also means that "patched" backup images would be coming in a later build number.
          tin.tran Tin Tran added a comment -

          Hi Matt Ingenthron

          Thank you for the update, does that mean the new backup image will have 2 versions? 

          one for Couchbase Server 6.5.1 and one for 6.6.0?

          Thank you Matt.

          tin.tran Tin Tran added a comment - Hi Matt Ingenthron Thank you for the update, does that mean the new backup image will have 2 versions?  one for Couchbase Server 6.5.1 and one for 6.6.0? Thank you Matt.

          Yes, unfortunately.

          At the moment, per guidance from the tools team team, `cbbackupmgr` is only supported as compatible within a particular major.minor. So that support restriction falls down lower to what we're doing with cbbackupmgr.

          ingenthr Matt Ingenthron added a comment - Yes, unfortunately. At the moment, per guidance from the tools team team, `cbbackupmgr` is only supported as compatible within a particular major.minor. So that support restriction falls down lower to what we're doing with cbbackupmgr.
          tin.tran Tin Tran added a comment -

          Thank you Matt Ingenthron, I will keep a lookout for the images and make the accordance recommendations to customers.

          tin.tran Tin Tran added a comment - Thank you Matt Ingenthron , I will keep a lookout for the images and make the accordance recommendations to customers.

          People

            daniel.ma Daniel Ma (Inactive)
            tin.tran Tin Tran
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty