Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-1691

Unable to find Restore Label/Name of Restore

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • None
    • None
    • kubernetes, operator
    • None
    • 1

    Description

      Steps to recreate:

      1) Using the image that is the fix for K8S-1679.

      Type Reason Age From Message
       ---- ------ ---- ---- -------
       Normal Scheduled 104s default-scheduler Successfully assigned default/my-backup-full-1601335800-cb8dg to minikube
       Normal Pulled 103s kubelet, minikube Container image "couchbase/operator-backup:6.5.1-111" already present on machine
       Normal Created 103s kubelet, minikube Created container cbbackupmgr-full
       Normal Started 103s kubelet, minikube Started container cbbackupmgr-full

      2) Back up was created successfully:

      2020-09-28 23:30:10,855 - root - INFO - epoch: 1601335810 
      2020-09-28 23:30:10,856 - root - INFO - timestamp: 2020-09-28T23_30_10 
      2020-09-28 23:30:10,856 - root - INFO - Namespace(backup_ret='24.00', cacert=None, cluster='cb-example', config=
      'true', end=None, log_ret='24.00', mode='backup', repo=None, start=None, strategy='full_incremental', verbosity=
      'INFO') 
      2020-09-28 23:30:10,856 - root - INFO - start logRetention check 
      2020-09-28 23:30:10,856 - root - INFO - removed 0 logs in /data/scriptlogs/incremental 
      2020-09-28 23:30:10,856 - root - INFO - mode: BACKUP 
      2020-09-28 23:30:10,856 - root - INFO - Perform CONFIG: new Repo to be created 
      2020-09-28 23:30:10,856 - root - INFO - Strategy: full_incremental 
      2020-09-28 23:30:10,856 - root - INFO - Perform FULL BACKUP 
      mkdir: cannot create directory ‘/data/backups’: File exists 
      rm: cannot remove '/data/backups/lock.lk': No such file or directory 
      2020-09-28 23:30:10,861 - root - INFO - backup archive /data/backups already exists 
      2020-09-28 23:30:10,861 - root - INFO - config true, config needs to be performed 
      2020-09-28 23:30:10,861 - root - INFO - attempting to create repo: cb-example-2020-09-28T23_30_10 
      2020-09-28 23:30:10,878 - root - INFO - b'Backup repository `cb-example-2020-09-28T23_30_10` created successfull
      y in archive `/data/backups`\n' 
      2020-09-28 23:30:10,879 - root - INFO - attempt to write status 
      2020-09-28 23:30:10,879 - root - INFO - status should have been hopefully written to 
      2020-09-28 23:30:10,879 - root - INFO - attempting to query K8S objects 
      2020-09-28 23:30:10,880 - root - INFO - k8s config loaded 
      2020-09-28 23:30:10,880 - root - INFO - working in namespace: default 
      2020-09-28 23:30:10,923 - root - INFO - BACKUP start 
      2020-09-28 23:30:10,923 - root - INFO - backing up to: cb-example-2020-09-28T23_30_10 
      2020-09-28 23:30:14,985 - root - INFO - Starting Backup cleanup 
      2020-09-28 23:30:14,985 - root - INFO - Time 2020-09-28 23:30:14.985481, Retention 1 day, 0:00:00, Threshold 202
      0-09-27 23:30:14.985481 
      2020-09-28 23:30:14,985 - root - INFO - Considering repository cb-example-2020-09-28T23_30_10 ... 
      2020-09-28 23:30:14,985 - root - INFO - Creation time within threshold, continuing 
      2020-09-28 23:30:15,010 - root - INFO - Exiting Script, success. Output: {"location": "2020-09-28T23_30_10.97088
      4413Z", "duration_seconds": "3.976066463", "avg_data_transfer_rate_bytes_sec": 762839, "total_items": 7303, "tot
      al_items_size_bytes": 3030384, "buckets": {"beer-sample": {"mutations_backedup": "7303", "mutations_failed": "0"
      , "deletions_backedup": "0", "deletions_failed": "0"}}} 
      (422) 
      Reason: Unprocessable Entity 
      HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json',
       'Date': 'Mon, 28 Sep 2020 23:30:10 GMT', 'Content-Length': '383'}) 
      HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"CouchbaseBack
      up.couchbase.com \"my-backup\" is invalid: status.backups.full: Required value","reason":"Invalid","details":{"n
      ame":"my-backup","group":"couchbase.com","kind":"CouchbaseBackup","causes":[{"reason":"FieldValueRequired","mess
      age":"Required value","field":"status.backups.full"}]},"code":422}

      3) We see the following error when creating a restore, this is with specifying a specific repo in the restore resource yaml:

      2020-09-28 23:32:55,423 - root - INFO - epoch: 1601335975
      2020-09-28 23:32:55,423 - root - INFO - timestamp: 2020-09-28T23_32_55
      2020-09-28 23:32:55,423 - root - INFO - Namespace(backup_ret=None, cacert=None, cluster='cb-example', config=False, end='oldest', log_ret='24.00', mode='restore', repo='cb-example-2020-09-28T23_30_10', start='oldest', strategy=None, verbosity='INFO')
      2020-09-28 23:32:55,423 - root - INFO - start logRetention check
      2020-09-28 23:32:55,423 - root - INFO - removed 0 logs in /data/scriptlogs/restore
      2020-09-28 23:32:55,424 - root - INFO - mode: RESTORE
      mkdir: cannot create directory ‘/data/backups’: File exists
      rm: cannot remove '/data/backups/lock.lk': No such file or directory
      2020-09-28 23:32:55,428 - root - INFO - attempting to query K8S objects
      2020-09-28 23:32:55,428 - root - INFO - k8s config loaded
      2020-09-28 23:32:55,429 - root - INFO - working in namespace: default
      Traceback (most recent call last):
       File "/opt/couchbase/bin/backup_script", line 1125, in <module>
       main()
       File "/opt/couchbase/bin/backup_script", line 166, in main
       k8s_setup(args)
       File "/opt/couchbase/bin/backup_script", line 455, in k8s_setup
       cbrestore = currentpod.metadata.labels[RESTORE_LABEL]
      KeyError: 'couchbase_restore'

       

      Strangely, repeating the process with the following image, restore was sucessful:

      couchbase/operator-backup:6.5.0

       

      2020-09-28 23:20:25,983 - root - INFO - epoch: 1601335225 
      2020-09-28 23:20:25,983 - root - INFO - timestamp: 2020-09-28T23_20_25 
      2020-09-28 23:20:25,983 - root - INFO - Namespace(backup_ret=None, cacert=None, cluster='cb-example', config=Fal
      se, end='oldest', log_ret='24.00', mode='restore', repo='cb-example-2020-09-28T23_15_41', start='oldest', strate
      gy=None, verbosity='INFO') 
      2020-09-28 23:20:25,983 - root - INFO - start logRetention check 
      2020-09-28 23:20:25,983 - root - INFO - removed 0 logs in /data/scriptlogs/restore 
      2020-09-28 23:20:25,983 - root - INFO - mode: RESTORE 
      mkdir: cannot create directory ‘/data/backups’: File exists 
      rm: cannot remove '/data/backups/lock.lk': No such file or directory 
      2020-09-28 23:20:25,988 - root - INFO - attempting to query K8S objects 
      NoneType: None 
      2020-09-28 23:20:25,989 - root - INFO - k8s config loaded 
      2020-09-28 23:20:25,989 - root - INFO - working in namespace: default 
      2020-09-28 23:20:26,014 - root - INFO - running on POD[my-restore-n5htm] for JOB[my-restore] for RESTORE[my-rest
      ore] 
      2020-09-28 23:20:26,014 - root - INFO - current CouchbaseBackupRestore object: my-restore 
      2020-09-28 23:20:26,101 - root - INFO - rfoldersepos  
      2020-09-28 23:20:26,101 - root - INFO - ['/data/backups/cb-example-2020-09-28T23_15_41/2020-09-28T23_15_41.68796
      0388Z/'] 
      2020-09-28 23:20:26,192 - root - INFO - RESTORE start 
      2020-09-28 23:20:26,192 - root - INFO - restoring from: cb-example-2020-09-28T23_15_41 
      2020-09-28 23:20:29,499 - root - INFO - rfoldersepos  
      2020-09-28 23:20:29,500 - root - INFO - ['/data/backups/cb-example-2020-09-28T23_15_41/2020-09-28T23_15_41.68796
      0388Z/'] 
      2020-09-28 23:20:29,545 - root - INFO - Exiting Script, success. Output: {"buckets": {"beer-sample": {"mutations
      ": {"succeeded": "0", "failed": "0", "skipped": "7303"}, "deletions": {"succeeded": "0", "failed": "0", "skipped
      ": "0"}}}, "backups_restored": "1", "duration_seconds": "3.235310705", "avg_data_transfer_rate_bytes_sec": 91471
      8, "total_items": "7303", "total_items_size_bytes": 2956984}

      Logs are attached

      Attachments

        1. backup.yaml
          0.4 kB
        2. cb-example.yaml
          2 kB
        3. cbopinfo-20200928T163603-0700.tar.gz
          395 kB
        4. restore.yaml
          0.3 kB
        For Gerrit Dashboard: K8S-1691
        # Subject Branch Project Status CR V

        Activity

          People

            daniel.ma Daniel Ma (Inactive)
            tin.tran Tin Tran (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty