Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-1690

Backup/restore with Operator doesn't find the repo after changing the backup resource config

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • None
    • kubernetes, operator
    • None
    • 1

    Description

      The Restore resource cannot start due to 

      {"level":"error","ts":1601331434.5607915,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"no corresponding CouchbaseBackup Repo found"

      Steps to reproduce:

      1) create a backup resource as per the backup.yaml, using the couchbase/operator-backup:6.5.1-111 image

      Events: 
        Type    Reason     Age    From               Message 
        ----    ------     ----   ----               ------- 
        Normal  Scheduled  3m29s  default-scheduler  Successfully assigned default/my-backup-
      full-1601331000-tc7c4 to minikube 
        Normal  Pulling    3m29s  kubelet, minikube  Pulling image "couchbase/operator-backup:6.5.1-111" 
        Normal  Pulled     41s    kubelet, minikube  Successfully pulled image "couchbase/operator-backup:6.5.1-111" 
        Normal  Created    37s    kubelet, minikube  Created container cbbackupmgr-full 
        Normal  Started    37s    kubelet, minikube  Started container cbbackupmgr-full

      2) Let backup runs once, from the logs we can see the backup ran successfully.

       

      2020-09-28 22:12:56,415 - root - INFO - epoch: 1601331176 
      2020-09-28 22:12:56,415 - root - INFO - timestamp: 2020-09-28T22_12_56 
      2020-09-28 22:12:56,415 - root - INFO - Namespace(backup_ret='24.00', cacert=None, cluster='cb-example', config=
      'true', end=None, log_ret='24.00', mode='backup', repo=None, start=None, strategy='full_incremental', verbosity=
      'INFO') 
      2020-09-28 22:12:56,415 - root - INFO - start logRetention check 
      2020-09-28 22:12:56,416 - root - INFO - removed 0 logs in /data/scriptlogs/incremental 
      2020-09-28 22:12:56,416 - root - INFO - mode: BACKUP 
      2020-09-28 22:12:56,416 - root - INFO - Perform CONFIG: new Repo to be created 
      2020-09-28 22:12:56,416 - root - INFO - Strategy: full_incremental 
      2020-09-28 22:12:56,416 - root - INFO - Perform FULL BACKUP 
      rm: cannot remove '/data/backups/lock.lk': No such file or directory 
      2020-09-28 22:12:56,421 - root - INFO - performing config as backup archive was just created 
      2020-09-28 22:12:56,422 - root - INFO - config true, config needs to be performed 
      2020-09-28 22:12:56,422 - root - INFO - attempting to create repo: cb-example-2020-09-28T22_12_56 
      2020-09-28 22:12:56,437 - root - INFO - b'Backup repository `cb-example-2020-09-28T22_12_56` created successfull
      y in archive `/data/backups`\n' 
      2020-09-28 22:12:56,437 - root - INFO - attempt to write status 
      2020-09-28 22:12:56,437 - root - INFO - status should have been hopefully written to 
      2020-09-28 22:12:56,437 - root - INFO - attempting to query K8S objects 
      2020-09-28 22:12:56,438 - root - INFO - k8s config loaded 
      2020-09-28 22:12:56,438 - root - INFO - working in namespace: default 
      2020-09-28 22:12:56,476 - root - INFO - BACKUP start 
      2020-09-28 22:12:56,477 - root - INFO - backing up to: cb-example-2020-09-28T22_12_56 
      2020-09-28 22:13:00,649 - root - INFO - Starting Backup cleanup 
      2020-09-28 22:13:00,651 - root - INFO - Time 2020-09-28 22:13:00.651335, Retention 1 day, 0:00:00, Threshold 202
      0-09-27 22:13:00.651335 
      2020-09-28 22:13:00,651 - root - INFO - Considering repository cb-example-2020-09-28T22_12_56 ... 
      2020-09-28 22:13:00,651 - root - INFO - Creation time within threshold, continuing 
      2020-09-28 22:13:00,671 - root - INFO - Exiting Script, success. Output: {"location": "2020-09-28T22_12_56.52662
      3889Z", "duration_seconds": "4.082307783", "avg_data_transfer_rate_bytes_sec": 796108, "total_items": 7303, "tot
      al_items_size_bytes": 3250585, "buckets": {"beer-sample": {"mutations_backedup": "7303", "mutations_failed": "0"
      , "deletions_backedup": "0", "deletions_failed": "0"}}} 
      Directory '/data/scriptlogs' created successfully 
      Directory '/data/scriptlogs/incremental' created successfully 
      Directory '/data/scriptlogs/full_only' created successfully 
      Directory '/data/scriptlogs/restore' created successfully
       
      

      3) However, describing the backup resource doesn't show the repo name:

      • needs to note here that this seems to happen when the backup resource config is updated, such as changing the schedule, which caused the repo info to not be shown.

        kubectl describe couchbasebackups my-backup                                               
        Name:         my-backup 
        Namespace:    default 
        Labels:       cluster=cb-backup 
        Annotations:  <none> 
        API Version:  couchbase.com/v2 
        Kind:         CouchbaseBackup 
        Metadata: 
          Creation Timestamp:  2020-09-28T22:06:32Z 
          Generation:          5 
          Managed Fields: 
            API Version:  couchbase.com/v2 
            Fields Type:  FieldsV1 
            fieldsV1: 
              f:metadata: 
                f:labels: 
                  .: 
                  f:cluster: 
              f:spec: 
                .: 
                f:backOffLimit: 
                f:backupRetention: 
                f:failedJobsHistoryLimit: 
                f:full: 
                  .: 
                  f:schedule: 
                f:incremental: 
                  .: 
                  f:schedule: 
                f:logRetention: 
                f:size: 
                f:strategy: 
                f:successfulJobsHistoryLimit: 
            Manager:         kubectl 
            Operation:       Update 
            Time:            2020-09-28T22:13:48Z 
          Resource Version:  24795 
          Self Link:         /apis/couchbase.com/v2/namespaces/default/couchbasebackups/my-backup 
          UID:               af185437-df7d-478e-8c69-0b06739c7487 
        Spec: 
          Back Off Limit:             4 
          Backoff Limit:              2 
          Backup Retention:           24h 
          Failed Jobs History Limit:  10 
          Full: 
            Schedule:  */5 3 * * * 
          Incremental: 
            Schedule:                     */5 2 * * * 
          Log Retention:                  24h 
          Size:                           5Gi 
          Strategy:                       full_incremental 
          Successful Jobs History Limit:  10 
        Events: 
          Type    Reason           Age   From  Message 
          ----    ------           ----  ----  ------- 
          Normal  BackupStarted    10m         Backup `my-backup` started 
          Normal  BackupCompleted  10m         Backup `my-backup` completed

        4) We see the following error when creates a restore resource (restore.yaml)

        {"level":"error","ts":1601331434.5607915,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"no corresponding CouchbaseBackup Repo found"

        5) Running an incremental-backup after the first backup still doesn't show the repo:

        2020-09-28 22:26:17,056 - root - INFO - epoch: 1601331977 
        2020-09-28 22:26:17,056 - root - INFO - timestamp: 2020-09-28T22_26_17 
        2020-09-28 22:26:17,056 - root - INFO - Namespace(backup_ret='24.00', cacert=None, cluster='cb-example', config=
        'false', end=None, log_ret='24.00', mode='backup', repo=None, start=None, strategy='full_incremental', verbosity
        ='INFO') 
        2020-09-28 22:26:17,056 - root - INFO - start logRetention check 
        2020-09-28 22:26:17,056 - root - INFO - removed 0 logs in /data/scriptlogs/incremental 
        2020-09-28 22:26:17,056 - root - INFO - mode: BACKUP 
        2020-09-28 22:26:17,056 - root - INFO - Strategy: full_incremental 
        2020-09-28 22:26:17,056 - root - INFO - Perform INCREMENTAL BACKUP 
        mkdir: cannot create directory ‘/data/backups’: File exists 
        rm: cannot remove '/data/backups/lock.lk': No such file or directory 
        2020-09-28 22:26:17,060 - root - INFO - backup archive /data/backups already exists 
        2020-09-28 22:26:17,060 - root - WARNING - an incremental job was first run. 
        2020-09-28 22:26:17,060 - root - WARNING - please make sure you alter your schedules so a full backup is perform
        ed first 
        2020-09-28 22:26:17,060 - root - INFO - attempting to query K8S objects 
        2020-09-28 22:26:17,061 - root - INFO - k8s config loaded 
        2020-09-28 22:26:17,061 - root - INFO - working in namespace: default 
        2020-09-28 22:26:17,103 - root - INFO - BACKUP start 
        2020-09-28 22:26:17,103 - root - INFO - backing up to: cb-example-2020-09-28T22_12_56 
        2020-09-28 22:26:21,130 - root - INFO - Starting Backup cleanup 
        2020-09-28 22:26:21,130 - root - INFO - Time 2020-09-28 22:26:21.130923, Retention 1 day, 0:00:00, Threshold 202
        0-09-27 22:26:21.130923 
        2020-09-28 22:26:21,131 - root - INFO - Considering repository cb-example-2020-09-28T22_12_56 ... 
        2020-09-28 22:26:21,131 - root - INFO - Creation time within threshold, continuing 
        2020-09-28 22:26:21,166 - root - INFO - Exiting Script, success. Output: {"location": "2020-09-28T22_26_17.15325
        6303Z", "duration_seconds": "3.939892963", "avg_data_transfer_rate_bytes_sec": 7280, "total_items": 0, "total_it
        ems_size_bytes": 28672, "buckets": {"beer-sample": {"mutations_backedup": "0", "mutations_failed": "0", "deletio
        ns_backedup": "0", "deletions_failed": "0"}}}

        It seems during the time the backup runs, the backup resource would show the repo names, but once the backup is finished the backup resource doesn't show.

       

      cbopinfo is attached

      Attachments

        1. backup.yaml
          0.4 kB
        2. cbopinfo-20200925T132537+0000.tar.gz
          2.45 MB
        3. cbopinfo-20200928T155936-0700.tar.gz
          775 kB
        4. cbopinfo-20201001T132628-0700.tar.gz
          790 kB
        5. cbopinfo-20201001T132628-0700.tar.gz
          790 kB
        6. describerestore.out
          3 kB
        7. restore.yaml
          0.2 kB
        8. restorelogs.out
          9 kB
        9. restorelogs.out
          9 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          tin.tran Tin Tran added a comment -

          Hi Daniel Ma

          In addition, for this case, the customer is also see the that when the restore ran, it picked up a repo value named `repo`, even though they do not specify the repo field.

           

          status:
           archive: /data/backups
           backups:
           - full: 2020-09-25T12_04_12.97667236Z
           incrementals:
           - 2020-09-25T12_08_14.641631229Z
           name: onenode-cb-2020-09-25T12_04_12
           duration: 0s
           failed: true
           job: restore-onenode-cb
           lastFailure: "2020-09-25T13:03:54Z"
           lastRun: "2020-09-25T13:03:53Z"
           output: requested restore repo does not exist
           pod: restore-onenode-cb-rtm4q
           repo: repo
           running: false

           

          The customers logs are here:

          cbopinfo-20200925T132537+0000.tar.gz

          However I could not reproduce this repo : repo as my restore pod never got created due to the missing repo field.

           

          tin.tran Tin Tran added a comment - Hi Daniel Ma In addition, for this case, the customer is also see the that when the restore ran, it picked up a repo value named `repo`, even though they do not specify the repo field.   status: archive: /data/backups backups: - full: 2020 - 09 -25T12_04_12.97667236Z incrementals: - 2020 - 09 -25T12_08_14.641631229Z name: onenode-cb- 2020 - 09 -25T12_04_12 duration: 0s failed: true job: restore-onenode-cb lastFailure: "2020-09-25T13:03:54Z" lastRun: "2020-09-25T13:03:53Z" output: requested restore repo does not exist pod: restore-onenode-cb-rtm4q repo: repo running: false   The customers logs are here: cbopinfo-20200925T132537+0000.tar.gz However I could not reproduce this repo : repo as my restore pod never got created due to the missing repo field.  
          tin.tran Tin Tran added a comment - - edited

          Hi Daniel Ma

          After some more testing, I think I know why you could not reproduce the error above.

          1) Initially, after the first backup run, everything looks good, and describing the backup resource would show the repo information

          Status: 
            Archive:  /data/backups 
            Backups: 
              Full:         2020-10-01T20_45_40.470125018Z 
              Name:         cb-example-2020-10-01T20_45_40 
            Capacity Used:  300.15Gi 
            Cronjob:        my-backup-full 
            Duration:       4s 
            Job:            my-backup-full-1601585100 
            Last Run:       2020-10-01T20:45:40Z 
            Last Success:   2020-10-01T20:45:44Z 
            Output:         {"location": "2020-10-01T20_45_40.470125018Z", "duration_seconds": "3.978735214", "avg_data_transfer_rate_bytes_sec": 7208, "total_items": 0, "t
          otal_items_size_bytes": 28672, "buckets": {"beer-sample": {"mutations_backedup": "0", "mutations_failed": "0", "deletions_backedup": "0", "deletions_failed": "0"}
          }} 
            Pod:            my-backup-full-1601585100-n5bqt 
            Repo:           cb-example-2020-10-01T20_45_40 
            Running:        false 
          Events: 
            Type    Reason           Age   From  Message 
            ----    ------           ----  ----  ------- 
            Normal  BackupStarted    84s         Backup `my-backup` started 
            Normal  BackupCompleted  80s         Backup `my-backup` completed

          2) After that, if I updated the backup resource config with a different schedule for the cronjob. <- This action somehow caused the backup resource to not show the repo information anymore, and describing the resource also doesn't show the repo, until the next backup run. Is this an expected behavior? This is with Kubernetes v1.1.5.11.

          d/o/2.0.2 kcb couchbasebackups                                                                                           
          Name:         my-backup 
          Namespace:    default 
          Labels:       cluster=cb-backup 
          Annotations:  <none> 
          API Version:  couchbase.com/v2 
          Kind:         CouchbaseBackup 
          Metadata: 
            Creation Timestamp:  2020-10-01T21:06:18Z 
            Generation:          14 
            Resource Version:    2526 
            Self Link:           /apis/couchbase.com/v2/namespaces/default/couchbasebackups/my-backup 
            UID:                 09b20617-0a08-41a9-9eac-8b7350e3d7b5 
          Spec: 
            Back Off Limit:             4 
            Backoff Limit:              2 
            Backup Retention:           24h 
            Failed Jobs History Limit:  10 
            Full: 
              Schedule:  */5 * * * * 
            Incremental: 
              Schedule:                     */5 5 * * * 
            Log Retention:                  24h 
            Size:                           5Gi 
            Strategy:                       full_incremental 
            Successful Jobs History Limit:  10 
          Status: 
            Archive:  /data/backups 
            Backups: 
              Full:         2020-10-01T21_10_45.018294556Z 
              Name:         cb-example-2020-10-01T21_10_44 
            Capacity Used:  300.38Gi 
            Cronjob:        my-backup-full 
            Duration:       0s 
            Job:            my-backup-full-1601586600 
            Last Run:       2020-10-01T21:10:44Z 
            Last Success:   2020-10-01T21:10:45Z 
            Output:         {"location": "2020-10-01T21_10_45.018294556Z", "duration_seconds": "3.350242m", "avg_data_transfer_rate_bytes_sec": 0
          , "total_items": 0, "total_items_size_bytes": 0, "buckets": {}} 
            Pod:            my-backup-full-1601586600-4s8d4 
            Repo:           cb-example-2020-10-01T21_10_44 
            Running:        false 
          Events: 
            Type    Reason           Age   From  Message 
            ----    ------           ----  ----  ------- 
            Normal  BackupStarted    22s         Backup `my-backup` started 
            Normal  BackupCompleted  21s         Backup `my-backup` completed 
           
           
           
          ⋊> ~/d/o/2.0.2 kcr backup.yaml 
                                                                                                          
          couchbasebackup.couchbase.com/my-backup replaced 
           
           
          ⋊> ~/d/o/2.0.2 kcb couchbasebackups                                                                                           
          Name:         my-backup 
          Namespace:    default 
          Labels:       cluster=cb-backup 
          Annotations:  <none> 
          API Version:  couchbase.com/v2 
          Kind:         CouchbaseBackup 
          Metadata: 
            Creation Timestamp:  2020-10-01T21:06:18Z 
            Generation:          15 
            Resource Version:    2596 
            Self Link:           /apis/couchbase.com/v2/namespaces/default/couchbasebackups/my-backup 
            UID:                 09b20617-0a08-41a9-9eac-8b7350e3d7b5 
          Spec: 
            Back Off Limit:             4 
            Backoff Limit:              2 
            Backup Retention:           24h 
            Failed Jobs History Limit:  10 
            Full: 
              Schedule:  */5 5 * * * 
            Incremental: 
              Schedule:                     */5 5 * * * 
            Log Retention:                  24h 
            Size:                           5Gi 
            Strategy:                       full_incremental 
            Successful Jobs History Limit:  10 
          Events: 
            Type    Reason           Age   From  Message 
            ----    ------           ----  ----  ------- 
            Normal  BackupStarted    32s         Backup `my-backup` started 
            Normal  BackupCompleted  31s         Backup `my-backup` completed

           

          3) During this time, if the restore is created, then the following would be show in the operator logs constantly:

           

          {"level":"error","ts":1601331434.5607915,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"no corresponding CouchbaseBackup Repo found"

          And the restore pod would not be created.

          4) If the repo information is available, then the restore pod is created, however, it seems to pick up `repo` as the value for the --repo field. We can see the following from the restore logs:

          2020-10-01 21:23:36,720 - root - INFO - epoch: 1601587416 
          2020-10-01 21:23:36,720 - root - INFO - timestamp: 2020-10-01T21_23_36 
          2020-10-01 21:23:36,720 - root - INFO - Namespace(backup_ret=None, cacert=None, cluster='cb-example', config=False, end='oldest', log_r
          et='24.00', mode='restore', repo='repo', start='oldest', strategy=None, verbosity='INFO') 
          2020-10-01 21:23:36,720 - root - INFO - start logRetention check 
          2020-10-01 21:23:36,720 - root - INFO - removed 0 logs in /data/scriptlogs/restore 
          2020-10-01 21:23:36,720 - root - INFO - mode: RESTORE 
          mkdir: cannot create directory ‘/data/backups’: File exists 
          rm: cannot remove '/data/backups/lock.lk': No such file or directory 
          2020-10-01 21:23:36,726 - root - INFO - attempting to query K8S objects 
          NoneType: None 
          2020-10-01 21:23:36,727 - root - INFO - k8s config loaded 
          2020-10-01 21:23:36,727 - root - INFO - working in namespace: default 
          2020-10-01 21:23:36,753 - root - INFO - running on POD[my-restore-hkmv5] for JOB[my-restore] for RESTORE[my-restore] 
          2020-10-01 21:23:36,753 - root - INFO - current CouchbaseBackupRestore object: my-restore 
          2020-10-01 21:23:36,844 - root - INFO - rfoldersepos  
          2020-10-01 21:23:36,844 - root - INFO - ['/data/backups/cb-example-2020-10-01T21_10_44/2020-10-01T21_13_09.224780719Z/', '/data/backups
          /cb-example-2020-10-01T21_10_44/2020-10-01T21_21_09.953950164Z/', '/data/backups/cb-example-2020-10-01T21_10_44/2020-10-01T21_14_19.361
          379248Z/', '/data/backups/cb-example-2020-10-01T21_10_44/2020-10-01T21_10_45.018294556Z/'] 
          2020-10-01 21:23:36,931 - root - INFO - RESTORE start 
          2020-10-01 21:23:36,931 - root - INFO - restoring from: repo 
          2020-10-01 21:23:36,931 - root - ERROR - requested restore repo does not exist 
          2020-10-01 21:23:36,983 - root - INFO - rfoldersepos  
          2020-10-01 21:23:36,983 - root - INFO - ['/data/backups/cb-example-2020-10-01T21_10_44/2020-10-01T21_13_09.224780719Z/', '/data/backups
          /cb-example-2020-10-01T21_10_44/2020-10-01T21_21_09.953950164Z/', '/data/backups/cb-example-2020-10-01T21_10_44/2020-10-01T21_14_19.361
          379248Z/', '/data/backups/cb-example-2020-10-01T21_10_44/2020-10-01T21_10_45.018294556Z/'] 
          2020-10-01 21:23:37,026 - root - ERROR - Exiting Script. Failed with output: requested restore repo does not exist 
          (404) 
          Reason: Not Found 
          HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Thu, 01 Oct 2020 21:23:37 GMT', 'Content-Length': '
          250'}) 
          HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"couchbasebackups.couchbase.com \"my-
          restore\" not found","reason":"NotFound","details":{"name":"my-restore","group":"couchbase.com","kind":"couchbasebackups"},"code":404}

           Describing the restore pod would show us this:

           

          Args: 
                cb-example 
                --mode 
                restore 
                --repo 
                repo 
                --start 
                oldest 
                --end 
                oldest 
                --log-ret
          

           

          The `repo:repo` occurs with this image: image: couchbase/operator-backup:6.5.0

          I tested and the same behavior when updating the backup config is also observed with 

             image: couchbase/operator-backup:6.5.1-111

           

          attached are my logs:

          describerestore.outrestorelogs.outcbopinfo-20201001T132628-0700.tar.gz

           

          Thank you Daniel.

          tin.tran Tin Tran added a comment - - edited Hi  Daniel Ma After some more testing, I think I know why you could not reproduce the error above. 1) Initially, after the first backup run, everything looks good, and describing the backup resource would show the repo information Status:  Archive:  /data/backups  Backups:    Full:          2020 - 10 -01T20_45_40.470125018Z    Name:         cb-example- 2020 - 10 -01T20_45_40  Capacity Used:   300 .15Gi  Cronjob:        my-backup-full  Duration:       4s  Job:            my-backup-full- 1601585100  Last Run:        2020 - 10 -01T20: 45 :40Z  Last Success:    2020 - 10 -01T20: 45 :44Z  Output:         { "location" : "2020-10-01T20_45_40.470125018Z" , "duration_seconds" : "3.978735214" , "avg_data_transfer_rate_bytes_sec" : 7208 , "total_items" : 0 , "t otal_items_size_bytes ": 28672, " buckets ": {" beer-sample ": {" mutations_backedup ": " 0 ", " mutations_failed ": " 0 ", " deletions_backedup ": " 0 ", " deletions_failed ": " 0 "} }}  Pod:            my-backup-full- 1601585100 -n5bqt  Repo:           cb-example- 2020 - 10 -01T20_45_40  Running:         false Events:  Type    Reason           Age   From  Message  ----    ------           ----  ----  -------  Normal  BackupStarted    84s         Backup `my-backup` started  Normal  BackupCompleted  80s         Backup `my-backup` completed 2)  After that, if I updated the backup resource config with a different schedule for the cronjob . <- This action somehow caused the backup resource to not show the repo information anymore, and describing the resource also doesn't show the repo, until the next backup run. Is this an expected behavior? This is with Kubernetes v1.1.5.11. d/o/ 2.0 . 2 kcb couchbasebackups                                                                                            Name:         my-backup Namespace:     default Labels:       cluster=cb-backup Annotations:  <none> API Version:  couchbase.com/v2 Kind:         CouchbaseBackup Metadata:  Creation Timestamp:   2020 - 10 -01T21: 06 :18Z  Generation:           14  Resource Version:     2526  Self Link:           /apis/couchbase.com/v2/namespaces/ default /couchbasebackups/my-backup  UID:                 09b20617-0a08-41a9-9eac-8b7350e3d7b5 Spec:  Back Off Limit:              4  Backoff Limit:               2  Backup Retention:           24h  Failed Jobs History Limit:   10  Full:    Schedule:  */ 5 * * * *  Incremental:    Schedule:                     */ 5 5 * * *  Log Retention:                  24h  Size:                           5Gi  Strategy:                       full_incremental  Successful Jobs History Limit:   10 Status:  Archive:  /data/backups  Backups:    Full:          2020 - 10 -01T21_10_45.018294556Z    Name:         cb-example- 2020 - 10 -01T21_10_44  Capacity Used:   300 .38Gi  Cronjob:        my-backup-full  Duration:       0s  Job:            my-backup-full- 1601586600  Last Run:        2020 - 10 -01T21: 10 :44Z  Last Success:    2020 - 10 -01T21: 10 :45Z  Output:         { "location" : "2020-10-01T21_10_45.018294556Z" , "duration_seconds" : "3.350242m" , "avg_data_transfer_rate_bytes_sec" : 0 , "total_items" : 0 , "total_items_size_bytes" : 0 , "buckets" : {}}  Pod:            my-backup-full- 1601586600 -4s8d4  Repo:           cb-example- 2020 - 10 -01T21_10_44  Running:         false Events:  Type    Reason           Age   From  Message  ----    ------           ----  ----  -------  Normal  BackupStarted    22s         Backup `my-backup` started  Normal  BackupCompleted  21s         Backup `my-backup` completed       ⋊> ~/d/o/ 2.0 . 2 kcr backup.yaml                                                                                                  couchbasebackup.couchbase.com/my-backup replaced     ⋊> ~/d/o/ 2.0 . 2 kcb couchbasebackups                                                                                            Name:         my-backup Namespace:     default Labels:       cluster=cb-backup Annotations:  <none> API Version:  couchbase.com/v2 Kind:         CouchbaseBackup Metadata:  Creation Timestamp:   2020 - 10 -01T21: 06 :18Z  Generation:           15  Resource Version:     2596  Self Link:           /apis/couchbase.com/v2/namespaces/ default /couchbasebackups/my-backup  UID:                 09b20617-0a08-41a9-9eac-8b7350e3d7b5 Spec:  Back Off Limit:              4  Backoff Limit:               2  Backup Retention:           24h  Failed Jobs History Limit:   10  Full:    Schedule:  */ 5 5 * * *  Incremental:    Schedule:                     */ 5 5 * * *  Log Retention:                  24h  Size:                           5Gi  Strategy:                       full_incremental  Successful Jobs History Limit:   10 Events:  Type    Reason           Age   From  Message  ----    ------           ----  ----  -------  Normal  BackupStarted    32s         Backup `my-backup` started  Normal  BackupCompleted  31s         Backup `my-backup` completed   3) During this time, if the restore is created, then the following would be show in the operator logs constantly:   {"level":"error","ts":1601331434.5607915,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"no corresponding CouchbaseBackup Repo found" And the restore pod would not be created. 4) If the repo information is available, then the restore pod is created, however, it seems to pick up `repo` as the value for the --repo field. We can see the following from the restore logs: 2020 - 10 - 01 21 : 23 : 36 , 720 - root - INFO - epoch: 1601587416 2020 - 10 - 01 21 : 23 : 36 , 720 - root - INFO - timestamp: 2020 - 10 -01T21_23_36 2020 - 10 - 01 21 : 23 : 36 , 720 - root - INFO - Namespace(backup_ret=None, cacert=None, cluster= 'cb-example' , config=False, end= 'oldest' , log_r et= '24.00' , mode= 'restore' , repo= 'repo' , start= 'oldest' , strategy=None, verbosity= 'INFO' ) 2020 - 10 - 01 21 : 23 : 36 , 720 - root - INFO - start logRetention check 2020 - 10 - 01 21 : 23 : 36 , 720 - root - INFO - removed 0 logs in /data/scriptlogs/restore 2020 - 10 - 01 21 : 23 : 36 , 720 - root - INFO - mode: RESTORE mkdir: cannot create directory ‘/data/backups’: File exists rm: cannot remove '/data/backups/lock.lk' : No such file or directory 2020 - 10 - 01 21 : 23 : 36 , 726 - root - INFO - attempting to query K8S objects NoneType: None 2020 - 10 - 01 21 : 23 : 36 , 727 - root - INFO - k8s config loaded 2020 - 10 - 01 21 : 23 : 36 , 727 - root - INFO - working in namespace: default 2020 - 10 - 01 21 : 23 : 36 , 753 - root - INFO - running on POD[my-restore-hkmv5] for JOB[my-restore] for RESTORE[my-restore] 2020 - 10 - 01 21 : 23 : 36 , 753 - root - INFO - current CouchbaseBackupRestore object: my-restore 2020 - 10 - 01 21 : 23 : 36 , 844 - root - INFO - rfoldersepos   2020 - 10 - 01 21 : 23 : 36 , 844 - root - INFO - [ '/data/backups/cb-example-2020-10-01T21_10_44/2020-10-01T21_13_09.224780719Z/' , '/data/backups /cb-example- 2020 - 10 -01T21_10_44/ 2020 - 10 -01T21_21_09.953950164Z/ ', ' /data/backups/cb-example- 2020 - 10 -01T21_10_44/ 2020 - 10 -01T21_14_19. 361 379248Z/ ', ' /data/backups/cb-example- 2020 - 10 -01T21_10_44/ 2020 - 10 -01T21_10_45.018294556Z/'] 2020 - 10 - 01 21 : 23 : 36 , 931 - root - INFO - RESTORE start 2020 - 10 - 01 21 : 23 : 36 , 931 - root - INFO - restoring from: repo 2020 - 10 - 01 21 : 23 : 36 , 931 - root - ERROR - requested restore repo does not exist 2020 - 10 - 01 21 : 23 : 36 , 983 - root - INFO - rfoldersepos   2020 - 10 - 01 21 : 23 : 36 , 983 - root - INFO - [ '/data/backups/cb-example-2020-10-01T21_10_44/2020-10-01T21_13_09.224780719Z/' , '/data/backups /cb-example- 2020 - 10 -01T21_10_44/ 2020 - 10 -01T21_21_09.953950164Z/ ', ' /data/backups/cb-example- 2020 - 10 -01T21_10_44/ 2020 - 10 -01T21_14_19. 361 379248Z/ ', ' /data/backups/cb-example- 2020 - 10 -01T21_10_44/ 2020 - 10 -01T21_10_45.018294556Z/'] 2020 - 10 - 01 21 : 23 : 37 , 026 - root - ERROR - Exiting Script. Failed with output: requested restore repo does not exist ( 404 ) Reason: Not Found HTTP response headers: HTTPHeaderDict({ 'Content-Type' : 'application/json' , 'Date' : 'Thu, 01 Oct 2020 21:23:37 GMT' , 'Content-Length' : ' 250 '}) HTTP response body: { "kind" : "Status" , "apiVersion" : "v1" , "metadata" :{}, "status" : "Failure" , "message" :"couchbasebackups.couchbase.com \"my- restore\ " not found" , "reason" : "NotFound" , "details" :{ "name" : "my-restore" , "group" : "couchbase.com" , "kind" : "couchbasebackups" }, "code" : 404 }  Describing the restore pod would show us this:   Args:      cb-example      --mode      restore      --repo      repo      --start      oldest      --end      oldest      --log-ret   The `repo:repo` occurs with this image:  image: couchbase/operator-backup:6.5.0 I tested and the same behavior when updating the backup config is also observed with     image: couchbase/operator-backup:6.5.1-111   attached are my logs: describerestore.out restorelogs.out cbopinfo-20201001T132628-0700.tar.gz   Thank you Daniel.
          daniel.ma Daniel Ma (Inactive) added a comment - re: https://issues.couchbase.com/browse/K8S-1690?focusedCommentId=440092&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-440092 This issue is known and is not a critical bug, "repo" is just the default value for repo

          People

            daniel.ma Daniel Ma (Inactive)
            tin.tran Tin Tran
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty