Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2540

Image 1.1.1 Incremental backup periodically fails with ERROR Stdout: b'Error backing up cluster: directory not empty\n'

    XMLWordPrintable

Details

    • Bug
    • Status: Reopened
    • Major
    • Resolution: Unresolved
    • None
    • not-targeted
    • kubernetes
    • None
    • 1

    Description

      The incremental backup pod would fail with the following error on the first run of the schedule

      2021-11-18T21:00:05 INFO Backing up to repository: cb-test-2021-11-18T21_00_04
      2021-11-18T21:00:05 ERROR Subprocess call exited with non-zero return code 1
      2021-11-18T21:00:05 ERROR Arguments: cbbackupmgr backup --archive /data/backups --repo cb-test-2021-11-18T21_00_04 --cluster http://cb-test-srv:8091 --username Administrator --password password --purge --no-progress-bar --threads 1
      2021-11-18T21:00:05 ERROR Stdout: b'Error backing up cluster: directory not empty\n'
      2021-11-18T21:00:05 ERROR Stderr: b''
      2021-11-18T21:00:05 ERROR cbbackupmgr error during backup actions
      2021-11-18T21:00:05 ERROR Command '['cbbackupmgr', 'backup', '--archive', '/data/backups', '--repo', 'cb-test-2021-11-18T21_00_04', '--cluster', 'http://cb-test-srv:8091', '--username', 'Administrator', '--password', 'password', '--purge', '--no-progress-bar', '--threads', '1']' returned non-zero exit status 1.
      

      The run immediately after is successful.

      *Steps to reproduce:

      Setting up backup : backup.yaml
      Server setup: cbtest2.yaml

      On the first run of the schedule incremental run, it would error with this

      kcl my-backup-incremental-1637269200-fn8sz
      2021-11-18T21:00:04 INFO couchbase-operator-backup/1.1.1 (commit/38e3ef0bda2977443574b576f47d7cd213bdbeca)
      2021-11-18T21:00:04 INFO Timestamp: 2021-11-18 21:00:04.831859
      2021-11-18T21:00:04 INFO Arguments: cluster=cb-test, mode=backup, strategy=full_incremental, config=False, backup_ret=24.0, repo=None, start=None, end=None, include_buckets=None, exclude_buckets=None, map_buckets=None, enable_bucket_config=False, disable_views=False, disable_gsi_indexes=False, disable_ft_indexes=False, disable_ft_alias=False, disable_data=False, disable_analytics=False, disable_eventing=False, cacert=None, log_ret=24.0, verbosity=INFO, s3_bucket=None, s3_region=None, threads=1, full=False, incremental=True, include_data=None, exclude_data=None, map_data=None
      2021-11-18T21:00:04 INFO Checking connection to Kubernetes API...
      2021-11-18T21:00:04 ERROR Failed to update resource status: (409)
      Reason: Conflict
      HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '19273227-8f82-424d-8b41-08702ce96bfb', 'X-Kubernetes-Pf-Prioritylevel-Uid': '0e2c8d46-482d-486a-bb89-18d6457bef7f', 'Date': 'Thu, 18 Nov 2021 21:00:04 GMT', 'Content-Length': '364'})
      HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on couchbasebackups.couchbase.com \"my-backup\": the object has been modified; please apply your changes to the latest version and try again","reason":"Conflict","details":{"name":"my-backup","group":"couchbase.com","kind":"couchbasebackups"},"code":409}
       
       
      2021-11-18T21:00:05 INFO Couchbase backup manager output: b'{"name":"backups","archive_uuid":"64b78ef3-ff09-42fd-82e1-bb7779ad3c6c","repos":[{"name":"cb-test-2021-11-18T20_30_13","size":46462185,"count":1,"backups":[{"date":"2021-11-18T20_30_14.094948387Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":46462185,"buckets":[{"name":"gamesim-sample","size":46462108,"items":586,"mutations":586,"tombstones":0,"views_count":1,"fts_count":0,"index_count":1,"analytics_count":0,"query_udfs":0}],"complete":true,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]},{"name":"cb-test-2021-11-18T20_45_01","size":46462185,"count":1,"backups":[{"date":"2021-11-18T20_45_04.969936012Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":46462185,"buckets":[{"name":"gamesim-sample","size":46462108,"items":586,"mutations":586,"tombstones":0,"views_count":1,"fts_count":0,"index_count":1,"analytics_count":0,"query_udfs":0}],"complete":true,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]},{"name":"cb-test-2021-11-18T21_00_04","size":77,"count":1,"backups":[{"date":"2021-11-18T21_00_04.963046876Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":77,"buckets":[],"complete":false,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]}],"size":92924447}\n'
      2021-11-18T21:00:05 INFO Incremental backup using existing repository: cb-test-2021-11-18T21_00_04
      2021-11-18T21:00:05 INFO Starting backup
      2021-11-18T21:00:05 INFO Couchbase backup manager output: b'{"name":"backups","archive_uuid":"64b78ef3-ff09-42fd-82e1-bb7779ad3c6c","repos":[{"name":"cb-test-2021-11-18T20_30_13","size":46462185,"count":1,"backups":[{"date":"2021-11-18T20_30_14.094948387Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":46462185,"buckets":[{"name":"gamesim-sample","size":46462108,"items":586,"mutations":586,"tombstones":0,"views_count":1,"fts_count":0,"index_count":1,"analytics_count":0,"query_udfs":0}],"complete":true,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]},{"name":"cb-test-2021-11-18T20_45_01","size":46462185,"count":1,"backups":[{"date":"2021-11-18T20_45_04.969936012Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":46462185,"buckets":[{"name":"gamesim-sample","size":46462108,"items":586,"mutations":586,"tombstones":0,"views_count":1,"fts_count":0,"index_count":1,"analytics_count":0,"query_udfs":0}],"complete":true,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]},{"name":"cb-test-2021-11-18T21_00_04","size":77,"count":1,"backups":[{"date":"2021-11-18T21_00_04.963046876Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":77,"buckets":[],"complete":false,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]}],"size":92924447}\n'
      2021-11-18T21:00:05 INFO Backing up to repository: cb-test-2021-11-18T21_00_04
      2021-11-18T21:00:05 ERROR Subprocess call exited with non-zero return code 1
      2021-11-18T21:00:05 ERROR Arguments: cbbackupmgr backup --archive /data/backups --repo cb-test-2021-11-18T21_00_04 --cluster http://cb-test-srv:8091 --username Administrator --password password --purge --no-progress-bar --threads 1
      2021-11-18T21:00:05 ERROR Stdout: b'Error backing up cluster: directory not empty\n'
      2021-11-18T21:00:05 ERROR Stderr: b''
      2021-11-18T21:00:05 ERROR cbbackupmgr error during backup actions
      2021-11-18T21:00:05 ERROR Command '['cbbackupmgr', 'backup', '--archive', '/data/backups', '--repo', 'cb-test-2021-11-18T21_00_04', '--cluster', 'http://cb-test-srv:8091', '--username', 'Administrator', '--password', 'password', '--purge', '--no-progress-bar', '--threads', '1']' returned non-zero exit status 1.
      

      The run right after would succeed.
      No such error for full backup.

      Attachments

        1. backup.yaml
          0.4 kB
        2. cbtest2.yaml
          2 kB
        3. image-2021-11-18-15-09-21-556.png
          image-2021-11-18-15-09-21-556.png
          15 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          simon.murray Simon Murray added a comment -

          I cannot reproduce this using our standard test suite.  Your cron specs look very odd, run a full every 15 minutes and an incremental every 45... if you've got jobs aliasing it's hardly surprising it's going badly wrong.

          simon.murray Simon Murray added a comment - I cannot reproduce this using our standard test suite.  Your cron specs look very odd, run a full every 15 minutes and an incremental every 45... if you've got jobs aliasing it's hardly surprising it's going badly wrong.
          tin.tran Tin Tran added a comment -

          Hi Simon Murray
          I'm reopening this issue as another customer has hit this error.

          They have set the following backup as followed:

          NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
          couchbase-ee-ha-backup-full 0 0 * * * False 0 72m 38d
          couchbase-ee-ha-backup-incremental 0/30 * * * * False 0 12m 9d

          I will create the appropriate CBSE

          tin.tran Tin Tran added a comment - Hi Simon Murray I'm reopening this issue as another customer has hit this error. They have set the following backup as followed: NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE couchbase-ee-ha-backup-full 0 0 * * * False 0 72m 38d couchbase-ee-ha-backup-incremental 0/30 * * * * False 0 12m 9d I will create the appropriate CBSE
          tin.tran Tin Tran added a comment - - edited

          Hi Prateek Kumar, Arunkumar Senthilnathan said to mention you here for help looking at this issue to see if you could reproduce.

          Thank you Prateek, Arun.

          tin.tran Tin Tran added a comment - - edited Hi Prateek Kumar , Arunkumar Senthilnathan said to mention you here for help looking at this issue to see if you could reproduce. Thank you Prateek, Arun.

          Tin Tran, I'll put the mentioned backup cronjob schedule in our test suite and check if we can reproduce the issue.

          prateek.kumar Prateek Kumar added a comment - Tin Tran , I'll put the mentioned backup cronjob schedule in our test suite and check if we can reproduce the issue.

          People

            prateek.kumar Prateek Kumar
            tin.tran Tin Tran
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty