Image 1.1.1 Incremental backup periodically fails with ERROR Stdout: b'Error backing up cluster: directory not empty\n'

2021-11-18T21:00:05 INFO Backing up to repository: cb-test-2021-11-18T21_00_04

2021-11-18T21:00:05 ERROR Subprocess call exited with non-zero return code 1

2021-11-18T21:00:05 ERROR Arguments: cbbackupmgr backup --archive /data/backups --repo cb-test-2021-11-18T21_00_04 --cluster http://cb-test-srv:8091 --username Administrator --password password --purge --no-progress-bar --threads 1

2021-11-18T21:00:05 ERROR Stdout: b'Error backing up cluster: directory not empty\n'

2021-11-18T21:00:05 ERROR Stderr: b''

2021-11-18T21:00:05 ERROR cbbackupmgr error during backup actions

2021-11-18T21:00:05 ERROR Command '['cbbackupmgr', 'backup', '--archive', '/data/backups', '--repo', 'cb-test-2021-11-18T21_00_04', '--cluster', 'http://cb-test-srv:8091', '--username', 'Administrator', '--password', 'password', '--purge', '--no-progress-bar', '--threads', '1']' returned non-zero exit status 1.

kcl my-backup-incremental-1637269200-fn8sz

2021-11-18T21:00:04 INFO couchbase-operator-backup/1.1.1 (commit/38e3ef0bda2977443574b576f47d7cd213bdbeca)

2021-11-18T21:00:04 INFO Timestamp: 2021-11-18 21:00:04.831859

2021-11-18T21:00:04 INFO Arguments: cluster=cb-test, mode=backup, strategy=full_incremental, config=False, backup_ret=24.0, repo=None, start=None, end=None, include_buckets=None, exclude_buckets=None, map_buckets=None, enable_bucket_config=False, disable_views=False, disable_gsi_indexes=False, disable_ft_indexes=False, disable_ft_alias=False, disable_data=False, disable_analytics=False, disable_eventing=False, cacert=None, log_ret=24.0, verbosity=INFO, s3_bucket=None, s3_region=None, threads=1, full=False, incremental=True, include_data=None, exclude_data=None, map_data=None

2021-11-18T21:00:04 INFO Checking connection to Kubernetes API...

2021-11-18T21:00:04 ERROR Failed to update resource status: (409)

Reason: Conflict

HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '19273227-8f82-424d-8b41-08702ce96bfb', 'X-Kubernetes-Pf-Prioritylevel-Uid': '0e2c8d46-482d-486a-bb89-18d6457bef7f', 'Date': 'Thu, 18 Nov 2021 21:00:04 GMT', 'Content-Length': '364'})

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on couchbasebackups.couchbase.com \"my-backup\": the object has been modified; please apply your changes to the latest version and try again","reason":"Conflict","details":{"name":"my-backup","group":"couchbase.com","kind":"couchbasebackups"},"code":409}

2021-11-18T21:00:05 INFO Couchbase backup manager output: b'{"name":"backups","archive_uuid":"64b78ef3-ff09-42fd-82e1-bb7779ad3c6c","repos":[{"name":"cb-test-2021-11-18T20_30_13","size":46462185,"count":1,"backups":[{"date":"2021-11-18T20_30_14.094948387Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":46462185,"buckets":[{"name":"gamesim-sample","size":46462108,"items":586,"mutations":586,"tombstones":0,"views_count":1,"fts_count":0,"index_count":1,"analytics_count":0,"query_udfs":0}],"complete":true,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]},{"name":"cb-test-2021-11-18T20_45_01","size":46462185,"count":1,"backups":[{"date":"2021-11-18T20_45_04.969936012Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":46462185,"buckets":[{"name":"gamesim-sample","size":46462108,"items":586,"mutations":586,"tombstones":0,"views_count":1,"fts_count":0,"index_count":1,"analytics_count":0,"query_udfs":0}],"complete":true,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]},{"name":"cb-test-2021-11-18T21_00_04","size":77,"count":1,"backups":[{"date":"2021-11-18T21_00_04.963046876Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":77,"buckets":[],"complete":false,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]}],"size":92924447}\n'

2021-11-18T21:00:05 INFO Incremental backup using existing repository: cb-test-2021-11-18T21_00_04

2021-11-18T21:00:05 INFO Starting backup

2021-11-18T21:00:05 INFO Couchbase backup manager output: b'{"name":"backups","archive_uuid":"64b78ef3-ff09-42fd-82e1-bb7779ad3c6c","repos":[{"name":"cb-test-2021-11-18T20_30_13","size":46462185,"count":1,"backups":[{"date":"2021-11-18T20_30_14.094948387Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":46462185,"buckets":[{"name":"gamesim-sample","size":46462108,"items":586,"mutations":586,"tombstones":0,"views_count":1,"fts_count":0,"index_count":1,"analytics_count":0,"query_udfs":0}],"complete":true,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]},{"name":"cb-test-2021-11-18T20_45_01","size":46462185,"count":1,"backups":[{"date":"2021-11-18T20_45_04.969936012Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":46462185,"buckets":[{"name":"gamesim-sample","size":46462108,"items":586,"mutations":586,"tombstones":0,"views_count":1,"fts_count":0,"index_count":1,"analytics_count":0,"query_udfs":0}],"complete":true,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]},{"name":"cb-test-2021-11-18T21_00_04","size":77,"count":1,"backups":[{"date":"2021-11-18T21_00_04.963046876Z","type":"FULL","source":"http://cb-test-srv:8091","events":0,"fts_alias":0,"analytics_count":0,"query_udfs":0,"size":77,"buckets":[],"complete":false,"source_cluster_uuid":"f4aaca015e424e613f83ccf47afe3529"}]}],"size":92924447}\n'

2021-11-18T21:00:05 INFO Backing up to repository: cb-test-2021-11-18T21_00_04

2021-11-18T21:00:05 ERROR Subprocess call exited with non-zero return code 1

2021-11-18T21:00:05 ERROR Arguments: cbbackupmgr backup --archive /data/backups --repo cb-test-2021-11-18T21_00_04 --cluster http://cb-test-srv:8091 --username Administrator --password password --purge --no-progress-bar --threads 1

2021-11-18T21:00:05 ERROR Stdout: b'Error backing up cluster: directory not empty\n'

2021-11-18T21:00:05 ERROR Stderr: b''

2021-11-18T21:00:05 ERROR cbbackupmgr error during backup actions

2021-11-18T21:00:05 ERROR Command '['cbbackupmgr', 'backup', '--archive', '/data/backups', '--repo', 'cb-test-2021-11-18T21_00_04', '--cluster', 'http://cb-test-srv:8091', '--username', 'Administrator', '--password', 'password', '--purge', '--no-progress-bar', '--threads', '1']' returned non-zero exit status 1.