[Backup] Failure trying to remove stale lockfile on S3 bucket.

2020-11-23 09:19:14,522 - root - ERROR - cbbackupmgr error during config action

2020-11-23 09:19:14,522 - root - ERROR - error code: 1, b"Backup repository creation failed: failure trying to remove stale lockfile: the process '54' running on 'full-incremental-full-1606123080-hl9rv' already holds the lock\n"

2020-11-23 09:19:20,393 - root - INFO - raising backup fail event

(403)

Reason: Forbidden

HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c63addd4-b189-45d2-8482-813dbb29fd09', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 23 Nov 2020 09:19:20 GMT', 'Content-Length': '411'})

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"couchbasebackups.couchbase.com \"cbbackup\" is forbidden: User \"system:serviceaccount:test-vdc5f:couchbase-backup\" cannot get resource \"couchbasebackups\" in API group \"couchbase.com\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"cbbackup","group":"couchbase.com","kind":"couchbasebackups"},"code":403}

Traceback (most recent call last):

  File "/opt/couchbase/bin/backup_script", line 1220, in config_repo

    result = cbbackupmgr(args_list)

  File "/opt/couchbase/bin/backup_script", line 80, in cbbackupmgr

    return str(subprocess.check_output(["cbbackupmgr"] + cmd).decode("utf-8")).strip()

  File "/opt/couchbase/lib/python/runtime/lib/python3.7/subprocess.py", line 395, in check_output

    **kwargs).stdout

  File "/opt/couchbase/lib/python/runtime/lib/python3.7/subprocess.py", line 487, in run

    output=stdout, stderr=stderr)

subprocess.CalledProcessError: Command '['cbbackupmgr', 'config', '--archive', 's3://caobackups3/archive', '--repo', 'test-couchbase-xndsq-2020-11-23T09_19_13', '--obj-staging-dir', '/data/staging', '--obj-region', 'us-west-2', '--obj-access-key-id', 'AKIAXQQ2DIGAWSCXJO7I', '--obj-secret-access-key', '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']' returned non-zero exit status 1.During handling of the above exception, another exception occurred:Traceback (most recent call last):

  File "/opt/couchbase/bin/backup_script", line 1234, in <module>

    main()

  File "/opt/couchbase/bin/backup_script", line 153, in main

    config_repo(args, archive, timestamp)

  File "/opt/couchbase/bin/backup_script", line 1230, in config_repo

    % grepexc.output))

  File "/opt/couchbase/bin/backup_script", line 707, in exit_script

    raise_failed_event(get_obj_ref())

  File "/opt/couchbase/bin/backup_script", line 808, in get_obj_ref

    cbbackup = get_backup(BACKUP)

  File "/opt/couchbase/bin/backup_script", line 317, in get_backup

    NAMESPACE, CBBACKUP_PLURAL, name)

  File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/api/custom_objects_api.py", line 954, in get_namespaced_custom_object

    (data) = self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs)  # noqa: E501

  File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/api/custom_objects_api.py", line 1057, in get_namespaced_custom_object_with_http_info

    collection_formats=collection_formats)

  File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 345, in call_api

    _preload_content, _request_timeout)

  File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 176, in __call_api

    _request_timeout=_request_timeout)

  File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 366, in request

    headers=headers)

  File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/rest.py", line 241, in GET

    query_params=query_params)

  File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in request

    raise ApiException(http_resp=r)

kubernetes.client.rest.ApiException: (403)

Reason: Forbidden

HTTP response headers: HTTPHeaderDict({'Audit-Id': '93f140b6-94a0-4aeb-bbb4-62d157a2f136', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 23 Nov 2020 09:19:20 GMT', 'Content-Length': '411'})

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"couchbasebackups.couchbase.com \"cbbackup\" is forbidden: User \"system:serviceaccount:test-vdc5f:couchbase-backup\" cannot get resource \"couchbasebackups\" in API group \"couchbase.com\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"cbbackup","group":"couchbase.com","kind":"couchbasebackups"},"code":403}