Description
Job: http://qa.sc.couchbase.com/job/roo-test/102/console
Image: dmacouch/operator-backup:6.6.0
couchbase server: couchbase/server:6.6.0 , server upgrade: couchbase/server:6.6.0
S3 Bucket: s3://caobackups3
Tests Failed: TestBackupRetentionS3 , TestUpdateBackupStatusS3
The job runs 4 tests in parallel.
cbbackupmgr Error:
2020-11-23 09:19:14,522 - root - ERROR - cbbackupmgr error during config action |
2020-11-23 09:19:14,522 - root - ERROR - error code: 1, b"Backup repository creation failed: failure trying to remove stale lockfile: the process '54' running on 'full-incremental-full-1606123080-hl9rv' already holds the lock\n" |
2020-11-23 09:19:20,393 - root - INFO - raising backup fail event |
(403) |
Reason: Forbidden
|
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c63addd4-b189-45d2-8482-813dbb29fd09', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 23 Nov 2020 09:19:20 GMT', 'Content-Length': '411'}) |
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"couchbasebackups.couchbase.com \"cbbackup\" is forbidden: User \"system:serviceaccount:test-vdc5f:couchbase-backup\" cannot get resource \"couchbasebackups\" in API group \"couchbase.com\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"cbbackup","group":"couchbase.com","kind":"couchbasebackups"},"code":403} |
Traceback (most recent call last):
|
File "/opt/couchbase/bin/backup_script", line 1220, in config_repo |
result = cbbackupmgr(args_list)
|
File "/opt/couchbase/bin/backup_script", line 80, in cbbackupmgr |
return str(subprocess.check_output(["cbbackupmgr"] + cmd).decode("utf-8")).strip() |
File "/opt/couchbase/lib/python/runtime/lib/python3.7/subprocess.py", line 395, in check_output |
**kwargs).stdout
|
File "/opt/couchbase/lib/python/runtime/lib/python3.7/subprocess.py", line 487, in run |
output=stdout, stderr=stderr)
|
subprocess.CalledProcessError: Command '['cbbackupmgr', 'config', '--archive', 's3://caobackups3/archive', '--repo', 'test-couchbase-xndsq-2020-11-23T09_19_13', '--obj-staging-dir', '/data/staging', '--obj-region', 'us-west-2', '--obj-access-key-id', 'AKIAXQQ2DIGAWSCXJO7I', '--obj-secret-access-key', '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']' returned non-zero exit status 1.During handling of the above exception, another exception occurred:Traceback (most recent call last): |
File "/opt/couchbase/bin/backup_script", line 1234, in <module> |
main()
|
File "/opt/couchbase/bin/backup_script", line 153, in main |
config_repo(args, archive, timestamp)
|
File "/opt/couchbase/bin/backup_script", line 1230, in config_repo |
% grepexc.output))
|
File "/opt/couchbase/bin/backup_script", line 707, in exit_script |
raise_failed_event(get_obj_ref())
|
File "/opt/couchbase/bin/backup_script", line 808, in get_obj_ref |
cbbackup = get_backup(BACKUP)
|
File "/opt/couchbase/bin/backup_script", line 317, in get_backup |
NAMESPACE, CBBACKUP_PLURAL, name)
|
File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/api/custom_objects_api.py", line 954, in get_namespaced_custom_object |
(data) = self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs) # noqa: E501
|
File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/api/custom_objects_api.py", line 1057, in get_namespaced_custom_object_with_http_info |
collection_formats=collection_formats)
|
File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 345, in call_api |
_preload_content, _request_timeout)
|
File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 176, in __call_api |
_request_timeout=_request_timeout)
|
File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 366, in request |
headers=headers)
|
File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/rest.py", line 241, in GET |
query_params=query_params)
|
File "/opt/couchbase/lib/python/runtime/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in request |
raise ApiException(http_resp=r)
|
kubernetes.client.rest.ApiException: (403) |
Reason: Forbidden
|
HTTP response headers: HTTPHeaderDict({'Audit-Id': '93f140b6-94a0-4aeb-bbb4-62d157a2f136', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 23 Nov 2020 09:19:20 GMT', 'Content-Length': '411'}) |
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"couchbasebackups.couchbase.com \"cbbackup\" is forbidden: User \"system:serviceaccount:test-vdc5f:couchbase-backup\" cannot get resource \"couchbasebackups\" in API group \"couchbase.com\" in the namespace \"default\"","reason":"Forbidden","details":{"name":"cbbackup","group":"couchbase.com","kind":"couchbasebackups"},"code":403} |
cbopinfo logs of both failures attached.