Details
-
Bug
-
Resolution: Fixed
-
Major
-
2.1.0
-
None
-
16: Autoscaling/PE/Docs
-
1
Description
Job: http://qa.sc.couchbase.com/job/roo-test/215/console
Test Case: TestFullOnlyS3 (https://github.com/couchbase/couchbase-operator/blob/master/test/e2e/backup_test.go#L237)
Couchbase Server: couchbase/server: 6.6.0 (since the CBSE for the same issue uses 6.6.0)
Operator: 2.1.0-250 (GA for 2.1.0)
Istio Enabled with MTLS_PERMISSIVE
Backup Image: 6.6.0-116
Backtrace:
FAIL: TestOperator/TestReplaceFullOnlyBackupS3 (494.44s) |
07:47:59 crd_util.go:27: creating couchbase cluster: test-couchbase-m2t95 |
07:47:59 util.go:1357: time out waiting for backup event BackupStarted, Backup `full-only` started |
07:47:59 util.go:1358: goroutine 892 [running]: |
07:47:59 runtime/debug.Stack(0x1ca110d, 0x0, 0x0) |
07:47:59 /jenkins/workspace/roo-test/go/src/runtime/debug/stack.go:24 +0xab |
07:47:59 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc000665d00, 0x2614fe0, 0xc000458050) |
07:47:59 /jenkins/workspace/roo-test/test/e2e/e2eutil/util.go:1353 +0x34 |
07:47:59 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustWaitForBackupEvent(0xc000665d00, 0xc0001c6180, 0xc0002b46c0, 0xc000f98f00, 0x45d964b800) |
07:47:59 /jenkins/workspace/roo-test/test/e2e/e2eutil/wait_util.go:611 +0xad |
07:47:59 github.com/couchbase/couchbase-operator/test/e2e.testReplaceFullOnlyBackup(0xc000665d00, 0xc000066f01) |
07:47:59 /jenkins/workspace/roo-test/test/e2e/backup_test.go:439 +0x4fc |
07:47:59 github.com/couchbase/couchbase-operator/test/e2e.TestReplaceFullOnlyBackupS3(0xc000665d00) |
07:47:59 /jenkins/workspace/roo-test/test/e2e/backup_test.go:482 +0x3e |
07:47:59 testing.tRunner(0xc000665d00, 0x23b0810) |
07:47:59 /jenkins/workspace/roo-test/go/src/testing/testing.go:909 +0x19a |
07:47:59 created by testing.(*T).Run |
07:47:59 /jenkins/workspace/roo-test/go/src/testing/testing.go:960 +0x652 |
cbbackupmgr Error:
2021-04-12T13:29:00 INFO Checking connection to Kubernetes API... |
2021-04-12T13:29:00 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1ac18>: Failed to establish a new connection: [Errno 111] Connection refused')': /version/ |
2021-04-12T13:29:00 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1acf8>: Failed to establish a new connection: [Errno 111] Connection refused')': /version/ |
2021-04-12T13:29:00 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1a898>: Failed to establish a new connection: [Errno 111] Connection refused')': /version/ |
2021-04-12T13:29:00 INFO Unable to contact Kubernetes API: HTTPSConnectionPool(host='10.243.240.1', port=443): Max retries exceeded with url: /version/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1a978>: Failed to establish a new connection: [Errno 111] Connection refused')) |
2021-04-12T13:29:05 INFO attempting to query K8S objects |
2021-04-12T13:29:05 INFO k8s config loaded |
2021-04-12T13:29:05 INFO Starting backup |
2021-04-12T13:29:06 ERROR Failed to list repositories: return code: 1 |
2021-04-12T13:29:06 ERROR Failed to list repositories: output: b'Error getting archive information: Error opening archive at /data/staging/archive due to `Not an archive directory`\n' |
Traceback (most recent call last):
|
File "/opt/couchbase/bin/backup.py", line 1059, in <module> |
main()
|
File "/opt/couchbase/bin/backup.py", line 269, in main |
do_backup(context)
|
File "/opt/couchbase/bin/backup.py", line 719, in do_backup |
repo_names = get_repo_names(context)
|
File "/opt/couchbase/bin/backup.py", line 684, in get_repo_names |
return [repo["name"] for repo in get_repos(context)] |
TypeError: 'NoneType' object is not iterable |
Ok
|
All Backup to S3 tests are failing with the same error.
Logs attached contains cbopinfo of all S3 test failures.
The S3 tests pass with latest commit of Operator version: 2.2 with same parameters mentioned above with the exception of server version used: 6.6.1 in 2.2 scenario. (http://qa.sc.couchbase.com/job/roo-test/213/console)
P.S.: Using this ticket as a common object for all S3 failures. In case the test cases are found to be failing due to separate reasons upon further look, new tickets will be created.
Attachments
For Gerrit Dashboard: K8S-2118 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
151276,3 | K8S-2118: BackuptoS3: Error getting archive information | master | couchbase-operator-backup | Status: MERGED | +2 | +1 |