Details
-
Bug
-
Resolution: Fixed
-
Major
-
2.1.0
-
None
-
16: Autoscaling/PE/Docs
-
1
Description
Job: http://qa.sc.couchbase.com/job/roo-test/215/console
Test Case: TestFullOnlyS3 (https://github.com/couchbase/couchbase-operator/blob/master/test/e2e/backup_test.go#L237)
Couchbase Server: couchbase/server: 6.6.0 (since the CBSE for the same issue uses 6.6.0)
Operator: 2.1.0-250 (GA for 2.1.0)
Istio Enabled with MTLS_PERMISSIVE
Backup Image: 6.6.0-116
Backtrace:
FAIL: TestOperator/TestReplaceFullOnlyBackupS3 (494.44s) |
07:47:59 crd_util.go:27: creating couchbase cluster: test-couchbase-m2t95 |
07:47:59 util.go:1357: time out waiting for backup event BackupStarted, Backup `full-only` started |
07:47:59 util.go:1358: goroutine 892 [running]: |
07:47:59 runtime/debug.Stack(0x1ca110d, 0x0, 0x0) |
07:47:59 /jenkins/workspace/roo-test/go/src/runtime/debug/stack.go:24 +0xab |
07:47:59 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc000665d00, 0x2614fe0, 0xc000458050) |
07:47:59 /jenkins/workspace/roo-test/test/e2e/e2eutil/util.go:1353 +0x34 |
07:47:59 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustWaitForBackupEvent(0xc000665d00, 0xc0001c6180, 0xc0002b46c0, 0xc000f98f00, 0x45d964b800) |
07:47:59 /jenkins/workspace/roo-test/test/e2e/e2eutil/wait_util.go:611 +0xad |
07:47:59 github.com/couchbase/couchbase-operator/test/e2e.testReplaceFullOnlyBackup(0xc000665d00, 0xc000066f01) |
07:47:59 /jenkins/workspace/roo-test/test/e2e/backup_test.go:439 +0x4fc |
07:47:59 github.com/couchbase/couchbase-operator/test/e2e.TestReplaceFullOnlyBackupS3(0xc000665d00) |
07:47:59 /jenkins/workspace/roo-test/test/e2e/backup_test.go:482 +0x3e |
07:47:59 testing.tRunner(0xc000665d00, 0x23b0810) |
07:47:59 /jenkins/workspace/roo-test/go/src/testing/testing.go:909 +0x19a |
07:47:59 created by testing.(*T).Run |
07:47:59 /jenkins/workspace/roo-test/go/src/testing/testing.go:960 +0x652 |
cbbackupmgr Error:
2021-04-12T13:29:00 INFO Checking connection to Kubernetes API... |
2021-04-12T13:29:00 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1ac18>: Failed to establish a new connection: [Errno 111] Connection refused')': /version/ |
2021-04-12T13:29:00 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1acf8>: Failed to establish a new connection: [Errno 111] Connection refused')': /version/ |
2021-04-12T13:29:00 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1a898>: Failed to establish a new connection: [Errno 111] Connection refused')': /version/ |
2021-04-12T13:29:00 INFO Unable to contact Kubernetes API: HTTPSConnectionPool(host='10.243.240.1', port=443): Max retries exceeded with url: /version/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1a978>: Failed to establish a new connection: [Errno 111] Connection refused')) |
2021-04-12T13:29:05 INFO attempting to query K8S objects |
2021-04-12T13:29:05 INFO k8s config loaded |
2021-04-12T13:29:05 INFO Starting backup |
2021-04-12T13:29:06 ERROR Failed to list repositories: return code: 1 |
2021-04-12T13:29:06 ERROR Failed to list repositories: output: b'Error getting archive information: Error opening archive at /data/staging/archive due to `Not an archive directory`\n' |
Traceback (most recent call last):
|
File "/opt/couchbase/bin/backup.py", line 1059, in <module> |
main()
|
File "/opt/couchbase/bin/backup.py", line 269, in main |
do_backup(context)
|
File "/opt/couchbase/bin/backup.py", line 719, in do_backup |
repo_names = get_repo_names(context)
|
File "/opt/couchbase/bin/backup.py", line 684, in get_repo_names |
return [repo["name"] for repo in get_repos(context)] |
TypeError: 'NoneType' object is not iterable |
Ok
|
All Backup to S3 tests are failing with the same error.
Logs attached contains cbopinfo of all S3 test failures.
The S3 tests pass with latest commit of Operator version: 2.2 with same parameters mentioned above with the exception of server version used: 6.6.1 in 2.2 scenario. (http://qa.sc.couchbase.com/job/roo-test/213/console)
P.S.: Using this ticket as a common object for all S3 failures. In case the test cases are found to be failing due to separate reasons upon further look, new tickets will be created.