Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2118

BackuptoS3: Error getting archive information

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.2.0
    • 2.1.0
    • None
    • 16: Autoscaling/PE/Docs
    • 1

    Description

      Job: http://qa.sc.couchbase.com/job/roo-test/215/console 

      Test Case: TestFullOnlyS3 (https://github.com/couchbase/couchbase-operator/blob/master/test/e2e/backup_test.go#L237) 

      Couchbase Server: couchbase/server: 6.6.0 (since the CBSE for the same issue uses 6.6.0) 

      Operator: 2.1.0-250 (GA for 2.1.0)

      Istio Enabled with MTLS_PERMISSIVE

      Backup Image: 6.6.0-116

      Backtrace:

      FAIL: TestOperator/TestReplaceFullOnlyBackupS3 (494.44s)
      07:47:59         crd_util.go:27: creating couchbase cluster: test-couchbase-m2t95
      07:47:59         util.go:1357: time out waiting for backup event BackupStarted, Backup `full-only` started
      07:47:59         util.go:1358: goroutine 892 [running]:
      07:47:59             runtime/debug.Stack(0x1ca110d, 0x0, 0x0)
      07:47:59             	/jenkins/workspace/roo-test/go/src/runtime/debug/stack.go:24 +0xab
      07:47:59             github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc000665d00, 0x2614fe0, 0xc000458050)
      07:47:59             	/jenkins/workspace/roo-test/test/e2e/e2eutil/util.go:1353 +0x34
      07:47:59             github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustWaitForBackupEvent(0xc000665d00, 0xc0001c6180, 0xc0002b46c0, 0xc000f98f00, 0x45d964b800)
      07:47:59             	/jenkins/workspace/roo-test/test/e2e/e2eutil/wait_util.go:611 +0xad
      07:47:59             github.com/couchbase/couchbase-operator/test/e2e.testReplaceFullOnlyBackup(0xc000665d00, 0xc000066f01)
      07:47:59             	/jenkins/workspace/roo-test/test/e2e/backup_test.go:439 +0x4fc
      07:47:59             github.com/couchbase/couchbase-operator/test/e2e.TestReplaceFullOnlyBackupS3(0xc000665d00)
      07:47:59             	/jenkins/workspace/roo-test/test/e2e/backup_test.go:482 +0x3e
      07:47:59             testing.tRunner(0xc000665d00, 0x23b0810)
      07:47:59             	/jenkins/workspace/roo-test/go/src/testing/testing.go:909 +0x19a
      07:47:59             created by testing.(*T).Run
      07:47:59             	/jenkins/workspace/roo-test/go/src/testing/testing.go:960 +0x652 

      cbbackupmgr Error:

      2021-04-12T13:29:00 INFO Checking connection to Kubernetes API...
      2021-04-12T13:29:00 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1ac18>: Failed to establish a new connection: [Errno 111] Connection refused')': /version/
      2021-04-12T13:29:00 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1acf8>: Failed to establish a new connection: [Errno 111] Connection refused')': /version/
      2021-04-12T13:29:00 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1a898>: Failed to establish a new connection: [Errno 111] Connection refused')': /version/
      2021-04-12T13:29:00 INFO Unable to contact Kubernetes API: HTTPSConnectionPool(host='10.243.240.1', port=443): Max retries exceeded with url: /version/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f279af1a978>: Failed to establish a new connection: [Errno 111] Connection refused'))
      2021-04-12T13:29:05 INFO attempting to query K8S objects
      2021-04-12T13:29:05 INFO k8s config loaded
      2021-04-12T13:29:05 INFO Starting backup
      2021-04-12T13:29:06 ERROR Failed to list repositories: return code: 1
      2021-04-12T13:29:06 ERROR Failed to list repositories: output: b'Error getting archive information: Error opening archive at /data/staging/archive due to `Not an archive directory`\n'
      Traceback (most recent call last):
        File "/opt/couchbase/bin/backup.py", line 1059, in <module>
          main()
        File "/opt/couchbase/bin/backup.py", line 269, in main
          do_backup(context)
        File "/opt/couchbase/bin/backup.py", line 719, in do_backup
          repo_names = get_repo_names(context)
        File "/opt/couchbase/bin/backup.py", line 684, in get_repo_names
          return [repo["name"] for repo in get_repos(context)]
      TypeError: 'NoneType' object is not iterable
      Ok 

      All Backup to S3 tests are failing with the same error.

      Logs attached contains cbopinfo of all S3 test failures. 

      The S3 tests pass with latest commit of Operator version: 2.2 with same parameters mentioned above with the exception of server version used: 6.6.1 in 2.2 scenario. (http://qa.sc.couchbase.com/job/roo-test/213/console)

       

      P.S.: Using this ticket as a common object for all S3 failures. In case the test cases are found to be failing due to separate reasons upon further look, new tickets will be created.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            daniel.ma Daniel Ma (Inactive)
            prateek.kumar Prateek Kumar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty