Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-3347

Incomplete backup when running Operator 2.6.1 and server 7.6.0

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.7.0
    • None
    • None
    • None
    • 2 - 2024 Midnight Horizon, 5 - Forces of Destiny
    • 1

    Description

      Various backup tests a failing when running 2.6.1 and server 7.6.0. 

      For example, TestBackupFullIncremental:

       

      Error Messagecontext deadline exceeded: failed to wait for event BackupCompleted/Backup `backup-lbndf` completedStacktracegoroutine 497 [running]:
      runtime/debug.Stack()
      	runtime/debug/stack.go:24 +0x72
      github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc0007469c0, {0x6b01e00, 0xc00fc63420})
      	github.com/couchbase/couchbase-operator/test/e2e/e2eutil/util.go:1433 +0x36
      github.com/couchbase/couchbase-operator/test/e2e/e2eutil.mustWaitForResourceEventFromNow(0xc002c810b4?, 0xc?, {0x6b12ad8, 0xc002c5ab00}, 0x7?, 0x0?)
      	github.com/couchbase/couchbase-operator/test/e2e/e2eutil/wait_util.go:315 +0xfd
      github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustWaitForBackupEvent(...)
      	github.com/couchbase/couchbase-operator/test/e2e/e2eutil/wait_util.go:652
      github.com/couchbase/couchbase-operator/test/e2e.testFullIncremental(0xc0007469c0, 0x0?)
      	github.com/couchbase/couchbase-operator/test/e2e/backup_test.go:63 +0x1045
      github.com/couchbase/couchbase-operator/test/e2e.TestBackupFullIncremental(0x0?)
      	github.com/couchbase/couchbase-operator/test/e2e/backup_test.go:81 +0x31
      testing.tRunner(0xc0007469c0, 0x55f4eb8)
      	testing/testing.go:1576 +0x217
      created by testing.(*T).Run
      	testing/testing.go:1629 +0x806 

      The error in the logs:

       

      Error formatting macro: code: java.lang.StackOverflowError

      {"level":"info","ts":"2024-01-18T09:51:23Z","logger":"cluster","msg":"Reconciliation failed","cluster":"test-cng4k/test-couchbase-sh7ml","error":"timeout: unexpected status code: request failed POST http://test-couchbase-sh7ml-0000.test-couchbase-sh7ml.test-cng4k.svc:8091/settings/autoFailover 400 Bad Request: {\"errors\":{\"failoverServerGroup\":\"Unsupported key\"}}","stack":"github.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.Client.doRequest\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/core.go:212\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Client).Post\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/core.go:274\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On.func1\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:222\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On.func2.1\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:240\ngithub.com/couchbase/couchbase-operator/pkg/util/retryutil.Retry\n\tgithub.com/couchbase/couchbase-operator/pkg/util/retryutil/retryutil.go:14\ngithub.com/couchbase/couchbase-operator/pkg/util/retryutil.RetryFor\n\tgithub.com/couchbase/couchbase-operator/pkg/util/retryutil/retryutil.go:30\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On.func2\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:243\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:249\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).initInitialMember\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/member.go:493\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).configureInitialMember\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/member.go:406\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).create\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:335\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:152\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:492\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:535\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              usamah.jassat Usamah Jassat
              gilad.kalchheim Gilad Kalchheim
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty