Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-3287

Possible problem with cluster resizing

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.6.0
    • None
    • None
    • None
    • 2.6.1 -> 2.4.3 & 2.5.1
    • 1

    Description

      A resizing test, TestRzaResizeCluster, is failing because it's expecting 2 pods and is finding 3. 

      Error Messagetimeout: size does not match, wanted 2, got 3Stacktracegoroutine 128 [running]:
      runtime/debug.Stack()
      	runtime/debug/stack.go:24 +0x72
      github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc0009c5860, {0x6adf2e0, 0xc000d953c0})
      	github.com/couchbase/couchbase-operator/test/e2e/e2eutil/util.go:1424 +0x36
      github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustWaitClusterStatusHealthy(0xc0009fea80?, 0xc0009fea80?, 0xc0004421c0?, 0xc000b0fb80?)
      	github.com/couchbase/couchbase-operator/test/e2e/e2eutil/wait_util.go:585 +0x5b
      github.com/couchbase/couchbase-operator/test/e2e.TestRzaResizeCluster(0xc0009c5860)
      	github.com/couchbase/couchbase-operator/test/e2e/rza_test.go:246 +0xd10
      testing.tRunner(0xc0009c5860, 0x55de048)
      	testing/testing.go:1576 +0x217
      created by testing.(*T).Run
      	testing/testing.go:1629 +0x806 

      In the logs there's a removal error:

       

      {"level":"info","ts":"2023-11-22T09:40:06Z","logger":"cluster","msg":"Reconciliation failed","cluster":"test-75ssx/test-couchbase-m8v4r","error":"failed to schedule removal of member 'default': stripe scheduler: server group 'eastus2-1' in class 'default' empty: required resource attribute missing","stack":"github.com/couchbase/couchbase-operator/pkg/util/scheduler.(*stripeSchedulerImpl).Delete\n\tgithub.com/couchbase/couchbase-operator/pkg/util/scheduler/stripe.go:219\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).handleRemoveNode\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:835\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).exec\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:309\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMembers\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:267\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:177\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:492\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:535\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"} 

      Justin Ashworth 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              abhi.bose Abhi Bose (Inactive)
              gilad.kalchheim Gilad Kalchheim
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty