Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-3579

Operator tries to change invalid bucket configurations in a loop

    XMLWordPrintable

Details

    • 19 - A Rock and a Hard Place
    • 1

    Description

      Cluster Setup

      • Kind cluster locally run on Mac
      • 3 nodes with all services
      • 1 bucket
      • Cluster version : 7.2.5

      Steps taken in the scenario

      • Created a cluster
      • Created a bucket
      • Issued a deployment to change the bucket ram quota from 1048Mi to 4096Mi.
      • The data service quota is just 1Gi 
      • Operator tries to change the bucket ram quota in a loop and fails

      {"level":"info","ts":"2024-07-17T04:45:06Z","logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"{couchbaseutil.Bucket}.BucketMemoryQuota:1024->4096"}
      		 
      {"level":"info","ts":"2024-07-17T04:45:06Z","logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"request failed: unexpected status code POST http://cb-example-0001.cb-example.default.svc:8091/pools/default/buckets/default 400 Bad Request: {\"errors\":{\"ramQuota\":\"RAM quota specified is too large to be provisioned into this cluster.\"},\"summaries\":{\"ramSummary\":{\"total\":6442450944,\"otherBuckets\":0,\"nodesCount\":3,\"perNodeMegs\":4096,\"thisAlloc\":12884901888,\"thisUsed\":88523864,\"free\":-6442450944},\"hddSummary\":{\"total\":1622360039424,\"otherData\":113538406218,\"otherBuckets\":0,\"thisUsed\":26796540,\"free\":1508821633206}}}","stack":"github.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.Client.doRequest\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/core.go:240\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Client).Post\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/core.go:302\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On.func1\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:222\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:249\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileBuckets\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/bucket.go:371\ngithub.com/couchbase/couchbase-operator/pkg/cluster.reconcileFuncList.run\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:55\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:220\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:544\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:591\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
      		 
      {"level":"info","ts":"2024-07-17T04:45:06Z","logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"+{v2.ClusterStatus}.Conditions[?->2]:{Type:Error Status:True LastUpdateTime:2024-07-17T04:45:06Z LastTransitionTime:2024-07-17T04:45:06Z Reason:ErrorEncountered Message:request failed: unexpected status code POST http://cb-example-0001.cb-example.default.svc:8091/pools/default/buckets/default 400 Bad Request: {\"errors\":{\"ramQuota\":\"RAM quota specified is too large to be provisioned into this cluster.\"},\"summaries\":{\"ramSummary\":{\"total\":6442450944,\"otherBuckets\":0,\"nodesCount\":3,\"perNodeMegs\":4096,\"thisAlloc\":12884901888,\"thisUsed\":88523864,\"free\":-6442450944},\"hddSummary\":{\"total\":1622360039424,\"otherData\":113538406218,\"otherBuckets\":0,\"thisUsed\":26796540,\"free\":1508821633206}}}}"}
      		 
      {"level":"info","ts":"2024-07-17T04:45:07Z","logger":"cluster","msg":"unable to update status","cluster":"default/cb-example","error":"admission webhook \"couchbase-operator-admission.default.svc\" denied the request: validation failure list:\nbucket memory allocation (4Gi) exceeds data service quota (2Gi) on cluster cb-example"}
      		 
      {"level":"info","ts":"2024-07-17T04:45:07Z","logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"+{v2.ClusterStatus}.Conditions[?->2]:{Type:Error Status:True LastUpdateTime:2024-07-17T04:45:06Z LastTransitionTime:2024-07-17T04:45:06Z Reason:ErrorEncountered Message:request failed: unexpected status code POST http://cb-example-0001.cb-example.default.svc:8091/pools/default/buckets/default 400 Bad Request: {\"errors\":{\"ramQuota\":\"RAM quota specified is too large to be provisioned into this cluster.\"},\"summaries\":{\"ramSummary\":{\"total\":6442450944,\"otherBuckets\":0,\"nodesCount\":3,\"perNodeMegs\":4096,\"thisAlloc\":12884901888,\"thisUsed\":88523864,\"free\":-6442450944},\"hddSummary\":{\"total\":1622360039424,\"otherData\":113538406218,\"otherBuckets\":0,\"thisUsed\":26796540,\"free\":1508821633206}}}}"}
      		 
      {"level":"error","ts":"2024-07-17T04:45:07Z","logger":"cluster","msg":"Status update failed","cluster":"default/cb-example","error":"admission webhook \"couchbase-operator-admission.default.svc\" denied the request: validation failure list:\nbucket memory allocation (4Gi) exceeds data service quota (2Gi) on cluster cb-example","stacktrace":"github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile.func1\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:472\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:562\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:591\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}

      Issue

      The admission controller should reject the deployment with such configurations. The operator should not attempt to execute such invalid requests.

      Operator logs:

      https://cb-engineering.s3.amazonaws.com/K8S-3576/cbopinfo-20240717T102016+0530.tar.gz

      Cluster logs:
      https://cb-engineering.s3.amazonaws.com/K8S-3579/collectinfo-2024-07-17T045001-ns_1%40cb-example-0000.cb-example.default.svc.zip
      https://cb-engineering.s3.amazonaws.com/K8S-3579/collectinfo-2024-07-17T045001-ns_1%40cb-example-0001.cb-example.default.svc.zip
      https://cb-engineering.s3.amazonaws.com/K8S-3579/collectinfo-2024-07-17T045001-ns_1%40cb-example-0002.cb-example.default.svc.zip
       

      The cao tool and operator images were built locally on this commit

      commit e00cf70597dbc0a7422c82f0efd0a1a28f75bfcd (HEAD -> master, origin/master, origin/HEAD)
      		 
      Author: usamah jassat <usamah.jassat@couchbase.com>
      		 
      Date:   Thu Jul 11 15:55:19 2024 +0100    K8S-3564: fix TestServerGroupRescheduling when more SGs
      		 
          
      		 
          Change-Id: I13dabc775ad8f47e6f9f89b3445a19a4dd28112e
      		 
          Reviewed-on: [https://review.couchbase.org/c/couchbase-operator/+/212585]
      		 
          Reviewed-by: Justin Ashworth <justin.ashworth@couchbase.com>
      		 
          Tested-by: Build Bot <build@couchbase.com>

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            justin.ashworth Justin Ashworth
            raghav.sk Raghav S K
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty