Details
-
Bug
-
Resolution: Unresolved
-
Major
-
2.7.0
-
Initial Couchbase Version : 7.2.5-7596
Kubernetes Version : v1.30.0
CAO and operator : 2.7.0 built locally
Environment : Kind cluster
-
19 - A Rock and a Hard Place
-
1
Description
Cluster Setup
- Kind cluster locally run on Mac
- 3 nodes with all services
- 1 bucket
- Cluster version : 7.2.5
Steps taken in the scenario
- Created a cluster
- Created a bucket
- Issued a deployment to change the bucket ram quota from 1048Mi to 4096Mi.
- The data service quota is just 1Gi
- Operator tries to change the bucket ram quota in a loop and fails
{"level":"info","ts":"2024-07-17T04:45:06Z","logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"{couchbaseutil.Bucket}.BucketMemoryQuota:1024->4096"}
|
{"level":"info","ts":"2024-07-17T04:45:06Z","logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"request failed: unexpected status code POST http://cb-example-0001.cb-example.default.svc:8091/pools/default/buckets/default 400 Bad Request: {\"errors\":{\"ramQuota\":\"RAM quota specified is too large to be provisioned into this cluster.\"},\"summaries\":{\"ramSummary\":{\"total\":6442450944,\"otherBuckets\":0,\"nodesCount\":3,\"perNodeMegs\":4096,\"thisAlloc\":12884901888,\"thisUsed\":88523864,\"free\":-6442450944},\"hddSummary\":{\"total\":1622360039424,\"otherData\":113538406218,\"otherBuckets\":0,\"thisUsed\":26796540,\"free\":1508821633206}}}","stack":"github.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.Client.doRequest\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/core.go:240\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Client).Post\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/core.go:302\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On.func1\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:222\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:249\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileBuckets\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/bucket.go:371\ngithub.com/couchbase/couchbase-operator/pkg/cluster.reconcileFuncList.run\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:55\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:220\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:544\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:591\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
|
{"level":"info","ts":"2024-07-17T04:45:06Z","logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"+{v2.ClusterStatus}.Conditions[?->2]:{Type:Error Status:True LastUpdateTime:2024-07-17T04:45:06Z LastTransitionTime:2024-07-17T04:45:06Z Reason:ErrorEncountered Message:request failed: unexpected status code POST http://cb-example-0001.cb-example.default.svc:8091/pools/default/buckets/default 400 Bad Request: {\"errors\":{\"ramQuota\":\"RAM quota specified is too large to be provisioned into this cluster.\"},\"summaries\":{\"ramSummary\":{\"total\":6442450944,\"otherBuckets\":0,\"nodesCount\":3,\"perNodeMegs\":4096,\"thisAlloc\":12884901888,\"thisUsed\":88523864,\"free\":-6442450944},\"hddSummary\":{\"total\":1622360039424,\"otherData\":113538406218,\"otherBuckets\":0,\"thisUsed\":26796540,\"free\":1508821633206}}}}"}
|
{"level":"info","ts":"2024-07-17T04:45:07Z","logger":"cluster","msg":"unable to update status","cluster":"default/cb-example","error":"admission webhook \"couchbase-operator-admission.default.svc\" denied the request: validation failure list:\nbucket memory allocation (4Gi) exceeds data service quota (2Gi) on cluster cb-example"}
|
{"level":"info","ts":"2024-07-17T04:45:07Z","logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"+{v2.ClusterStatus}.Conditions[?->2]:{Type:Error Status:True LastUpdateTime:2024-07-17T04:45:06Z LastTransitionTime:2024-07-17T04:45:06Z Reason:ErrorEncountered Message:request failed: unexpected status code POST http://cb-example-0001.cb-example.default.svc:8091/pools/default/buckets/default 400 Bad Request: {\"errors\":{\"ramQuota\":\"RAM quota specified is too large to be provisioned into this cluster.\"},\"summaries\":{\"ramSummary\":{\"total\":6442450944,\"otherBuckets\":0,\"nodesCount\":3,\"perNodeMegs\":4096,\"thisAlloc\":12884901888,\"thisUsed\":88523864,\"free\":-6442450944},\"hddSummary\":{\"total\":1622360039424,\"otherData\":113538406218,\"otherBuckets\":0,\"thisUsed\":26796540,\"free\":1508821633206}}}}"}
|
{"level":"error","ts":"2024-07-17T04:45:07Z","logger":"cluster","msg":"Status update failed","cluster":"default/cb-example","error":"admission webhook \"couchbase-operator-admission.default.svc\" denied the request: validation failure list:\nbucket memory allocation (4Gi) exceeds data service quota (2Gi) on cluster cb-example","stacktrace":"github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile.func1\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:472\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:562\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:591\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
|
Issue
The admission controller should reject the deployment with such configurations. The operator should not attempt to execute such invalid requests.
Operator logs:
https://cb-engineering.s3.amazonaws.com/K8S-3576/cbopinfo-20240717T102016+0530.tar.gz
Cluster logs:
https://cb-engineering.s3.amazonaws.com/K8S-3579/collectinfo-2024-07-17T045001-ns_1%40cb-example-0000.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3579/collectinfo-2024-07-17T045001-ns_1%40cb-example-0001.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3579/collectinfo-2024-07-17T045001-ns_1%40cb-example-0002.cb-example.default.svc.zip
The cao tool and operator images were built locally on this commit
commit e00cf70597dbc0a7422c82f0efd0a1a28f75bfcd (HEAD -> master, origin/master, origin/HEAD)
|
Author: usamah jassat <usamah.jassat@couchbase.com> |
Date: Thu Jul 11 15:55:19 2024 +0100 K8S-3564: fix TestServerGroupRescheduling when more SGs |
|
Change-Id: I13dabc775ad8f47e6f9f89b3445a19a4dd28112e
|
Reviewed-on: [https://review.couchbase.org/c/couchbase-operator/+/212585] |
Reviewed-by: Justin Ashworth <justin.ashworth@couchbase.com> |
Tested-by: Build Bot <build@couchbase.com> |