Details
-
Bug
-
Resolution: Fixed
-
Major
-
2.3.0
-
1
Description
Steps to Reproduce:
- Create Operator & DAC pod with 2.3.0 build 279.
- Create 2 Couchbase Cluster: cb-example, cb-example-new with all services.
- cb-example is a 1 node cluster running 6.6.2
- cb-example-new is a 1 node cluster running 7.0.3
- Status of the cluster:
Prateeks-MacBook-Pro:Downloads prateekkumar$ kubectl get pods
NAME READY STATUS RESTARTS AGE
cb-example-0000 1/1 Running 0 5m6s
cb-example-new-0000 1/1 Running 0 5m6s
couchbase-operator-5b4cb9f599-dzfbg 1/1 Running 0 6m54s
couchbase-operator-admission-65469748f6-jd99w 1/1 Running 0 133m
6. Port-forward cb-example-0000 pod to access Couchbase UI.
7. Click on the Add Servers option and provide Host name as cb-example-new-srv.default (see image1.png, image3.png)
8. Click on Rebalance Option to rebalance in the new server added above. Rebalance Starts (see image2.png)
9. Rebalance completes successfully. Report attached.
10. Check Operator logs, Error messages is repeated in the cluster:
{"level":"info","ts":1647614100.0069945,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"failed to rebalance: unexpected status code: request failed POST http://cb-example-0000.cb-example.default.svc:8091/controller/rebalance 400 Bad Request:
{\"mismatch\":1}","stack":"github.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.Client.doRequest\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/core.go:209\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Client).Post\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/core.go:258\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On.func1\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:220\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:247\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).rebalance\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/rebalance.go:210\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).handleRebalance\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:1036\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).exec\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:307\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMembers\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:256\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:170\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:481\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:524\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1647614102.217759,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example-new","error":"member error","stack":"github.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On.func1\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:217\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:247\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).getStatus\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/status.go:112\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).GetStatus\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/status.go:99\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).newReconcileMachine\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:206\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:165\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:481\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:524\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}I checked with the Server QE folks who mentioned that when we use the Add Server option, Server will not kick off Rebalance in operation automatically, that is a manual Operation. Hence, Step 8. (Although, we see the error messages even if we don't click on Rebalance option in the UI after Step 7) Logs attached: cbopinfo-20220318T200705+0530.tar.gz
We see same error in the operator logs even when HostName is given as cb-example-new-0000.cb-example-new.default.svc in the Add Server Option. Logs attached: cbopinfo-20220318T202359+0530.tar.gz
We also see the error in the operator logs when 7.0.3(cb-example-new) is added to the 7.0.0(cb-example) node instead of adding 7.0.3 to 6.6.2 node mentioned above. However we get additional "member error" before "failed to rebalance" error:
{"level":"info","ts":1647615872.9013839,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example-new","error":"member error","stack":"github.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On.func1\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:217\ngithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil.(*Request).On\n\tgithub.com/couchbase/couchbase-operator/pkg/util/couchbaseutil/api.go:247\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).getStatus\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/status.go:112\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).GetStatus\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/status.go:99\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).newReconcileMachine\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:206\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:165\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:481\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:524\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"} |
Logs attached: cbopinfo-20220318T203738+0530.tar.gz