Details
-
Bug
-
Resolution: Unresolved
-
Major
-
2.7.0
-
Cluster version : 7.0.5-7659
Kubernetes Version : v1.30.0
CAO and operator : 2.7.0 built locally
Environment : Kind cluster
-
1
Description
Cluster Setup
- Kind cluster locally run on Mac
- 2 nodes with all services
- 1 bucket
- Initial Cluster version : 7.0.5-7659
Steps taken in the scenario
- Created a cluster
- Created 1 bucket
- Changed the cluster config to add a logging sidecar.
- Swap rebalance is issued by the operator to reconcile the cluster to the changes
- A new pod is added
- Immediately rebalance is issued
- Rebalance fails with not_all_nodes_are_ready_yet error
Rebalance exited with reason {not_all_nodes_are_ready_yet,
|
['ns_1@cb-example-0002.cb-example.default.svc']}. |
Rebalance Operation Id = 787043bdcbfc70901e0b1ce0832fda77
|
Issue
- The operator should check the status of the pod after adding before triggering a rebalance
- The pod has to be warmed up and all the services should be up and running before adding. It should be ready for a rebalance.
Operator logs :
https://cb-engineering.s3.amazonaws.com/K8S-3598/cbopinfo-20240725T170701+0530.tar.gz
Cluster logs :
https://cb-engineering.s3.amazonaws.com/K8S-3598/collectinfo-2024-07-25T114628-ns_1%40cb-example-0005.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3598/collectinfo-2024-07-25T114628-ns_1%40cb-example-0001.cb-example.default.svc.zip
The cao tool and operator images were built locally on this commit
commit c2e920ddbcfa9b4819d47ad81d0a35c359dd1dc6 (HEAD -> master, origin/master, origin/HEAD)
|
Author: usamah jassat <usamah.jassat@couchbase.com> |
Date: Wed Jul 17 15:11:19 2024 +0100 K8S-3581: don't attempt backend migration when rebalance required |
|
Change-Id: I2d2b6d6d4f8dbb0a30db5bd54a05631d17631eee
|
Reviewed-on: https://review.couchbase.org/c/couchbase-operator/+/212890 |
Reviewed-by: Yusuf Ramzan <yusuf.ramzan@couchbase.com> |
Tested-by: Build Bot <build@couchbase.com> |
Attachments
Gerrit Reviews
For Gerrit Dashboard: K8S-3598 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
213604,5 | K8S-3598 Fixed not all nodes ready for rebalance | master | couchbase-operator | Status: MERGED | +2 | +1 |
213741,1 | K8S-3598 Fixed not all nodes ready for rebalance | 2.7.x | couchbase-operator | Status: ABANDONED | 0 | 0 |
213742,2 | K8S-3598 Fixed not all nodes ready for rebalance | 2.7.x | couchbase-operator | Status: MERGED | +2 | +1 |