Details
-
Bug
-
Resolution: Unresolved
-
Minor
-
2.7.0
-
Cluster version : 7.6.1-3200
Kubernetes Version : v1.30.0
CAO and operator : 2.7.0 built locally
Environment : Kind cluster
-
19 - A Rock and a Hard Place
-
1
Description
Cluster Setup
- Kind cluster locally run on Mac
- 5 nodes in cluster
- 8 bucket
- Cluster version : 7.6.1-3200
Steps taken in the scenario
- Created a cluster
- Created 8 buckets
- Had a lot of operations in between
- The cluster was currently having 5 pods/nodes
- Two of the pods were deleted using
$kubectl delete pod cb-example-0003 |
$kubectl delete pod cb-example-0004 |
- Issued a scale down concurrently to 3 pods/nodes
- Operator creates 2 new pods cb-example-0007 and cb-example-0008
- Tries rebalancing and fails as pods are unavailable
- After manually failing over, operator initiates a rebalance ejecting cb-example-0003,cb-example-0004, cb-example-0007 and cb-example-0008
Issue
- The operator starts creating pods when scaling down
{"level":"info","ts":"2024-07-24T16:33:22Z","logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"+{v2.ClusterStatus}.Conditions[?->2]:{Type:Scaling Status:True LastUpdateTime:2024-07-24T16:33:22Z LastTransitionTime:2024-07-24T16:33:22Z Reason:ClusterScaling Message:The operator is attempting to scale the cluster};+{v2.ClusterStatus}.Conditions[?->3]:{Type:ScalingUp Status:True LastUpdateTime:2024-07-24T16:33:22Z LastTransitionTime:2024-07-24T16:33:22Z Reason:ScalingUp Message:Scaling Server Class all_services from 3 to 5}"}{"level":"info","ts":"2024-07-24T16:33:23Z","logger":"kubernetes","msg":"Creating pod","cluster":"default/cb-example","name":"cb-example-0008","image":"couchbase/server:7.6.1"}{"level":"info","ts":"2024-07-24T16:33:23Z","logger":"kubernetes","msg":"Creating pod","cluster":"default/cb-example","name":"cb-example-0007","image":"couchbase/server:7.6.1"}{"level":"info","ts":"2024-07-24T16:33:55Z","logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"{v2.ClusterStatus}.Size:5->6;+{v2.ClusterStatus}.Members.Unready[?->2]:cb-example-0007"}{"level":"info","ts":"2024-07-24T16:33:56Z","logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"{v2.ClusterStatus}.Size:6->7;+{v2.ClusterStatus}.Members.Unready[?->3]:cb-example-0008"}{"level":"info","ts":"2024-07-24T16:34:08Z","logger":"cluster","msg":"Pod added to cluster","cluster":"default/cb-example","name":"cb-example-0007"}{"level":"info","ts":"2024-07-24T16:34:19Z","logger":"cluster","msg":"Pod added to cluster","cluster":"default/cb-example","name":"cb-example-0008"} |
Operator logs :
https://cb-engineering.s3.amazonaws.com/K8S-3595/cbopinfo-20240724T221614+0530.tar.gz
Cluster logs :
https://cb-engineering.s3.amazonaws.com/K8S-3595/collectinfo-2024-07-24T165841-ns_1%40cb-example-0000.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3595/collectinfo-2024-07-24T165841-ns_1%40cb-example-0001.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3595/collectinfo-2024-07-24T165841-ns_1%40cb-example-0002.cb-example.default.svc.zip
The cao tool and operator images were built locally on this commit
commit c2e920ddbcfa9b4819d47ad81d0a35c359dd1dc6 (HEAD -> master, origin/master, origin/HEAD)
|
Author: usamah jassat <usamah.jassat@couchbase.com> |
Date: Wed Jul 17 15:11:19 2024 +0100 K8S-3581: don't attempt backend migration when rebalance required |
|
Change-Id: I2d2b6d6d4f8dbb0a30db5bd54a05631d17631eee
|
Reviewed-on: https://review.couchbase.org/c/couchbase-operator/+/212890 |
Reviewed-by: Yusuf Ramzan <yusuf.ramzan@couchbase.com> |
Tested-by: Build Bot <build@couchbase.com> |