Details
-
Bug
-
Resolution: Fixed
-
Major
-
2.7.0
-
Initial Cluster version : 7.6.0-2176
Upgrade Cluster version : 7.6.1-3200
Kubernetes Version : v1.30.0
CAO and operator : 2.7.0 built locally
Environment : Kind cluster
-
18 -Lost to Eternity
-
1
Description
Cluster Setup
- Kind cluster locally run on Mac
- 6 nodes in cluster
- 9 buckets
- Initial Cluster version : 7.6.0-2176
- Upgrade Cluster version : 7.6.1-3200
Steps taken in the scenario
- Created a cluster
- Created 9 buckets
- Issued an upgrade with delta recovery
- When one of the pods(cb-example-0001) was picked up for failover, manually deleted 2 other pods
$kubectl delete pod cb-example-0004 |
$kubectl delete pod cb-example-0005 |
- The pods are failed over and then operator attempts to recreate it back
- Instead of recreating with upgrade version 7.6.1, the pods are created with 7.6.0.
- The upgrade then continues from cb-example-0001.
- Then later cb-example-0004 and cb-example-0005 are upgraded as well.
Issue
- The operator should create pods with upgrade version instead of older version
{"level":"info","ts":"2024-07-25T05:08:29Z","logger":"cluster","msg":"Pods failed over","cluster":"default/cb-example"} |
{"level":"info","ts":"2024-07-25T05:08:29Z","logger":"kubernetes","msg":"Creating pod","cluster":"default/cb-example","name":"cb-example-0004","image":"couchbase/server:7.6.0"} |
|
{"level":"info","ts":"2024-07-25T05:09:10Z","logger":"kubernetes","msg":"Creating pod","cluster":"default/cb-example","name":"cb-example-0005","image":"couchbase/server:7.6.0"} |
Operator logs :
https://cb-engineering.s3.amazonaws.com/K8S-3597/cbopinfo-20240725T111733+0530.tar.gz
Cluster logs :
The cao tool and operator images were built locally on this commit
commit c2e920ddbcfa9b4819d47ad81d0a35c359dd1dc6 (HEAD -> master, origin/master, origin/HEAD)
|
Author: usamah jassat <usamah.jassat@couchbase.com> |
Date: Wed Jul 17 15:11:19 2024 +0100 K8S-3581: don't attempt backend migration when rebalance required |
|
Change-Id: I2d2b6d6d4f8dbb0a30db5bd54a05631d17631eee
|
Reviewed-on: https://review.couchbase.org/c/couchbase-operator/+/212890 |
Reviewed-by: Yusuf Ramzan <yusuf.ramzan@couchbase.com> |
Tested-by: Build Bot <build@couchbase.com> |