Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
2.7.0
-
Couchbase Version : 7.6.0-2176
Kubernetes Version : v1.30.0
CAO and operator : 2.7.0 built locally
Environment : Kind cluster
-
19 - A Rock and a Hard Place
-
5
Description
Cluster Setup
- Kind cluster locally run on Mac
- 5 nodes with all services
- 2 buckets
- Cluster version : 7.6.0-2176
Steps taken in the scenario
- Created a cluster
- Created 2 buckets
- Changed the storage backend of one of the buckets in cluster from couchstore to magma
- Swap rebalance was successfully completed for 4 pods.
- During swap rebalance of 5th pod (cb-example-0000.cb-example.default.svc replaced with cb-example-0009.cb-example.default.svc), killed memcached on a pod, rebalance fails
- Post this, it retries the rebalance by ejecting cb-example-0009.cb-example.default.svc instead of ejecting cb-example-0000.cb-example.default.svc
- New pod is created ( cb-example-0010.cb-example.default.svc)
- During swap rebalance of 5th pod (cb-example-0000.cb-example.default.svc replaced with cb-example-0010.cb-example.default.svc), killed memcached on a pod, rebalance fails
- Post this, it retries the rebalance by ejecting cb-example-0010.cb-example.default.svc instead of ejecting cb-example-0000.cb-example.default.svc
- New pod is created ( cb-example-0011.cb-example.default.svc)
- Swap rebalance of this succeeds as no chaos action occurs(memcached kill is not manually done this time).
Issue
- When swap rebalance fails, it should retry the swap rebalance and not eject the newly added pod
Operator logs:
https://cb-engineering.s3.amazonaws.com/K8S-3586/cbopinfo-20240718T080208+0530.tar.gz
Cluster logs:
https://cb-engineering.s3.amazonaws.com/K8S-3586/collectinfo-2024-07-18T023307-ns_1%40cb-example-0005.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3586/collectinfo-2024-07-18T023307-ns_1%40cb-example-0006.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3586/collectinfo-2024-07-18T023307-ns_1%40cb-example-0007.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3586/collectinfo-2024-07-18T023307-ns_1%40cb-example-0008.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3586/collectinfo-2024-07-18T023307-ns_1%40cb-example-0011.cb-example.default.svc.zip
The cao tool and operator images were built locally on this commit
commit c2e920ddbcfa9b4819d47ad81d0a35c359dd1dc6 (HEAD -> master, origin/master, origin/HEAD)
|
Author: usamah jassat <usamah.jassat@couchbase.com> |
Date: Wed Jul 17 15:11:19 2024 +0100 K8S-3581: don't attempt backend migration when rebalance required |
|
Change-Id: I2d2b6d6d4f8dbb0a30db5bd54a05631d17631eee
|
Reviewed-on: https://review.couchbase.org/c/couchbase-operator/+/212890 |
Reviewed-by: Yusuf Ramzan <yusuf.ramzan@couchbase.com> |
Tested-by: Build Bot <build@couchbase.com> |