Details
-
Bug
-
Resolution: Not a Bug
-
Critical
-
2.6.4
Description
Kubernetes Version | 1.25 |
Couchbase Server | 7.2.5 (Pre Upgrade) → 7.6.1 (Post Upgrade) |
Operator | 2.6.4-108 |
Cluster Setup
- Each node is an m5.4xlarge instance. (16 vCPUs and 64GB RAM)
- 6 Data Service, 4 Index Service & Query Service Nodes.
- 10 Buckets (with 1 replica), Full Eviction and Auto-failover set to 5s.
- 110GB data per bucket → ~1.1TB data loaded onto cluster before beginning of upgrade.
- 50 Primary Indexes with 1 Replica each. (Total 100 Indexes)
Upgrade Process
- SwapRebalance Upgrade to update Couchbase Server from 7.2.5 to 7.6.1.
- Continuous query and data workload on the buckets during the update process.
- Around 40-50% CPU load on all servers during the upgrade.
Issue
- 6 KV Nodes got upgraded successfully without any issues.
- 2 Query-Index nodes got upgraded successfully.
- While, the 3rd Query-Index node was being upgraded, the index rebalances kept failing repeatedly. The node was going up to 90-94% on RAM usage.
- This was happening for a few hours.
- The upgrade was not successful.
Logs
Before Upgrade
http://supportal.couchbase.com/snapshot/fbbf7ff7fc529a2a3db10b55b3dafc84::0
2024-05-20_cb_7.2.5_swap_rebalance_cbopinfo-20240520T144749+0530.tar.gz
During Upgrade when Index Rebalances Failed
http://supportal.couchbase.com/snapshot/fbbf7ff7fc529a2a3db10b55b3dafc84::1
2024-05-20_cb_7.6.1_during_swap_rebalance_cbopinfo-20240520T234427+0530.tar.gz
Screenshots
State of Primary Indexes when the 3rd Query-Index node was being updated.
State of the CB Nodes when the 3rd Query-Index node was being updated.