Index Rebalancing Repeatedly Failed while Swap Rebalance Upgrade [CAO 2.6.4]

Cluster Setup

• Each node is an m5.4xlarge instance. (16 vCPUs and 64GB RAM) 
• 6 Data Service, 4 Index Service & Query Service Nodes.
• 10 Buckets (with 1 replica), Full Eviction and Auto-failover set to 5s.
• 110GB data per bucket → ~1.1TB data loaded onto cluster before beginning of upgrade.
• 50 Primary Indexes with 1 Replica each. (Total 100 Indexes)

Upgrade Process

• SwapRebalance Upgrade to update Couchbase Server from 7.2.5 to 7.6.1.
• Continuous query and data workload on the buckets during the update process.
• Around 40-50% CPU load on all servers during the upgrade.

Issue

• 6 KV Nodes got upgraded successfully without any issues.
• 2 Query-Index nodes got upgraded successfully.
• While, the 3rd Query-Index node was being upgraded, the index rebalances kept failing repeatedly. The node was going up to 90-94% on RAM usage.
• This was happening for a few hours.
• The upgrade was not successful.

Logs

Before Upgrade

http://supportal.couchbase.com/snapshot/fbbf7ff7fc529a2a3db10b55b3dafc84::0

2024-05-20_cb_7.2.5_swap_rebalance_cbopinfo-20240520T144749+0530.tar.gz

During Upgrade when Index Rebalances Failed

http://supportal.couchbase.com/snapshot/fbbf7ff7fc529a2a3db10b55b3dafc84::1

2024-05-20_cb_7.6.1_during_swap_rebalance_cbopinfo-20240520T234427+0530.tar.gz

Screenshots

State of Primary Indexes when the 3rd Query-Index node was being updated.

State of the CB Nodes when the 3rd Query-Index node was being updated.