New pod added before failed over node is ejected and cluster is stabilized

Cluster Setup

• Kind cluster locally run on Mac
• 2 nodes with kv,index,n1ql
• 1 bucket
• Initial Cluster version : 7.0.5-7659
• Upgrade Cluster version : 7.2.3-6705

Steps taken in the scenario

• Created a cluster
• Created 1 bucket
• Issued an upgrade from 7.0.5-7659 to 7.2.3-6705 using swap rebalance
• Swap rebalance fails when cb-example-0002 tries to replace cb-example-0001
• cb-example-0001 in version 7.0.5 is failed over by the operator due to memcached crash.
• The pod is recovered and is inactiveAdded state.
• Further a new pod is spun up and added to cluster - cb-example-0003
• Rebalance attempts to remove  cb-example-0001 and cb-example-0000 which is basically both failed over pod and existing pod.

Issue

• When the first swap rebalance has failed, that should be rectified and cluster should be in balanced state when new pod is added.
• Trying to add new pods to unbalanced clusters can cause multitude of failures and is not the recommended practice.

Operator logs : https://cb-engineering.s3.amazonaws.com/K8S-3590/cbopinfo-20240802T100337+0530.tar.gz

Cluster logs : 
https://cb-engineering.s3.amazonaws.com/K8S-3590/collectinfo-2024-08-02T035818-ns_1%40cb-example-0000.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3590/collectinfo-2024-08-02T035818-ns_1%40cb-example-0001.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3590/collectinfo-2024-08-02T035818-ns_1%40cb-example-0002.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/K8S-3590/collectinfo-2024-08-02T035818-ns_1%40cb-example-0003.cb-example.default.svc.zip
Cluster deployment : https://cb-engineering.s3.amazonaws.com/K8S-3590/couchbase-cluster.yaml

Cluster upgrade deployment : https://cb-engineering.s3.amazonaws.com/K8S-3590/couchbase-cluster-upgraded.yaml

Bucket deployment : https://cb-engineering.s3.amazonaws.com/K8S-3590/couchbase-buckets.yaml

 The cao tool and operator images were built locally on this commit

commit 16a1e4fe24d2a7791836b4add7a24962e3c56b8a(HEAD -> 2.7.x, origin/2.7.x)

Author: usamah jassat <usamah.jassat@couchbase.com>

Date:   Thu Aug 1 11:21:00 2024 +0100

    K8S-3602: Fix openshift tests

    Change-Id: I1f5ee7f1a2b7668cf8cfbcc04e3e5f8b2addb73f

    Reviewed-on: https://review.couchbase.org/c/couchbase-operator/+/213577

    Tested-by: Build Bot <build@couchbase.com>

    Reviewed-by: Justin Ashworth <justin.ashworth@couchbase.com>