See Manchester legends 10CC's Dreadlock Holiday for context!
I found this through the chaotic monkey that is GKE Autopilot, it turns out that if the Operator deployment gets rescheduled while it's waiting for a pod to get scheduled (magnified quite a lot by cluster autoscaling!!) then we end up in a situation where:
- We try to get a list of callable members
- None are working so server throws a wobbly when we call /pools/default to determine what's clustered or not
- Spin in loop of death forever
We need a mechanism to "off" uninitialized nodes. We version our resources, so we should be able to say a 2.2 pod without the pod.couchbase.com/initialized annotation can be "retired". This can happen transparently without any special configuration. It also preserves any pre 2.2 pods or initialized ones for log collection.