Major Description
When a Pod is deleted and auto-failed over we create a new Pod and perform delta recovery. However if new Pod fails to be created then the operator proceeds to delete the recovery Pod along with the recovery volumes.
For example here, new pod (cb-example2-0002) hit a context deadline
time="2018-09-17T19:54:51Z" level=info msg="An auto-failover has taken place" cluster-name=cb-example2 module=cluster |
time="2018-09-17T19:54:51Z" level=info msg="Creating a pod (cb-example2-0002) running Couchbase enterprise-5.5.1" cluster-n |
ame=cb-example2 module=cluster
|
time="2018-09-17T19:56:18Z" level=info msg="skip killing pod: probability: 0.5, got p: 0.6384366862599459" module=chaos |
time="2018-09-17T19:56:52Z" level=error msg="node http://cb-example2-0002.cb-example2.default.svc:8091 could not be recovered: context deadline exceeded" cluster-name=cb-example2 module=cluster |
time="2018-09-17T19:56:52Z" level=info msg="planning removal of http://cb-example2-0002.cb-example2.default.svc:8091" cluster-name=cb-example2 module=cluster |
time="2018-09-17T19:56:52Z" level=info msg="Creating a pod (cb-example2-0005) running Couchbase enterprise-5.5.1" cluster-name=cb-example2 module=cluster |
time="2018-09-17T19:57:34Z" level=info msg="added member (cb-example2-0005)" cluster-name=cb-example2 module=cluster |