Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-580

Volumes may be deleted when a recovery Pod fails to start

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 1.1.0
    • operator
    • None

    Description

      When a Pod is deleted and auto-failed over we create a new Pod and perform delta recovery.  However if new Pod fails to be created then the operator proceeds to delete the recovery Pod along with the recovery volumes.   

      For example here, new pod (cb-example2-0002) hit a context deadline

       time="2018-09-17T19:54:51Z" level=info msg="An auto-failover has taken place" cluster-name=cb-example2 module=cluster
      time="2018-09-17T19:54:51Z" level=info msg="Creating a pod (cb-example2-0002) running Couchbase enterprise-5.5.1" cluster-n
      ame=cb-example2 module=cluster
      time="2018-09-17T19:56:18Z" level=info msg="skip killing pod: probability: 0.5, got p: 0.6384366862599459" module=chaos
      time="2018-09-17T19:56:52Z" level=error msg="node http://cb-example2-0002.cb-example2.default.svc:8091 could not be recovered: context deadline exceeded" cluster-name=cb-example2 module=cluster
      time="2018-09-17T19:56:52Z" level=info msg="planning removal of http://cb-example2-0002.cb-example2.default.svc:8091" cluster-name=cb-example2 module=cluster
      time="2018-09-17T19:56:52Z" level=info msg="Creating a pod (cb-example2-0005) running Couchbase enterprise-5.5.1" cluster-name=cb-example2 module=cluster
      time="2018-09-17T19:57:34Z" level=info msg="added member (cb-example2-0005)" cluster-name=cb-example2 module=cluster
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          There are no commits yet on this issue.

          People

            tommie Tommie McAfee
            tommie Tommie McAfee
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty