Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-580

Volumes may be deleted when a recovery Pod fails to start

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 1.1.0
    • operator
    • None

    Description

      When a Pod is deleted and auto-failed over we create a new Pod and perform delta recovery.  However if new Pod fails to be created then the operator proceeds to delete the recovery Pod along with the recovery volumes.   

      For example here, new pod (cb-example2-0002) hit a context deadline

       time="2018-09-17T19:54:51Z" level=info msg="An auto-failover has taken place" cluster-name=cb-example2 module=cluster
      time="2018-09-17T19:54:51Z" level=info msg="Creating a pod (cb-example2-0002) running Couchbase enterprise-5.5.1" cluster-n
      ame=cb-example2 module=cluster
      time="2018-09-17T19:56:18Z" level=info msg="skip killing pod: probability: 0.5, got p: 0.6384366862599459" module=chaos
      time="2018-09-17T19:56:52Z" level=error msg="node http://cb-example2-0002.cb-example2.default.svc:8091 could not be recovered: context deadline exceeded" cluster-name=cb-example2 module=cluster
      time="2018-09-17T19:56:52Z" level=info msg="planning removal of http://cb-example2-0002.cb-example2.default.svc:8091" cluster-name=cb-example2 module=cluster
      time="2018-09-17T19:56:52Z" level=info msg="Creating a pod (cb-example2-0005) running Couchbase enterprise-5.5.1" cluster-name=cb-example2 module=cluster
      time="2018-09-17T19:57:34Z" level=info msg="added member (cb-example2-0005)" cluster-name=cb-example2 module=cluster
      

      Attachments

        For Gerrit Dashboard: K8S-580
        # Subject Branch Project Status CR V

        Activity

          tommie Tommie McAfee created issue -
          tommie Tommie McAfee made changes -
          Field Original Value New Value
          Rank Ranked higher
          simon.murray Simon Murray made changes -
          Rank Ranked lower
          mikew Mike Wiederhold [X] (Inactive) made changes -
          Sprint Kubernetes 8/13-8/27 [ 549 ]
          mikew Mike Wiederhold [X] (Inactive) made changes -
          Rank Ranked higher
          tommie Tommie McAfee made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Closed [ 6 ]
          tommie Tommie McAfee made changes -
          Sprint Kubernetes 8/13-8/27 [ 549 ]

          People

            tommie Tommie McAfee
            tommie Tommie McAfee
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty