Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-622

LogPV for ephemeral pod: Operator fails to recover the cluster when operator pod deleted with server pod

    XMLWordPrintable

Details

    Description

      Scenario:

      1. Deploy couchbase cluster with 2 service classes and 2nd class with log pv defined
      2. Kill service_config_2's pod along with operator pod

      Operator restarts successfully and detects node 0002 is down. But it prints cannot manage node and it is struck in the same state.

      Pod & PV status

      couchbase-operator]$ kubectl get pods ; kubectl get pv | grep logs
      NAME                                  READY     STATUS    RESTARTS   AGE
      couchbase-operator-768875db94-dnwk7   1/1       Running   0          39m
      test-couchbase-zr8h9-0000             1/1       Running   0          41m
      test-couchbase-zr8h9-0001             1/1       Running   0          41m
      test-couchbase-zr8h9-0003             1/1       Running   0          40m
      test-couchbase-zr8h9-0004             1/1       Running   0          40m
      2:pvc-50a6a04d-cbac-11e8-98c3-080027ee3776   2Gi        RWO            Delete           Bound     ashwin/pvc-couchbase-log-pv-test-couchbase-zr8h9-0002-00-logs     standard                 41m
      3:pvc-5987d222-cbac-11e8-98c3-080027ee3776   2Gi        RWO            Delete           Bound     ashwin/pvc-couchbase-log-pv-test-couchbase-zr8h9-0003-00-logs     standard                 40m
      4:pvc-623998ba-cbac-11e8-98c3-080027ee3776   2Gi        RWO            Delete           Bound     ashwin/pvc-couchbase-log-pv-test-couchbase-zr8h9-0004-00-logs     standard                 40m

      Operator error prints:

       is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      time="2018-10-09T10:50:18Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      time="2018-10-09T10:50:26Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      time="2018-10-09T10:50:34Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      time="2018-10-09T10:50:42Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      time="2018-10-09T10:50:50Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      time="2018-10-09T10:50:58Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      time="2018-10-09T10:51:06Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      time="2018-10-09T10:51:14Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      time="2018-10-09T10:51:22Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            I think this would be better suited to Tommie as he wrote the code, throw it back over the fence if you want to

            simon.murray Simon Murray added a comment - I think this would be better suited to Tommie as he wrote the code, throw it back over the fence if you want to

            This is expected behavior.   What's happening is the cluster cannot auto-failover because the 2 nodes were not data.  So you're in a position where manual failover is required since the operator will not force remove the nodes from cluster without user permission.  Further, when operator is deleted it starts up and still sees that cluster is irreconcilable and continues to report "Failover/Rebalance is recommended"

            To continue reconcile, you can use gocbmgr to trigger failover.  

            tommie Tommie McAfee added a comment - This is expected behavior.   What's happening is the cluster cannot auto-failover because the 2 nodes were not data.  So you're in a position where manual failover is required since the operator will not force remove the nodes from cluster without user permission.  Further, when operator is deleted it starts up and still sees that cluster is irreconcilable and continues to report "Failover/Rebalance is recommended" To continue reconcile, you can use gocbmgr to trigger failover.  

            Tommie McAfee but the same scenario works fine if the operator pod is not killed.

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Tommie McAfee but the same scenario works fine if the operator pod is not killed.

            This is expected because when the operator is killed we cannot gather state to recreate members.  If we decide to push change for config maps then the behavior will be same but this isn't for 1.1:  http://review.couchbase.org/#/c/100098/

             

            tommie Tommie McAfee added a comment - This is expected because when the operator is killed we cannot gather state to recreate members.  If we decide to push change for config maps then the behavior will be same but this isn't for 1.1:  http://review.couchbase.org/#/c/100098/  

            1.1 Release Note:

            Clusters without Persistent Volumes require manual failover when nodes are down and auto-failover cannot be performed.  In the event that the operator also crashes before nodes have been failed-over, the operator will require manual failover of down nodes before proceeding to reconcile any changes made to cluster.   

            tommie Tommie McAfee added a comment - 1.1 Release Note: Clusters without Persistent Volumes require manual failover when nodes are down and auto-failover cannot be performed.  In the event that the operator also crashes before nodes have been failed-over, the operator will require manual failover of down nodes before proceeding to reconcile any changes made to cluster.   

            Description for release notes:

            Summary: Known Issue Clusters without persistent volumes require manual failover when nodes are down and auto-failover cannot be performed. In the event that the Operator also crashes before nodes have been failed over, the Operator will require manual failover of down nodes before proceeding to reconcile any changes made to the cluster.

            eric.schneider Eric Schneider (Inactive) added a comment - - edited Description for release notes: Summary: Known Issue Clusters without persistent volumes require manual failover when nodes are down and auto-failover cannot be performed. In the event that the Operator also crashes before nodes have been failed over, the Operator will require manual failover of down nodes before proceeding to reconcile any changes made to the cluster.

            Note to myself - for 1.2, the 1 case where the behavior will be different is when the cluster is actually capable of performing an autofailover but the operator happens to die in the process.  This is because the configmap change will allow us to recognize the Pod as ours after restart.

            tommie Tommie McAfee added a comment - Note to myself - for 1.2, the 1 case where the behavior will be different is when the cluster is actually capable of performing an autofailover but the operator happens to die in the process.  This is because the configmap change will allow us to recognize the Pod as ours after restart.
            simon.murray Simon Murray added a comment -

            Moving to backlog for now.  If sindhu or matt have issues when trying upgrades in testing then we can re-prioritize

            simon.murray Simon Murray added a comment - Moving to backlog for now.  If sindhu or matt have issues when trying upgrades in testing then we can re-prioritize

            People

              tommie Tommie McAfee
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty