Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-622

LogPV for ephemeral pod: Operator fails to recover the cluster when operator pod deleted with server pod

    XMLWordPrintable

Details

    Description

      Scenario:

      1. Deploy couchbase cluster with 2 service classes and 2nd class with log pv defined
      2. Kill service_config_2's pod along with operator pod

      Operator restarts successfully and detects node 0002 is down. But it prints cannot manage node and it is struck in the same state.

      Pod & PV status

      couchbase-operator]$ kubectl get pods ; kubectl get pv | grep logs
      		 
      NAME                                  READY     STATUS    RESTARTS   AGE
      		 
      couchbase-operator-768875db94-dnwk7   1/1       Running   0          39m
      		 
      test-couchbase-zr8h9-0000             1/1       Running   0          41m
      		 
      test-couchbase-zr8h9-0001             1/1       Running   0          41m
      		 
      test-couchbase-zr8h9-0003             1/1       Running   0          40m
      		 
      test-couchbase-zr8h9-0004             1/1       Running   0          40m
      		 
      2:pvc-50a6a04d-cbac-11e8-98c3-080027ee3776   2Gi        RWO            Delete           Bound     ashwin/pvc-couchbase-log-pv-test-couchbase-zr8h9-0002-00-logs     standard                 41m
      		 
      3:pvc-5987d222-cbac-11e8-98c3-080027ee3776   2Gi        RWO            Delete           Bound     ashwin/pvc-couchbase-log-pv-test-couchbase-zr8h9-0003-00-logs     standard                 40m
      		 
      4:pvc-623998ba-cbac-11e8-98c3-080027ee3776   2Gi        RWO            Delete           Bound     ashwin/pvc-couchbase-log-pv-test-couchbase-zr8h9-0004-00-logs     standard                 40m

      Operator error prints:

       is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      		 
      time="2018-10-09T10:50:18Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      		 
      time="2018-10-09T10:50:26Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      		 
      time="2018-10-09T10:50:34Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      		 
      time="2018-10-09T10:50:42Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      		 
      time="2018-10-09T10:50:50Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      		 
      time="2018-10-09T10:50:58Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      		 
      time="2018-10-09T10:51:06Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      		 
      time="2018-10-09T10:51:14Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster
      		 
      time="2018-10-09T10:51:22Z" level=error msg="failed to update members: Cluster contains node `test-couchbase-zr8h9-0002` which cannot be managed. Failover/Rebalance is recommended." cluster-name=test-couchbase-zr8h9 module=cluster

      Attachments

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. K8S-1218 Cluster Hibernation

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              tommie Tommie McAfee (Inactive)
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty