Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-599

PV: New pod is created when cb pod is killed with persistent volume in healthy condition

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 1.1.0
    • operator

    Description

       

      Testcase: TestPersistentVolumeKillPodAndOperator

      Scenario:

      1. Couchbase cluster of size 4 is created with default persistent volume mount
      2. Operator and pod 0001 is killed in parallel

      Observation:

      Once operator respawned, it is creating a new pod 0004 to replace the killed pod without attempting delta recovery of pod 0001

      Cluster events:

      Event schema validation failed:
      NewMemberAdded     | New member test-couchbase-54gn7-0000 added to cluster              
      NewMemberAdded     | New member test-couchbase-54gn7-0001 added to cluster              
      NewMemberAdded     | New member test-couchbase-54gn7-0002 added to cluster 
      NewMemberAdded     | New member test-couchbase-54gn7-0003 added to cluster 
      RebalanceStarted   | A rebalance has been started to balance data across the cluster
      RebalanceCompleted | A rebalance has completed          
      BucketCreated      | A new bucket `PVBucket` was created           
      NewMemberAdded     | New member test-couchbase-54gn7-0004 added to cluster              | <== no anyof members matched
      RebalanceStarted   | A rebalance has been started to balance data across the cluster
      MemberRemoved      | Existing member test-couchbase-54gn7-0001 removed from the cluster
      RebalanceCompleted | A rebalance has completed            

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          In fact we should probably let updateMembers add every node we get from ClusterInfo.Nodes to c.members since they would have all been members prior to operator crash.  The operator would just continue to reconcile members based on their state.

           

          tommie Tommie McAfee added a comment - In fact we should probably let updateMembers add every node we get from ClusterInfo.Nodes to c.members since they would have all been members prior to operator crash.  The operator would just continue to reconcile members based on their state.  

          Looks like the node was auto-failedover between the time the operator was killed and begun it's next reconcile.  (30s)

          [user:info,2018-09-25T23:55:49.926Z,ns_1@test-couchbase-54gn7-0000.test-couchbase-54gn7.default.svc:<0.715.0>:auto_failover:log_failover_success:561]Node ('ns_1@test-couchbase-54gn7-0001.test-couchbase-54gn7.default.svc') was automatically failed over. Reason: All monitors report node is unhealthy. 

           

          Since member is not yet rebalance out it is in FailedNode state, which leads to issue here since we don't recognize nodes in that state as valid members.

          func (c *Cluster) updateMembers(known couchbaseutil.MemberSet) error {
            ...
            members := couchbaseutil.MemberSet{}
            members.Append(status.ActiveNodes)
            members.Append(status.PendingAddNodes)
            members.Append(status.FailedAddNodes)
            members.Append(status.DownNodes) 

           

          And when node is omitted from cluster members the operator identifies it as unmanaged, deletes it, and creates a new node.

          time="2018-09-25T23:55:54Z" level=info msg="unmanaged nodes: test-couchbase-54gn7-0001.test-couchbase-54gn7.default.svc:8091" cluster-name=test-couchbase-54gn7 module=cluster 

           

          So one way to fix this is to add FailedNodes when we call updateMembers.  I can't think of any issues with that.

           

          tommie Tommie McAfee added a comment - Looks like the node was auto-failedover between the time the operator was killed and begun it's next reconcile.  (30s) [user:info, 2018 - 09 -25T23: 55 : 49 .926Z,ns_1 @test -couchbase-54gn7- 0000 .test-couchbase-54gn7. default .svc:< 0.715 . 0 >:auto_failover:log_failover_success: 561 ]Node ( 'ns_1@test-couchbase-54gn7-0001.test-couchbase-54gn7.default.svc' ) was automatically failed over. Reason: All monitors report node is unhealthy.   Since member is not yet rebalance out it is in FailedNode state, which leads to issue here since we don't recognize nodes in that state as valid members. func (c *Cluster) updateMembers(known couchbaseutil.MemberSet) error { ... members := couchbaseutil.MemberSet{}   members.Append(status.ActiveNodes)   members.Append(status.PendingAddNodes)   members.Append(status.FailedAddNodes)   members.Append(status.DownNodes)   And when node is omitted from cluster members the operator identifies it as unmanaged, deletes it, and creates a new node. time= "2018-09-25T23:55:54Z" level=info msg= "unmanaged nodes: test-couchbase-54gn7-0001.test-couchbase-54gn7.default.svc:8091" cluster-name=test-couchbase-54gn7 module=cluster   So one way to fix this is to add FailedNodes when we call updateMembers.  I can't think of any issues with that.  
          simon.murray Simon Murray added a comment -

          Recreate locally:

          time="2018-09-26T09:40:01Z" level=info msg="Node status:" cluster-name=cb-example module=cluster
          time="2018-09-26T09:40:01Z" level=info msg="┌─────────────────┬──────────────┬────────────────┐" cluster-name=cb-example module=cluster
          time="2018-09-26T09:40:01Z" level=info msg="│ Server          │ Class        │ Status         │" cluster-name=cb-example module=cluster
          time="2018-09-26T09:40:01Z" level=info msg="├─────────────────┼──────────────┼────────────────┤" cluster-name=cb-example module=cluster
          time="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0000 │ all_services │ managed+active │" cluster-name=cb-example module=cluster
          time="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0001 │ all_services │ managed+active │" cluster-name=cb-example module=cluster
          time="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0002 │ all_services │ managed+down   │" cluster-name=cb-example module=cluster
          time="2018-09-26T09:40:01Z" level=info msg="└─────────────────┴──────────────┴────────────────┘" cluster-name=cb-example module=cluster

          All running pods and PVCs are correctly aggregated (in updateMembers()) and the state is correct.  There is an entry for:

          persistentvolumeclaim/pvc-couchbase-test-couchbase-54gn7-0001-00-default/pvc-couchbase-test-couchbase-54gn7-0001-00-default.yaml

          In the cbopinfo collection and the labels look correct.

          Tommie McAfee anything jump out at you?  Seems test-couchbase-54gn7-0001 isn't getting picked up somehow on restart, and I'm not seeing this behaviour in minikube

          simon.murray Simon Murray added a comment - Recreate locally: time="2018-09-26T09:40:01Z" level=info msg="Node status:" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="┌─────────────────┬──────────────┬────────────────┐" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="│ Server │ Class │ Status │" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="├─────────────────┼──────────────┼────────────────┤" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0000 │ all_services │ managed+active │" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0001 │ all_services │ managed+active │" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0002 │ all_services │ managed+down │" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="└─────────────────┴──────────────┴────────────────┘" cluster-name=cb-example module=cluster All running pods and PVCs are correctly aggregated (in updateMembers()) and the state is correct.  There is an entry for: persistentvolumeclaim/pvc-couchbase-test-couchbase-54gn7-0001-00-default/pvc-couchbase-test-couchbase-54gn7-0001-00-default.yaml In the cbopinfo collection and the labels look correct. Tommie McAfee anything jump out at you?  Seems test-couchbase-54gn7-0001 isn't getting picked up somehow on restart, and I'm not seeing this behaviour in minikube

          People

            tommie Tommie McAfee
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty