Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-537

PVC: Pod recovery/VolumeUnhealthy events are missing in the generated event list

    XMLWordPrintable

Details

    Description

       

      Testcase: TestPersistentVolumeKillAllPods

      Scenario:

      1. Created 4 node Couchbase cluster with PVC defined for all nodes
      2. Killing all couchbase-server pods
      3. All pods are recovered back and cluster is rebalanced.
      4. But recovery event is got only for 2 pods instead of 3 events

      Events:
      		 
        Type     Reason              Age              From                                 Message
      		 
        ----     ------              ----             ----                                 -------
      		 
        Normal   NewMemberAdded      10m              couchbase-operator-585d4b675d-j74s4  New member test-couchbase-xrlsc-0000 added to cluster
      		 
        Normal   NewMemberAdded      10m              couchbase-operator-585d4b675d-j74s4  New member test-couchbase-xrlsc-0001 added to cluster
      		 
        Normal   NewMemberAdded      9m               couchbase-operator-585d4b675d-j74s4  New member test-couchbase-xrlsc-0002 added to cluster
      		 
        Normal   NewMemberAdded      8m               couchbase-operator-585d4b675d-j74s4  New member test-couchbase-xrlsc-0003 added to cluster
      		 
        Normal   RebalanceStarted    8m               couchbase-operator-585d4b675d-j74s4  A rebalance has been started to balance data across the cluster
      		 
        Normal   RebalanceCompleted  8m               couchbase-operator-585d4b675d-j74s4  A rebalance has completed
      		 
        Normal   BucketCreated       8m               couchbase-operator-585d4b675d-j74s4  A new bucket `PVBucket` was created
      		 
        Warning  MemberDown          6m (x7 over 7m)  couchbase-operator-585d4b675d-j74s4  Existing member test-couchbase-xrlsc-0002 down
      		 
        Warning  MemberDown          6m (x8 over 7m)  couchbase-operator-585d4b675d-j74s4  Existing member test-couchbase-xrlsc-0000 down
      		 
        Normal   MemberRecovered     5m               couchbase-operator-585d4b675d-j74s4  Existing member test-couchbase-xrlsc-0000 recovered
      		 
        Warning  MemberDown          5m (x8 over 7m)  couchbase-operator-585d4b675d-j74s4  Existing member test-couchbase-xrlsc-0003 down
      		 
        Normal   MemberRecovered     5m               couchbase-operator-585d4b675d-j74s4  Existing member test-couchbase-xrlsc-0003 recovered
      		 
        Normal   RebalanceStarted    3m               couchbase-operator-585d4b675d-j74s4  A rebalance has been started to balance data across the cluster
      		 
        Normal   RebalanceCompleted  3m               couchbase-operator-585d4b675d-j74s4  A rebalance has completed

       

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. K8S-541 Log when events get dropped by kubernetes

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. K8S-597 Failover: MemberDown event not caught for one of the node during multinode failover

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: K8S-537
          # Subject Branch Project Status CR V
          98722,2 K8S-537: Raise recovery event after cluster down master couchbase-operator Status: MERGED +2 +1

          Activity

            Ascending order - Click to sort in descending order
            ashwin.govindarajulu Ashwin Govindarajulu created issue -
            simon.murray Simon Murray made changes -
            Field Original Value New Value
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Sprint Kubernetes 3/14-3/27 [ 549 ]
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Rank Ranked lower
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Link This issue relates to K8S-541 [ K8S-541 ]
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Assignee Mike Wiederhold [ mikew ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Sprint Kubernetes 3/14-3/27 [ 549 ]
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Rank Ranked higher
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Assignee Ashwin Govindarajulu [ ashwin.govindarajulu ] Mike Wiederhold [ mikew ]
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Assignee Mike Wiederhold [ mikew ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Attachment cbopinfo-20180816T173921+0000.tar.gz [ 56467 ]
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Assignee Ashwin Govindarajulu [ ashwin.govindarajulu ] Mike Wiederhold [ mikew ]
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Assignee Mike Wiederhold [ mikew ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Attachment cbopinfo-20180817T171141+0000.tar.gz [ 56475 ]
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Assignee Ashwin Govindarajulu [ ashwin.govindarajulu ] Mike Wiederhold [ mikew ]
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Sprint Kubernetes 3/14-3/27 [ 549 ]
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Rank Ranked lower
            dfinlay Dave Finlay made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Fix Version/s 1.1.0 [ 15163 ]
            Fix Version/s 1.0.0 [ 15036 ]
            Priority Critical [ 2 ] Major [ 3 ]
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Sprint Kubernetes 3/14-3/27 [ 549 ]
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Rank Ranked higher
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Assignee Mike Wiederhold [ mikew ] Tommie McAfee [ tommie ]
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Rank Ranked lower
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Summary PVC: Pod recovery event missing in the generated event list PVC: Pod recovery/VolumesUnhealthy events are missing in the generated event list
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Summary PVC: Pod recovery/VolumesUnhealthy events are missing in the generated event list PVC: Pod recovery/VolumeUnhealthy events are missing in the generated event list
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Labels functional-test functional-test releasenote
            mikew Mike Wiederhold [X] (Inactive) made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Link This issue relates to K8S-597 [ K8S-597 ]
            arunkumar Arunkumar Senthilnathan (Inactive) made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            arunkumar Arunkumar Senthilnathan (Inactive) made changes -
            Actual End 2018-11-05 10:32 (issue has been closed)
            arunkumar Arunkumar Senthilnathan (Inactive) made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            arunkumar Arunkumar Senthilnathan (Inactive) made changes -
            Fix Version/s 1.1.0 [ 15163 ]
            arunkumar Arunkumar Senthilnathan (Inactive) made changes -
            Fix Version/s 1.1.0 [ 15163 ]
            arunkumar Arunkumar Senthilnathan (Inactive) made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            arunkumar Arunkumar Senthilnathan (Inactive) made changes -
            Actual End 2018-11-05 10:32 2018-11-05 10:34 (issue has been resolved)

            People

              tommie Tommie McAfee
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty