Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.0.0
Description
Testcase: TestPersistentVolumeKillAllPods
Scenario:
- Created 4 node Couchbase cluster with PVC defined for all nodes
- Killing all couchbase-server pods
- All pods are recovered back and cluster is rebalanced.
- But recovery event is got only for 2 pods instead of 3 events
Events:
|
Type Reason Age From Message
|
---- ------ ---- ---- -------
|
Normal NewMemberAdded 10m couchbase-operator-585d4b675d-j74s4 New member test-couchbase-xrlsc-0000 added to cluster
|
Normal NewMemberAdded 10m couchbase-operator-585d4b675d-j74s4 New member test-couchbase-xrlsc-0001 added to cluster
|
Normal NewMemberAdded 9m couchbase-operator-585d4b675d-j74s4 New member test-couchbase-xrlsc-0002 added to cluster
|
Normal NewMemberAdded 8m couchbase-operator-585d4b675d-j74s4 New member test-couchbase-xrlsc-0003 added to cluster
|
Normal RebalanceStarted 8m couchbase-operator-585d4b675d-j74s4 A rebalance has been started to balance data across the cluster
|
Normal RebalanceCompleted 8m couchbase-operator-585d4b675d-j74s4 A rebalance has completed
|
Normal BucketCreated 8m couchbase-operator-585d4b675d-j74s4 A new bucket `PVBucket` was created
|
Warning MemberDown 6m (x7 over 7m) couchbase-operator-585d4b675d-j74s4 Existing member test-couchbase-xrlsc-0002 down
|
Warning MemberDown 6m (x8 over 7m) couchbase-operator-585d4b675d-j74s4 Existing member test-couchbase-xrlsc-0000 down
|
Normal MemberRecovered 5m couchbase-operator-585d4b675d-j74s4 Existing member test-couchbase-xrlsc-0000 recovered
|
Warning MemberDown 5m (x8 over 7m) couchbase-operator-585d4b675d-j74s4 Existing member test-couchbase-xrlsc-0003 down
|
Normal MemberRecovered 5m couchbase-operator-585d4b675d-j74s4 Existing member test-couchbase-xrlsc-0003 recovered
|
Normal RebalanceStarted 3m couchbase-operator-585d4b675d-j74s4 A rebalance has been started to balance data across the cluster
|
Normal RebalanceCompleted 3m couchbase-operator-585d4b675d-j74s4 A rebalance has completed
|
Attachments
Issue Links
| For Gerrit Dashboard: K8S-537 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 98722,2 | K8S-537: Raise recovery event after cluster down | master | couchbase-operator | Status: MERGED | +2 | +1 |
Activity
Ashwin,
Please re-run this test a few times and let me know if you see this behavior again. If you don't close this issue and we will address events being dropped by fixing K8S-541 in 1.1.0.
Mike Wiederhold [X], I reran with our RC build.
Still was able to reproduce the same issue on that aswell.
Mike Wiederhold [X] added new logs -> cbopinfo-20180816T173921+0000.tar.gz
Looks like the logs attached does not have required files in that.
I will rerun and share the logs with you MIke
Mike Wiederhold [X] please find the attached log file cbopinfo-20180817T171141+0000.tar.gz
Also in this case, Cluster is reported as balanced condition in cluster description. But there is no "RebalanceComplete" event in the event list.
NAME READY STATUS RESTARTS AGE
|
couchbase-operator-56699895c4-2k826 1/1 Running 0 11m
|
docker-registry-1-9fznr 1/1 Running 1 10d
|
registry-console-1-jcn85 1/1 Running 1 10d
|
router-1-mkhbw 1/1 Running 1 10d
|
test-couchbase-gfs6b-0000 1/1 Running 0 7m
|
test-couchbase-gfs6b-0001 1/1 Running 0 6m
|
test-couchbase-gfs6b-0002 1/1 Running 0 5m
|
test-couchbase-gfs6b-0003 1/1 Running 0 4m
|
|
|
|
|
Status:
|
Buckets:
|
PV Bucket:
|
Conflict Resolution: seqno
|
Enable Flush: true
|
Eviction Policy: fullEviction
|
Io Priority: high
|
Memory Quota: 100
|
Name: PVBucket
|
Replicas: 2
|
Type: couchbase
|
Cluster Id: 763e6e473071d61afb4ede1b26cee19b
|
Conditions:
|
Available:
|
Last Transition Time: 2018-08-17T17:11:19Z
|
Last Update Time: 2018-08-17T17:11:19Z
|
Reason: Cluster available
|
Status: True
|
Balanced:
|
Last Transition Time: 2018-08-17T17:11:40Z
|
Last Update Time: 2018-08-17T17:11:40Z
|
Message: Data is equally distributed across all nodes in the cluster
|
Reason: Cluster is balanced
|
Status: True
|
Control Paused: false
|
Current Version: enterprise-5.5.0
|
Members:
|
Index: 4
|
Ready:
|
Name: test-couchbase-gfs6b-0000
|
Name: test-couchbase-gfs6b-0001
|
Name: test-couchbase-gfs6b-0002
|
Name: test-couchbase-gfs6b-0003
|
Phase: Running
|
Reason:
|
Size: 4
|
|
|
Events:
|
Type Reason Age From Message
|
---- ------ ---- ---- -------
|
Normal NewMemberAdded 10m couchbase-operator-56699895c4-2k826 New member test-couchbase-gfs6b-0000 added to cluster
|
Normal NewMemberAdded 9m couchbase-operator-56699895c4-2k826 New member test-couchbase-gfs6b-0001 added to cluster
|
Normal NewMemberAdded 9m couchbase-operator-56699895c4-2k826 New member test-couchbase-gfs6b-0002 added to cluster
|
Normal NewMemberAdded 8m couchbase-operator-56699895c4-2k826 New member test-couchbase-gfs6b-0003 added to cluster
|
Normal RebalanceStarted 8m couchbase-operator-56699895c4-2k826 A rebalance has been started to balance data across the cluster
|
Normal RebalanceCompleted 8m couchbase-operator-56699895c4-2k826 A rebalance has completed
|
Normal BucketCreated 8m couchbase-operator-56699895c4-2k826 A new bucket `PVBucket` was created
|
Warning MemberDown 6m (x7 over 7m) couchbase-operator-56699895c4-2k826 Existing member test-couchbase-gfs6b-0003 down
|
Warning MemberDown 6m (x8 over 7m) couchbase-operator-56699895c4-2k826 Existing member test-couchbase-gfs6b-0001 down
|
Normal MemberRecovered 5m couchbase-operator-56699895c4-2k826 Existing member test-couchbase-gfs6b-0001 recovered
|
Warning MemberDown 5m (x8 over 7m) couchbase-operator-56699895c4-2k826 Existing member test-couchbase-gfs6b-0002 down
|
Normal MemberRecovered 4m couchbase-operator-56699895c4-2k826 Existing member test-couchbase-gfs6b-0002 recovered
|
Normal RebalanceStarted 3m couchbase-operator-56699895c4-2k826 A rebalance has been started to balance data across the cluster
|
Normal RebalanceIncomplete 3m couchbase-operator-56699895c4-2k826 A rebalance is incomplete
|
Moving this to 1.1.0. We are definitely raising the recovered event, but it seems to be getting lost somewhere in the Kubernetes code. This is only affecting two of the tests and the operator is doing the right things and logging the recovery. We will look into this again for the 1.1.0 release.
Mike Wiederhold [X], in test case TestPersistentVolumeWithSingleNodeService also, "VolumeUnhealthy" event is missed some times.
Type: Normal | Reason: ServiceCreated | Message: Service for admin console `test-couchbase-wc92f-ui` was created
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0000 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0001 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0002 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0003 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0004 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0005 added to cluster
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
Type: Normal | Reason: BucketCreated | Message: A new bucket `PVBucket` was created
|
Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-wc92f-0005 down
|
Type: Normal | Reason: MemberRecovered | Message: Existing member test-couchbase-wc92f-0005 recovered
|
Type: Normal | Reason: VolumeUnhealthy | Message: Member test-couchbase-wc92f-0005 volumes are unhealthy. Failover is recommended: Missing PersistentVolumeClaim for path /opt/couchbase/var/lib/couchbase
|
Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-wc92f-0005 failed over
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0006 added to cluster
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-wc92f-0005 removed from the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
|
|
but got:
|
Type: Normal | Reason: ServiceCreated | Message: Service for admin console `test-couchbase-wc92f-ui` was created
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0000 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0001 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0002 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0003 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0004 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0005 added to cluster
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
Type: Normal | Reason: BucketCreated | Message: A new bucket `PVBucket` was created
|
Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-wc92f-0005 down
|
Type: Normal | Reason: MemberRecovered | Message: Existing member test-couchbase-wc92f-0005 recovered
|
Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-wc92f-0005 failed over
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-wc92f-0006 added to cluster
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-wc92f-0005 removed from the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
I found 1 area where recovery event was raised, but I noticed events were mysteriously being dropped when I ran the scenario.
Also note the VolumeUnhealthy event is cached. If the test is deleting the same one it's possible we don't record duplicates.
Description for release notes:
Known Issue: In rare events Kubernetes seems to drop events logged by the operator.
Workaround: None.
I've filed
K8S-541to track dropping of events which can happen for various reasons. Given that the correct things are happening in terms of the cluster being rebuilt by the operator I'm not too concerned about the event being dropped, but I would like to add log messages for when this happens in the future.For 1.0.0 let's re-run this test and see if this happens frequently. Given that I can see that the code was run to raise the event I don't think we should see this issue in test re-runs.