Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.0.0
Description
Testcase: TestPersistentVolumeKillAllPods
Scenario:
- Created 4 node Couchbase cluster with PVC defined for all nodes
- Killing all couchbase-server pods
- All pods are recovered back and cluster is rebalanced.
- But recovery event is got only for 2 pods instead of 3 events
Events:
|
Type Reason Age From Message
|
---- ------ ---- ---- -------
|
Normal NewMemberAdded 10m couchbase-operator-585d4b675d-j74s4 New member test-couchbase-xrlsc-0000 added to cluster
|
Normal NewMemberAdded 10m couchbase-operator-585d4b675d-j74s4 New member test-couchbase-xrlsc-0001 added to cluster
|
Normal NewMemberAdded 9m couchbase-operator-585d4b675d-j74s4 New member test-couchbase-xrlsc-0002 added to cluster
|
Normal NewMemberAdded 8m couchbase-operator-585d4b675d-j74s4 New member test-couchbase-xrlsc-0003 added to cluster
|
Normal RebalanceStarted 8m couchbase-operator-585d4b675d-j74s4 A rebalance has been started to balance data across the cluster
|
Normal RebalanceCompleted 8m couchbase-operator-585d4b675d-j74s4 A rebalance has completed
|
Normal BucketCreated 8m couchbase-operator-585d4b675d-j74s4 A new bucket `PVBucket` was created
|
Warning MemberDown 6m (x7 over 7m) couchbase-operator-585d4b675d-j74s4 Existing member test-couchbase-xrlsc-0002 down
|
Warning MemberDown 6m (x8 over 7m) couchbase-operator-585d4b675d-j74s4 Existing member test-couchbase-xrlsc-0000 down
|
Normal MemberRecovered 5m couchbase-operator-585d4b675d-j74s4 Existing member test-couchbase-xrlsc-0000 recovered
|
Warning MemberDown 5m (x8 over 7m) couchbase-operator-585d4b675d-j74s4 Existing member test-couchbase-xrlsc-0003 down
|
Normal MemberRecovered 5m couchbase-operator-585d4b675d-j74s4 Existing member test-couchbase-xrlsc-0003 recovered
|
Normal RebalanceStarted 3m couchbase-operator-585d4b675d-j74s4 A rebalance has been started to balance data across the cluster
|
Normal RebalanceCompleted 3m couchbase-operator-585d4b675d-j74s4 A rebalance has completed
|
Mike Wiederhold [X] please find the attached log file cbopinfo-20180817T171141+0000.tar.gz
Also in this case, Cluster is reported as balanced condition in cluster description. But there is no "RebalanceComplete" event in the event list.
NAME READY STATUS RESTARTS AGE
couchbase-operator-56699895c4-2k826 1/1 Running 0 11m
docker-registry-1-9fznr 1/1 Running 1 10d
registry-console-1-jcn85 1/1 Running 1 10d
router-1-mkhbw 1/1 Running 1 10d
test-couchbase-gfs6b-0000 1/1 Running 0 7m
test-couchbase-gfs6b-0001 1/1 Running 0 6m
test-couchbase-gfs6b-0002 1/1 Running 0 5m
test-couchbase-gfs6b-0003 1/1 Running 0 4m
Status:
Buckets:
PV Bucket:
Conflict Resolution: seqno
Enable Flush: true
Eviction Policy: fullEviction
Io Priority: high
Memory Quota: 100
Name: PVBucket
Replicas: 2
Type: couchbase
Cluster Id: 763e6e473071d61afb4ede1b26cee19b
Conditions:
Available:
Last Transition Time: 2018-08-17T17:11:19Z
Last Update Time: 2018-08-17T17:11:19Z
Reason: Cluster available
Status: True
Balanced:
Last Transition Time: 2018-08-17T17:11:40Z
Last Update Time: 2018-08-17T17:11:40Z
Message: Data is equally distributed across all nodes in the cluster
Reason: Cluster is balanced
Status: True
Control Paused: false
Current Version: enterprise-5.5.0
Members:
Index: 4
Ready:
Name: test-couchbase-gfs6b-0000
Name: test-couchbase-gfs6b-0001
Name: test-couchbase-gfs6b-0002
Name: test-couchbase-gfs6b-0003
Phase: Running
Reason:
Size: 4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NewMemberAdded 10m couchbase-operator-56699895c4-2k826 New member test-couchbase-gfs6b-0000 added to cluster
Normal NewMemberAdded 9m couchbase-operator-56699895c4-2k826 New member test-couchbase-gfs6b-0001 added to cluster
Normal NewMemberAdded 9m couchbase-operator-56699895c4-2k826 New member test-couchbase-gfs6b-0002 added to cluster
Normal NewMemberAdded 8m couchbase-operator-56699895c4-2k826 New member test-couchbase-gfs6b-0003 added to cluster
Normal RebalanceStarted 8m couchbase-operator-56699895c4-2k826 A rebalance has been started to balance data across the cluster
Normal RebalanceCompleted 8m couchbase-operator-56699895c4-2k826 A rebalance has completed
Normal BucketCreated 8m couchbase-operator-56699895c4-2k826 A new bucket `PVBucket` was created
Warning MemberDown 6m (x7 over 7m) couchbase-operator-56699895c4-2k826 Existing member test-couchbase-gfs6b-0003 down
Warning MemberDown 6m (x8 over 7m) couchbase-operator-56699895c4-2k826 Existing member test-couchbase-gfs6b-0001 down
Normal MemberRecovered 5m couchbase-operator-56699895c4-2k826 Existing member test-couchbase-gfs6b-0001 recovered
Warning MemberDown 5m (x8 over 7m) couchbase-operator-56699895c4-2k826 Existing member test-couchbase-gfs6b-0002 down
Normal MemberRecovered 4m couchbase-operator-56699895c4-2k826 Existing member test-couchbase-gfs6b-0002 recovered
Normal RebalanceStarted 3m couchbase-operator-56699895c4-2k826 A rebalance has been started to balance data across the cluster
Normal RebalanceIncomplete 3m couchbase-operator-56699895c4-2k826 A rebalance is incomplete