Details
-
Bug
-
Resolution: Done
-
Major
-
1.0.0
Description
TestCase: TestMultiNodeAutoFailover
Scenario:
- Spawn 9 node cluster
- Kill nodes 0002, 0003, 0004 so that multinode failover to happen
In the couchbase-cluster logs it says it failed over the node 0003.
But operator events didn't catch that event.
#3604: [user:info,2018-08-07T01:04:45.306Z,ns_1@test-couchbase-r5j6p-0000.test-couchbase-r5j6p.default.svc:<0.709.0>:auto_failover:log_failover_success:561]Node ('ns_1@test-couchbase-r5j6p-0003.test-couchbas e-r5j6p.default.svc') was automatically failed over. Reason: All monitors report node is unhealthy.
|
Operator events:
util.go:189: Expected events to be:
|
Type: Normal | Reason: ServiceCreated | Message: Service for admin console `test-couchbase-r5j6p-ui` was created
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0000 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0001 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0002 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0003 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0004 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0005 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0006 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0007 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0008 added to cluster
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
Type: Normal | Reason: BucketCreated | Message: A new bucket `default` was created
|
Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0002 down
|
Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0003 down
|
Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0004 down
|
Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-r5j6p-0002 failed over
|
Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-r5j6p-0003 failed over
|
Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-r5j6p-0004 failed over
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0009 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0010 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0011 added to cluster
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-r5j6p-0002 removed from the cluster
|
Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-r5j6p-0003 removed from the cluster
|
Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-r5j6p-0004 removed from the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
|
but got:
|
Type: Normal | Reason: ServiceCreated | Message: Service for admin console `test-couchbase-r5j6p-ui` was created
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0000 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0001 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0002 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0003 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0004 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0005 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0006 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0007 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0008 added to cluster
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
Type: Normal | Reason: BucketCreated | Message: A new bucket `default` was created
|
Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0002 down
|
Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0003 down
|
Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0004 down
|
Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-r5j6p-0002 failed over
|
Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-r5j6p-0004 failed over
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0009 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0010 added to cluster
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0011 added to cluster
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: FailedAddNode | Message: Removed existing member test-couchbase-r5j6p-0003 because it failed before it could be added to the cluster
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-r5j6p-0002 removed from the cluster
|
Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-r5j6p-0004 removed from the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|