[K8S-544] events being returned in different order than operator reports - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Duplicate
Affects Version/s: None
Fix Version/s: 1.0.0
Component/s: None
Labels:
None

Description

Is some tests we kill 3 nodes and wait for down events for each of the three nodes in parallel, returning the order in which the events are seen. However, the operator will report a different ordering. In this example, nodes 2, 5, and 8 are killed in that order and wait for gorountine is spun up for each of these nodes, listening for down event. The goroutines report the order as 2, 8, then 5, but then the operator says the order was 2, 5, 8:

    util.go:187: Expected events to be:

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0000 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0001 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0002 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0003 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0004 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0005 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0006 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0007 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0008 added to cluster

    Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster

    Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed

    Type: Normal | Reason: BucketCreated | Message: A new bucket `testBucket` was created

    Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0002 down

    Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0008 down

    Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0005 down

    Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0008 failed over

    Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0002 failed over

    Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0005 failed over

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0009 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0010 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0011 added to cluster

    Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster

    Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0002 removed from the cluster

    Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0005 removed from the cluster

    Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0008 removed from the cluster

    Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed

    but got:

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0000 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0001 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0002 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0003 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0004 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0005 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0006 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0007 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0008 added to cluster

    Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster

    Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed

    Type: Normal | Reason: BucketCreated | Message: A new bucket `testBucket` was created

    Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0002 down

    Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0005 down

    Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0008 down

    Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0002 failed over

    Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0005 failed over

    Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0008 failed over

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0009 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0010 added to cluster

    Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0011 added to cluster

    Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster

    Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0002 removed from the cluster

    Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0005 removed from the cluster

    Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0008 removed from the cluster

    Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

cbinfo-default-test-couchbase-btf47-0000-20180815T220910-0700.zip
1.19 MB
16/Aug/18 10:48 PM
cbinfo-default-test-couchbase-btf47-0001-20180815T220910-0700.zip
1.06 MB
16/Aug/18 10:48 PM
cbinfo-default-test-couchbase-btf47-0003-20180815T220910-0700.zip
1.04 MB
16/Aug/18 10:49 PM
cbinfo-default-test-couchbase-btf47-0004-20180815T220910-0700.zip
1.04 MB
16/Aug/18 10:48 PM
cbinfo-default-test-couchbase-btf47-0006-20180815T220910-0700.zip
1.02 MB
16/Aug/18 10:49 PM
cbinfo-default-test-couchbase-btf47-0007-20180815T220910-0700.zip
1.01 MB
16/Aug/18 10:48 PM
cbopinfo-20180815T220909-0700.tar.gz
28 kB
16/Aug/18 10:48 PM
cbopinfo-20180815T220910-0700.tar.gz
33 kB
16/Aug/18 10:48 PM

Issue Links

duplicates

K8S-531 Couchbase operator event order is sometimes inconsistent

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Korrigan Clark created issue - 16/Aug/18 3:04 PM

Mike Wiederhold [X] (Inactive) made changes - 16/Aug/18 8:46 PM

Field	Original Value	New Value
Assignee	Mike Wiederhold [ mikew ]	Korrigan Clark [ korrigan.clark ]

Mike Wiederhold [X] (Inactive) made changes - 16/Aug/18 8:46 PM

Fix Version/s

1.0.0 [ 15036 ]

Mike Wiederhold [X] (Inactive) made changes - 16/Aug/18 10:47 PM

Comment

[ I need logs for this issue as well as the name of the test case because it's not clear that this is an operator issue. ]

Mike Wiederhold [X] (Inactive) made changes - 16/Aug/18 10:49 PM

Attachment		cbinfo-default-test-couchbase-btf47-0000-20180815T220910-0700.zip [ 56436 ]
Attachment		cbinfo-default-test-couchbase-btf47-0001-20180815T220910-0700.zip [ 56437 ]
Attachment		cbinfo-default-test-couchbase-btf47-0003-20180815T220910-0700.zip [ 56438 ]
Attachment		cbinfo-default-test-couchbase-btf47-0004-20180815T220910-0700.zip [ 56439 ]
Attachment		cbinfo-default-test-couchbase-btf47-0006-20180815T220910-0700.zip [ 56440 ]
Attachment		cbinfo-default-test-couchbase-btf47-0007-20180815T220910-0700.zip [ 56441 ]
Attachment		cbopinfo-20180815T220909-0700.tar.gz [ 56442 ]
Attachment		cbopinfo-20180815T220910-0700.tar.gz [ 56443 ]

Mike Wiederhold [X] (Inactive) made changes - 16/Aug/18 11:20 PM

Link

This issue duplicates ~~K8S-531~~ [ ~~K8S-531~~ ]

Mike Wiederhold [X] (Inactive) made changes - 16/Aug/18 11:20 PM

Resolution		Duplicate [ 3 ]
Status	Open [ 1 ]	Resolved [ 5 ]

Ashwin Govindarajulu added a comment - 28/Aug/18 2:59 AM

Closing since this is tracked in ~~K8S-531~~

Ashwin Govindarajulu added a comment - 28/Aug/18 2:59 AM Closing since this is tracked in K8S-531

Ashwin Govindarajulu made changes - 28/Aug/18 2:59 AM

Status

Resolved [ 5 ]

Closed [ 6 ]

People

Assignee:: Korrigan Clark

Reporter:: Korrigan Clark

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 16/Aug/18 3:04 PM

Updated:: 28/Aug/18 2:59 AM

Resolved:: 16/Aug/18 11:20 PM

Gerrit Reviews

There are no open Gerrit changes

PagerDuty

Couchbase Kubernetes

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty