Critical Description
TestResizeClusterWithBucket
If a rebalance is attempted and the first attempt fails, it will be retried again. Once the rebalance successfully completes, only a rebalance incomplete event is seen. It would make more sense to report rebalance complete in the scenario and only report rebalance incomplete in scenarios where all retries have been exhausted. Otherwise, tests will fail like this:
Expected events to be:
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0000 added to cluster |
Type: Normal | Reason: BucketCreated | Message: A new bucket `default` was created |
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0001 added to cluster |
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0002 added to cluster |
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-mjwqv-0002 removed from the cluster |
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-mjwqv-0001 removed from the cluster |
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
|
but got:
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0000 added to cluster |
Type: Normal | Reason: BucketCreated | Message: A new bucket `default` was created |
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0001 added to cluster |
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0002 added to cluster |
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-mjwqv-0002 removed from the cluster |
Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
|
Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
|
Type: Normal | Reason: RebalanceIncomplete | Message: A rebalance is incomplete
|
Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-mjwqv-0001 removed from the cluster |
This looks like rebalanced failed, but the cluster status before the events are checked is "balanced", ""healthy", and "ready".
Attachments
| For Gerrit Dashboard: K8S-543 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 98485,12 | K8S-543: Add debug level log messages for HTTP requests/responses | master | gocbmgr | Status: ABANDONED | -2 | +1 |
| 98583,6 | K8S-543: Check to see if the rebalance status is stale or timed out | master | gocbmgr | Status: MERGED | +2 | +1 |
| 98584,5 | K8S-543: Remove stale status check when checking the rebalance status | master | gocbmgr | Status: MERGED | +2 | +1 |
| 98591,2 | K8S-543: Update gocbmgr to pick up rebalance status fixes | master | couchbase-operator | Status: MERGED | +2 | +1 |