Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-543

rebalance status loop exits early and reports rebalance as incomplete

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 1.0.0
    • None
    • operator
    • None

    Description

      TestResizeClusterWithBucket

      If a rebalance is attempted and the first attempt fails, it will be retried again. Once the rebalance successfully completes, only a rebalance incomplete event is seen. It would make more sense to report rebalance complete in the scenario and only report rebalance incomplete in scenarios where all retries have been exhausted. Otherwise, tests will fail like this:

      Expected events to be:
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0000 added to cluster
          		Type: Normal | Reason: BucketCreated | Message: A new bucket `default` was created
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0001 added to cluster
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0002 added to cluster
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-mjwqv-0002 removed from the cluster
          		Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-mjwqv-0001 removed from the cluster
          		Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
          		
          		but got:
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0000 added to cluster
          		Type: Normal | Reason: BucketCreated | Message: A new bucket `default` was created
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0001 added to cluster
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-mjwqv-0002 added to cluster
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-mjwqv-0002 removed from the cluster
          		Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: RebalanceIncomplete | Message: A rebalance is incomplete
          		Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-mjwqv-0001 removed from the cluster
      

      This looks like rebalanced failed, but the cluster status before the events are checked is "balanced", ""healthy", and "ready".

      Attachments

        Activity

          People

            mikew Mike Wiederhold [X] (Inactive)
            korrigan.clark Korrigan Clark (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty