Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-544

events being returned in different order than operator reports

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 1.0.0
    • None
    • None
    • None

    Description

      Is some tests we kill 3 nodes and wait for down events for each of the three nodes in parallel, returning the order in which the events are seen. However, the operator will report a different ordering. In this example, nodes 2, 5, and 8 are killed in that order and wait for gorountine is spun up for each of these nodes, listening for down event. The goroutines report the order as 2, 8, then 5, but then the operator says the order was 2, 5, 8:

          util.go:187: Expected events to be:
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0000 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0001 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0002 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0003 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0004 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0005 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0006 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0007 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0008 added to cluster
       
          Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
       
          Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
       
          Type: Normal | Reason: BucketCreated | Message: A new bucket `testBucket` was created
       
          Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0002 down
       
          Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0008 down
       
          Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0005 down
       
          Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0008 failed over
       
          Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0002 failed over
       
          Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0005 failed over
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0009 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0010 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0011 added to cluster
       
          Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
       
          Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0002 removed from the cluster
       
          Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0005 removed from the cluster
       
          Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0008 removed from the cluster
       
          Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
       
          
       
          but got:
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0000 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0001 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0002 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0003 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0004 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0005 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0006 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0007 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0008 added to cluster
       
          Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
       
          Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
       
          Type: Normal | Reason: BucketCreated | Message: A new bucket `testBucket` was created
       
          Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0002 down
       
          Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0005 down
       
          Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-v9fxp-0008 down
       
          Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0002 failed over
       
          Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0005 failed over
       
          Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-v9fxp-0008 failed over
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0009 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0010 added to cluster
       
          Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-v9fxp-0011 added to cluster
       
          Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
       
          Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0002 removed from the cluster
       
          Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0005 removed from the cluster
       
          Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-v9fxp-0008 removed from the cluster
       
          Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              korrigan.clark Korrigan Clark (Inactive)
              korrigan.clark Korrigan Clark (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty