Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-516

Multi node auto failover: Cb Cluster failover the node, but couchbase-operator didn't catch that event

    XMLWordPrintable

Details

    Description

      TestCase: TestMultiNodeAutoFailover

      Scenario:

      1. Spawn 9 node cluster
      2. Kill nodes 0002, 0003, 0004 so that multinode failover to happen

      In the couchbase-cluster logs it says it failed over the node 0003.

      But operator events didn't catch that event.

      #3604: [user:info,2018-08-07T01:04:45.306Z,ns_1@test-couchbase-r5j6p-0000.test-couchbase-r5j6p.default.svc:<0.709.0>:auto_failover:log_failover_success:561]Node ('ns_1@test-couchbase-r5j6p-0003.test-couchbas      e-r5j6p.default.svc') was automatically failed over. Reason: All monitors report node is unhealthy.

      Operator events:

      util.go:189: Expected events to be:
          		Type: Normal | Reason: ServiceCreated | Message: Service for admin console `test-couchbase-r5j6p-ui` was created
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0000 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0001 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0002 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0003 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0004 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0005 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0006 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0007 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0008 added to cluster
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
          		Type: Normal | Reason: BucketCreated | Message: A new bucket `default` was created
          		Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0002 down
          		Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0003 down
          		Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0004 down
          		Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-r5j6p-0002 failed over
          		Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-r5j6p-0003 failed over
          		Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-r5j6p-0004 failed over
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0009 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0010 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0011 added to cluster
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-r5j6p-0002 removed from the cluster
          		Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-r5j6p-0003 removed from the cluster
          		Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-r5j6p-0004 removed from the cluster
          		Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
          		
          		but got:
          		Type: Normal | Reason: ServiceCreated | Message: Service for admin console `test-couchbase-r5j6p-ui` was created
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0000 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0001 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0002 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0003 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0004 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0005 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0006 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0007 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0008 added to cluster
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed
          		Type: Normal | Reason: BucketCreated | Message: A new bucket `default` was created
          		Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0002 down
          		Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0003 down
          		Type: Warning | Reason: MemberDown | Message: Existing member test-couchbase-r5j6p-0004 down
          		Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-r5j6p-0002 failed over
          		Type: Warning | Reason: MemberFailedOver | Message: Existing member test-couchbase-r5j6p-0004 failed over
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0009 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0010 added to cluster
          		Type: Normal | Reason: NewMemberAdded | Message: New member test-couchbase-r5j6p-0011 added to cluster
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: FailedAddNode | Message: Removed existing member test-couchbase-r5j6p-0003 because it failed before it could be added to the cluster
          		Type: Normal | Reason: RebalanceStarted | Message: A rebalance has been started to balance data across the cluster
          		Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-r5j6p-0002 removed from the cluster
          		Type: Normal | Reason: MemberRemoved | Message: Existing member test-couchbase-r5j6p-0004 removed from the cluster
          		Type: Normal | Reason: RebalanceCompleted | Message: A rebalance has completed

      Attachments

        Activity

          People

            mikew Mike Wiederhold [X] (Inactive)
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty