Auto-failed over node attempting to failover the healthy node in the cluster

Steps:

• 3 node kv cluster

  +---------------+

  | Nodes         |

  +---------------+

  | 172.23.108.69 |

  | 172.23.108.67 |

  | 172.23.108.68 |

  +---------------+

• 1 magma bucket with replicas=2

  +---------+-----------+---------+----------+------------+-----+-------+

  | Bucket  | Type      | Storage | Replicas | Durability | TTL | Items |

  +---------+-----------+---------+----------+------------+-----+-------+

  | default | couchbase | magma   | 2        | none       | 0   | 55574

  +---------+-----------+---------+----------+------------+-----+-------+

• Set auto-failover timeout=5 seconds
• Induce failure on node "172.23.108.68" using 'service network stop'
• Wait for auto-failover to kick-in

Observation:

Auto-fo triggered as expected and post failover the test recovers back the node.

Post recovery the node (172.23.108.68) is becoming an Orchestrator (MB-58749) and tries to failover the healthy node '172.23.108.67'

TAF test:

failover.AutoFailoverTests.AutoFailoverTests:

    test_autofailover,timeout=5,num_node_failures=1,failover_action=restart_network,nodes_init=3,replicas=2,durability=MAJORITY,num_items=100000,replicas=2