Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-605

Reconcile fails when 'ReadyMembers' are actually down

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 1.1.0
    • None
    • operator
    • None

    Description

      The following steps lead to a scenario where nodes are marked as Ready, but are actually down leading to continuously failing reconcile loop:

      • Start a 3 node cluster
      • Delete 2 nodes 0001 & 0002

      This leaves only node 0000 as ready:

        Members:
          Index:  3
          Ready:
            Name:  cb-example-0000
          Unready:
            Name:  cb-example-0001
            Name:  cb-example-0002
       

       

      • Wait for node 0001 to be started.
      • Delete node 0000 while 0001 is being started

      At this point 0001 is actually the Ready node, but only 0000 is still marked as ready.

      This causes reconcile to fail because we use the Ready members as API clients.  The fix is to Resync the ready members after reconcile fails.  

       

       

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            tommie Tommie McAfee (Inactive)
            tommie Tommie McAfee (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty