Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-110

Removed members should be deleted & replaced when operator resumes

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.7.0
    • 0.7.0
    • operator

    Description

      If a node is removed via UI while operator is paused then it's member should be removed when operator is resumed.  Associated pod should be deleted (if available).  Finally a replacement pod should be created and added to cluster.

       

      Current behavior is that the node remains in the unknown members after rebalance:

      time="2018-01-10T17:53:01Z" level=info msg="unknown nodes: test-couchbase-ndzhr-0001" cluster-name=test-couchbase-ndzhr module=cluster
      

      time="2018-01-10T17:53:01Z" level=info msg="Start reconciling" cluster-name=test-couchbase-ndzhr module=cluster
       
      time="2018-01-10T17:53:01Z" level=info msg="server config test_config_1: test-couchbase-ndzhr-0000" cluster-name=test-couchbase-ndzhr module=cluster
       
      time="2018-01-10T17:53:01Z" level=info msg="running members: test-couchbase-ndzhr-0000,test-couchbase-ndzhr-0001" cluster-name=test-couchbase-ndzhr module=cluster
       
      time="2018-01-10T17:53:01Z" level=info msg="cluster membership: test-couchbase-ndzhr-0000,test-couchbase-ndzhr-0001" cluster-name=test-couchbase-ndzhr module=cluster
       
      time="2018-01-10T17:53:01Z" level=info msg="active nodes: test-couchbase-ndzhr-0000" cluster-name=test-couchbase-ndzhr module=cluster
       
      time="2018-01-10T17:53:01Z" level=info msg="unknown nodes: test-couchbase-ndzhr-0001" cluster-name=test-couchbase-ndzhr module=cluster
       
      time="2018-01-10T17:53:01Z" level=info msg="Finish reconciling" cluster-name=test-couchbase-ndzhr module=cluster
       
      time="2018-01-10T17:53:09Z" level=info msg="Start reconciling" cluster-name=test-couchbase-ndzhr module=cluster
       
      time="2018-01-10T17:53:09Z" level=warning msg="cluster status: failed with error Code: 404, Endpoint: test-couchb

      Attachments

        For Gerrit Dashboard: K8S-110
        # Subject Branch Project Status CR V

        Activity

          Added a test for repro.  It currently fails waiting for node to be added:  http://review.couchbase.org/87636

          couchbase_util.go:104: rebalance out: test-couchbase-82r5h-0001
           
          cluster_test.go:504: Time out waiting for cluster event NewMemberAdded, New member test-couchbase-82r5h-0002 added to cluster:
           
          crd_util.go:73: deleting couchbase cluster: test-couchbase-82r5h

          tommie Tommie McAfee added a comment - Added a test for repro.  It currently fails waiting for node to be added:   http://review.couchbase.org/87636 couchbase_util.go: 104 : rebalance out: test-couchbase-82r5h- 0001   cluster_test.go: 504 : Time out waiting for cluster event NewMemberAdded, New member test-couchbase-82r5h- 0002 added to cluster:   crd_util.go: 73 : deleting couchbase cluster: test-couchbase-82r5h

          The issue you mentioned will be fixed by K8S-77. Can you take a look at this test case again once that gets merged? I ran it and it looks like there is a test case issue preventing the test from passing when the UI service references the node you removed.

          mikew Mike Wiederhold [X] (Inactive) added a comment - The issue you mentioned will be fixed by K8S-77 . Can you take a look at this test case again once that gets merged? I ran it and it looks like there is a test case issue preventing the test from passing when the UI service references the node you removed.

          Verified K8S-77 is the fix, and updated test.

          tommie Tommie McAfee added a comment - Verified K8S-77 is the fix, and updated test.

          People

            tommie Tommie McAfee
            tommie Tommie McAfee
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty