Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-528

Print valid end of reconcile after cluster update happens to improve the logging

    XMLWordPrintable

Details

    Description

      Valid prints are needed at the end of successful reconcile procedure.

      Currently we have removed the Start/Stop reconcile prints. But even the cluster is reconciled successfully, there is no print available from operator logs to get that.

      time="2018-08-10T09:36:43Z" level=info msg="is rebalancing: false" cluster-name=test-couchbase-cbxd6 module=cluster
      		 
      time="2018-08-10T09:36:43Z" level=info msg="needs rebalance: false" cluster-name=test-couchbase-cbxd6 module=cluster
      		 
      time="2018-08-10T09:36:50Z" level=info msg="Creating a pod (test-couchbase-tv6r8-0001) running Couchbase enterprise-5.5.0" cluster-name=test-couchbase-tv6r8 module=cluster
      		 
      time="2018-08-10T09:36:53Z" level=info msg="Creating a pod (test-couchbase-cbxd6-0001) running Couchbase enterprise-5.5.0" cluster-name=test-couchbase-cbxd6 module=cluster
      		 
      time="2018-08-10T09:37:30Z" level=info msg="added member (test-couchbase-tv6r8-0001)" cluster-name=test-couchbase-tv6r8 module=cluster
      		 
      time="2018-08-10T09:37:30Z" level=info msg="Creating a pod (test-couchbase-tv6r8-0002) running Couchbase enterprise-5.5.0" cluster-name=test-couchbase-tv6r8 module=cluster
      		 
      time="2018-08-10T09:37:33Z" level=info msg="added member (test-couchbase-cbxd6-0001)" cluster-name=test-couchbase-cbxd6 module=cluster
      		 
      time="2018-08-10T09:37:33Z" level=info msg="Creating a pod (test-couchbase-cbxd6-0002) running Couchbase enterprise-5.5.0" cluster-name=test-couchbase-cbxd6 module=cluster
      		 
      time="2018-08-10T09:38:06Z" level=info msg="added member (test-couchbase-tv6r8-0002)" cluster-name=test-couchbase-tv6r8 module=cluster
      		 
      time="2018-08-10T09:38:08Z" level=info msg="Rebalance progress: 0.000000" cluster-name=test-couchbase-tv6r8 module=cluster
      		 
      time="2018-08-10T09:38:10Z" level=info msg="added member (test-couchbase-cbxd6-0002)" cluster-name=test-couchbase-cbxd6 module=cluster
      		 
      time="2018-08-10T09:38:12Z" level=info msg="Rebalance progress: 0.000000" cluster-name=test-couchbase-cbxd6 module=cluster

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Ascending order - Click to sort in descending order
          mikew Mike Wiederhold [X] (Inactive) added a comment -

          If you want those log messages you can enable debug logging, but we don't plan on adding them back. We have some ideas for improving the log messages in the future, but printing every time the reconcile loop starts and ends just fills up the logs.

          mikew Mike Wiederhold [X] (Inactive) added a comment - If you want those log messages you can enable debug logging, but we don't plan on adding them back. We have some ideas for improving the log messages in the future, but printing every time the reconcile loop starts and ends just fills up the logs.
          ashwin.govindarajulu Ashwin Govindarajulu added a comment - - edited

          Mike Wiederhold [X] I am also suggesting not to print reconcile message for each loop, but I want to print the end of reconcile only if the create/update happens.

          For example, in the above printed logs you can see couchbase cluster test-couchbase-cbxd6 was created with 3 nodes. Here is would be better if we print the message after rebalance complete message Because rebalance was completed after 0.000% here. So user might also think the cluster is still rebalancing.

           

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - - edited Mike Wiederhold [X] I am also suggesting not to print reconcile message for each loop, but I want to print the end of reconcile only if the create/update happens. For example, in the above printed logs you can see couchbase cluster test-couchbase-cbxd6 was created with 3 nodes. Here is would be better if we print the message after rebalance complete message Because rebalance was completed after 0.000% here. So user might also think the cluster is still rebalancing.  
          simon.murray Simon Murray added a comment -

          So this is not entirely simple.  It's in the same bucket as having to check we are going to do something before we do it in order not to spam the world with cluster state every few seconds.

          To take a leaf from N1QL's book what I propose is:

          • Work out what to do and in what order (e.g. make a plan)
          • If we need to do anything
            • log the initial cluster state
            • do the work
            • report completion/error

          This should remove the code duplication, by decoupling the what from how, make the state machine much simpler, and fulfill the requirement to only dump stuff when it's pertinent.

          simon.murray Simon Murray added a comment - So this is not entirely simple.  It's in the same bucket as having to check we are going to do something before we do it in order not to spam the world with cluster state every few seconds. To take a leaf from N1QL's book what I propose is: Work out what to do and in what order (e.g. make a plan) If we need to do anything log the initial cluster state do the work report completion/error This should remove the code duplication, by decoupling the what from how, make the state machine much simpler, and fulfill the requirement to only dump stuff when it's pertinent.

          People

            simon.murray Simon Murray
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty