Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-511

cbopinfo: couchbase-operator/events.yaml file missing under deployment dir

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.0.0
    • kubernetes

    Description

      cbopinfo-20180803T190517-0700/default/deployment/couchbase-operator/events.yaml file missing in the collected logs

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          simon.murray Simon Murray added a comment -

          The operator doesn't create events for itself, only clusters

          simon.murray Simon Murray added a comment - The operator doesn't create events for itself, only clusters
          simon.murray Simon Murray added a comment -

          On reflection there are events, and they are being collected in other dumps so this fail needs some more investigation.

          simon.murray Simon Murray added a comment - On reflection there are events, and they are being collected in other dumps so this fail needs some more investigation.
          simon.murray Simon Murray added a comment -

          The deployment was created at: 2018-08-03T22:51:56

          The collection happened at: 2018-08-04T02:17:31Z

          Events only last 1 hour (in this case there would be a scaling up event right at the beginning), so this isn't a bug.

          simon.murray Simon Murray added a comment - The deployment was created at: 2018-08-03T22:51:56 The collection happened at: 2018-08-04T02:17:31Z Events only last 1 hour (in this case there would be a scaling up event right at the beginning), so this isn't a bug.

          These logs are collected within the test case execution.

          You can see that during the test execution, cbopinfo-20180805T172025-0700.tar.gz file was created to validate log files, but failed.

          So we again took a log file backup (cbopinfo-20180805T172441-0700.tar.gz) for debugging purpose.

          Attaching the same file for reference.

          --- FAIL: TestOperator/TestLogCollectUsingClusterNameAndNamespace (415.88s)
                  crd_util.go:41: creating couchbase cluster: test-couchbase-brwrm
                  crd_util.go:41: creating couchbase cluster: test-couchbase-mqbl2
                  crd_util.go:41: creating couchbase cluster: test-couchbase-59mtb
                  util.go:618: 2018-08-05 17:17:55.649842059 -0700 PDT m=+19367.743819416 waiting size (3), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:17:55.652834917 -0700 PDT m=+19367.746812223 waiting size (3), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:17:55.654801077 -0700 PDT m=+19367.748778400 waiting size (1), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:05.64973113 -0700 PDT m=+19377.743709070 waiting size (3), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:05.654312114 -0700 PDT m=+19377.748289483 waiting size (3), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:05.655982837 -0700 PDT m=+19377.749960073 waiting size (1), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:15.652465163 -0700 PDT m=+19387.746442632 waiting size (3), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:15.653731612 -0700 PDT m=+19387.747709084 waiting size (3), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:15.654143178 -0700 PDT m=+19387.748120625 waiting size (1), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:25.649049338 -0700 PDT m=+19397.743026770 waiting size (3), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:25.65452265 -0700 PDT m=+19397.748500050 waiting size (1), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:25.655606628 -0700 PDT m=+19397.749584132 waiting size (3), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:35.654415834 -0700 PDT m=+19407.748393330 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
                  util.go:618: 2018-08-05 17:18:35.655824481 -0700 PDT m=+19407.749801962 waiting size (3), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:35.656431154 -0700 PDT m=+19407.750408476 waiting size (1), healthy couchbase members: names ([])
                  util.go:618: 2018-08-05 17:18:45.649518779 -0700 PDT m=+19417.743496294 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
                  util.go:618: 2018-08-05 17:18:45.652279663 -0700 PDT m=+19417.746257166 waiting size (1), healthy couchbase members: names ([test-couchbase-mqbl2-0000])
                  util.go:618: 2018-08-05 17:18:45.653115674 -0700 PDT m=+19417.747093581 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
                  util.go:618: 2018-08-05 17:18:55.652561373 -0700 PDT m=+19427.746538906 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
                  util.go:618: 2018-08-05 17:18:55.654079763 -0700 PDT m=+19427.748057097 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
                  util.go:618: 2018-08-05 17:19:05.649128831 -0700 PDT m=+19437.743106177 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
                  util.go:618: 2018-08-05 17:19:05.653148592 -0700 PDT m=+19437.747125982 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
                  util.go:618: 2018-08-05 17:19:15.650989544 -0700 PDT m=+19447.744967135 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
                  util.go:618: 2018-08-05 17:19:15.653369465 -0700 PDT m=+19447.747346877 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
                  util.go:618: 2018-08-05 17:19:25.649412034 -0700 PDT m=+19457.743389602 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
                  util.go:618: 2018-08-05 17:19:25.655267605 -0700 PDT m=+19457.749245130 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
                  util.go:618: 2018-08-05 17:19:35.649775328 -0700 PDT m=+19467.743752696 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
                  util.go:618: 2018-08-05 17:19:35.653190808 -0700 PDT m=+19467.747168101 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
                  util.go:618: 2018-08-05 17:19:45.649123102 -0700 PDT m=+19477.743100382 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
                  util.go:618: 2018-08-05 17:19:45.652334279 -0700 PDT m=+19477.746311691 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
                  util.go:618: 2018-08-05 17:19:55.648846548 -0700 PDT m=+19487.742824030 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
                  util.go:618: 2018-08-05 17:19:55.651356324 -0700 PDT m=+19487.745333800 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
                  util.go:618: 2018-08-05 17:20:05.648440785 -0700 PDT m=+19497.742418290 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
                  util.go:618: 2018-08-05 17:20:05.651529568 -0700 PDT m=+19497.745506971 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
                  util.go:618: 2018-08-05 17:20:15.649186342 -0700 PDT m=+19507.743163856 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
                  util.go:618: 2018-08-05 17:20:15.65237111 -0700 PDT m=+19507.746348940 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
                  util.go:618: 2018-08-05 17:20:25.649427121 -0700 PDT m=+19517.743404600 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000 test-couchbase-brwrm-0001 test-couchbase-brwrm-0002])
                  util.go:618: 2018-08-05 17:20:25.651360846 -0700 PDT m=+19517.745338312 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000 test-couchbase-59mtb-0001 test-couchbase-59mtb-0002])
                  supportability_test.go:595: Collecting logs from single cluster
                  supportability_test.go:599: Returned: Wrote cluster information to cbopinfo-20180805T172025-0700.tar.gz
                  validation_test.go:59: Failure 1: File cbopinfo-20180805T172025-0700/default/deployment/couchbase-operator/events.yaml
                       Error: not found
                  validation_test.go:59: Failure 2: File cbopinfo-20180805T172025-0700/default/endpoints/couchbase-operator/couchbase-operator.yaml
                       Error: not found
                  validation_test.go:59: Failure 3: File cbopinfo-20180805T172025-0700/default/endpoints/kubernetes/kubernetes.yaml
                       Error: not found
                  validation_test.go:59: Failure 4: File cbopinfo-20180805T172025-0700/default/service/kubernetes/kubernetes.yaml
                       Error: not found
                  supportability_test.go:625: Collecting logs from cluster1 and cluster3
                  supportability_test.go:629: Returned: Wrote cluster information to cbopinfo-20180805T172029-0700.tar.gz
                  supportability_test.go:650: Collecting logs from all available cluster
                  supportability_test.go:654: Returned: Wrote cluster information to cbopinfo-20180805T172033-0700.tar.gz
                  supportability_test.go:677: Log Verification for kube-system
                  supportability_test.go:685: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns
                      Wrote cluster information to cbopinfo-20180805T172038-0700.tar.gz
                  supportability_test.go:715: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns
                      Wrote cluster information to cbopinfo-20180805T172100-0700.tar.gz
                  supportability_test.go:747: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns
                      Wrote cluster information to cbopinfo-20180805T172123-0700.tar.gz
                  supportability_test.go:780: Returned: Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0002:/tmp/cbinfo-default-test-couchbase-59mtb-0002-20180805T172153-0700.zip .
                      Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0000:/tmp/cbinfo-default-test-couchbase-59mtb-0000-20180805T172153-0700.zip .
                      Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0001:/tmp/cbinfo-default-test-couchbase-59mtb-0001-20180805T172153-0700.zip .
                      Wrote cluster information to cbopinfo-20180805T172153-0700.tar.gz
                  supportability_test.go:809: Test failed

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - These logs are collected within the test case execution. You can see that during the test execution, cbopinfo-20180805T172025-0700.tar.gz file was created to validate log files, but failed. So we again took a log file backup ( cbopinfo-20180805T172441-0700.tar.gz ) for debugging purpose. Attaching the same file for reference. --- FAIL: TestOperator/TestLogCollectUsingClusterNameAndNamespace (415.88s)         crd_util.go:41: creating couchbase cluster: test-couchbase-brwrm         crd_util.go:41: creating couchbase cluster: test-couchbase-mqbl2         crd_util.go:41: creating couchbase cluster: test-couchbase-59mtb         util.go:618: 2018-08-05 17:17:55.649842059 -0700 PDT m=+19367.743819416 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:17:55.652834917 -0700 PDT m=+19367.746812223 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:17:55.654801077 -0700 PDT m=+19367.748778400 waiting size (1), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:05.64973113 -0700 PDT m=+19377.743709070 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:05.654312114 -0700 PDT m=+19377.748289483 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:05.655982837 -0700 PDT m=+19377.749960073 waiting size (1), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:15.652465163 -0700 PDT m=+19387.746442632 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:15.653731612 -0700 PDT m=+19387.747709084 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:15.654143178 -0700 PDT m=+19387.748120625 waiting size (1), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:25.649049338 -0700 PDT m=+19397.743026770 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:25.65452265 -0700 PDT m=+19397.748500050 waiting size (1), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:25.655606628 -0700 PDT m=+19397.749584132 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:35.654415834 -0700 PDT m=+19407.748393330 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:18:35.655824481 -0700 PDT m=+19407.749801962 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:35.656431154 -0700 PDT m=+19407.750408476 waiting size (1), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:45.649518779 -0700 PDT m=+19417.743496294 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:18:45.652279663 -0700 PDT m=+19417.746257166 waiting size (1), healthy couchbase members: names ([test-couchbase-mqbl2-0000])         util.go:618: 2018-08-05 17:18:45.653115674 -0700 PDT m=+19417.747093581 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:18:55.652561373 -0700 PDT m=+19427.746538906 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:18:55.654079763 -0700 PDT m=+19427.748057097 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:05.649128831 -0700 PDT m=+19437.743106177 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:05.653148592 -0700 PDT m=+19437.747125982 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:15.650989544 -0700 PDT m=+19447.744967135 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:15.653369465 -0700 PDT m=+19447.747346877 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:25.649412034 -0700 PDT m=+19457.743389602 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:25.655267605 -0700 PDT m=+19457.749245130 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:35.649775328 -0700 PDT m=+19467.743752696 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:35.653190808 -0700 PDT m=+19467.747168101 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:45.649123102 -0700 PDT m=+19477.743100382 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:45.652334279 -0700 PDT m=+19477.746311691 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:55.648846548 -0700 PDT m=+19487.742824030 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:55.651356324 -0700 PDT m=+19487.745333800 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:20:05.648440785 -0700 PDT m=+19497.742418290 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:20:05.651529568 -0700 PDT m=+19497.745506971 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:20:15.649186342 -0700 PDT m=+19507.743163856 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:20:15.65237111 -0700 PDT m=+19507.746348940 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:20:25.649427121 -0700 PDT m=+19517.743404600 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000 test-couchbase-brwrm-0001 test-couchbase-brwrm-0002])         util.go:618: 2018-08-05 17:20:25.651360846 -0700 PDT m=+19517.745338312 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000 test-couchbase-59mtb-0001 test-couchbase-59mtb-0002])         supportability_test.go:595: Collecting logs from single cluster         supportability_test.go:599: Returned: Wrote cluster information to cbopinfo-20180805T172025-0700.tar.gz         validation_test.go:59: Failure 1: File cbopinfo-20180805T172025-0700/default/deployment/couchbase-operator/events.yaml              Error: not found         validation_test.go:59: Failure 2: File cbopinfo-20180805T172025-0700/default/endpoints/couchbase-operator/couchbase-operator.yaml              Error: not found         validation_test.go:59: Failure 3: File cbopinfo-20180805T172025-0700/default/endpoints/kubernetes/kubernetes.yaml              Error: not found         validation_test.go:59: Failure 4: File cbopinfo-20180805T172025-0700/default/service/kubernetes/kubernetes.yaml              Error: not found         supportability_test.go:625: Collecting logs from cluster1 and cluster3         supportability_test.go:629: Returned: Wrote cluster information to cbopinfo-20180805T172029-0700.tar.gz         supportability_test.go:650: Collecting logs from all available cluster         supportability_test.go:654: Returned: Wrote cluster information to cbopinfo-20180805T172033-0700.tar.gz         supportability_test.go:677: Log Verification for kube-system         supportability_test.go:685: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns             Wrote cluster information to cbopinfo-20180805T172038-0700.tar.gz         supportability_test.go:715: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns             Wrote cluster information to cbopinfo-20180805T172100-0700.tar.gz         supportability_test.go:747: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns             Wrote cluster information to cbopinfo-20180805T172123-0700.tar.gz         supportability_test.go:780: Returned: Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0002:/tmp/cbinfo-default-test-couchbase-59mtb-0002-20180805T172153-0700.zip .             Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0000:/tmp/cbinfo-default-test-couchbase-59mtb-0000-20180805T172153-0700.zip .             Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0001:/tmp/cbinfo-default-test-couchbase-59mtb-0001-20180805T172153-0700.zip .             Wrote cluster information to cbopinfo-20180805T172153-0700.tar.gz         supportability_test.go:809: Test failed
          simon.murray Simon Murray added a comment -

          For cbopinfo-20180805T172441-0700.tar.gz:

          Deployment creation time: 2018-08-05T21:20:27Z

          Logs collection time: 2018-08-06T00:32:48Z

          Again this is 3 hours later, hence no events exist for the object.

          simon.murray Simon Murray added a comment - For cbopinfo-20180805T172441-0700.tar.gz: Deployment creation time: 2018-08-05T21:20:27Z Logs collection time: 2018-08-06T00:32:48Z Again this is 3 hours later, hence no events exist for the object.
          simon.murray Simon Murray added a comment -

          If you re-deploy the operator just before running this test I can guarantee it will fix the issue

          simon.murray Simon Murray added a comment - If you re-deploy the operator just before running this test I can guarantee it will fix the issue

          So why the event existence is with respect to time ?

          We should be able to get the events at any point of time to know about the last event occurred in operator.

          In real time, we should be able to get the events in the longer run.

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - So why the event existence is with respect to time ? We should be able to get the events at any point of time to know about the last event occurred in operator. In real time, we should be able to get the events in the longer run.
          simon.murray Simon Murray added a comment -

          Kubernetes does not work that way, nothing we can do about it.

          If they collected every event ever their database would eventually fill up and the cluster will die.

          Likewise if we recorded every event we ever raised we'd run out of memory/disk eventually and be killed.

          simon.murray Simon Murray added a comment - Kubernetes does not work that way, nothing we can do about it. If they collected every event ever their database would eventually fill up and the cluster will die. Likewise if we recorded every event we ever raised we'd run out of memory/disk eventually and be killed.
          simon.murray Simon Murray added a comment -

          See https://github.com/kubernetes/kubernetes/issues/52521 K8S will not fix it.  We have everything we need for support in the log files already.

          simon.murray Simon Murray added a comment - See https://github.com/kubernetes/kubernetes/issues/52521 K8S will not fix it.  We have everything we need for support in the log files already.

          Closing this since it works as expected with k8s platform.

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Closing this since it works as expected with k8s platform.

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty