Simon Murray added a comment - 06/Aug/18 1:28 AM The operator doesn't create events for itself, only clusters

Simon Murray added a comment - 06/Aug/18 1:56 AM On reflection there are events, and they are being collected in other dumps so this fail needs some more investigation.

Simon Murray added a comment - 06/Aug/18 2:21 AM The deployment was created at: 2018-08-03T22:51:56 The collection happened at: 2018-08-04T02:17:31Z Events only last 1 hour (in this case there would be a scaling up event right at the beginning), so this isn't a bug.

Ashwin Govindarajulu added a comment - 06/Aug/18 2:43 AM These logs are collected within the test case execution. You can see that during the test execution, cbopinfo-20180805T172025-0700.tar.gz file was created to validate log files, but failed. So we again took a log file backup ( cbopinfo-20180805T172441-0700.tar.gz ) for debugging purpose. Attaching the same file for reference. --- FAIL: TestOperator/TestLogCollectUsingClusterNameAndNamespace (415.88s)         crd_util.go:41: creating couchbase cluster: test-couchbase-brwrm         crd_util.go:41: creating couchbase cluster: test-couchbase-mqbl2         crd_util.go:41: creating couchbase cluster: test-couchbase-59mtb         util.go:618: 2018-08-05 17:17:55.649842059 -0700 PDT m=+19367.743819416 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:17:55.652834917 -0700 PDT m=+19367.746812223 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:17:55.654801077 -0700 PDT m=+19367.748778400 waiting size (1), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:05.64973113 -0700 PDT m=+19377.743709070 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:05.654312114 -0700 PDT m=+19377.748289483 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:05.655982837 -0700 PDT m=+19377.749960073 waiting size (1), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:15.652465163 -0700 PDT m=+19387.746442632 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:15.653731612 -0700 PDT m=+19387.747709084 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:15.654143178 -0700 PDT m=+19387.748120625 waiting size (1), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:25.649049338 -0700 PDT m=+19397.743026770 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:25.65452265 -0700 PDT m=+19397.748500050 waiting size (1), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:25.655606628 -0700 PDT m=+19397.749584132 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:35.654415834 -0700 PDT m=+19407.748393330 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:18:35.655824481 -0700 PDT m=+19407.749801962 waiting size (3), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:35.656431154 -0700 PDT m=+19407.750408476 waiting size (1), healthy couchbase members: names ([])         util.go:618: 2018-08-05 17:18:45.649518779 -0700 PDT m=+19417.743496294 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:18:45.652279663 -0700 PDT m=+19417.746257166 waiting size (1), healthy couchbase members: names ([test-couchbase-mqbl2-0000])         util.go:618: 2018-08-05 17:18:45.653115674 -0700 PDT m=+19417.747093581 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:18:55.652561373 -0700 PDT m=+19427.746538906 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:18:55.654079763 -0700 PDT m=+19427.748057097 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:05.649128831 -0700 PDT m=+19437.743106177 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:05.653148592 -0700 PDT m=+19437.747125982 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:15.650989544 -0700 PDT m=+19447.744967135 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:15.653369465 -0700 PDT m=+19447.747346877 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:25.649412034 -0700 PDT m=+19457.743389602 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:25.655267605 -0700 PDT m=+19457.749245130 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:35.649775328 -0700 PDT m=+19467.743752696 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:35.653190808 -0700 PDT m=+19467.747168101 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:45.649123102 -0700 PDT m=+19477.743100382 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:45.652334279 -0700 PDT m=+19477.746311691 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:19:55.648846548 -0700 PDT m=+19487.742824030 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:19:55.651356324 -0700 PDT m=+19487.745333800 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:20:05.648440785 -0700 PDT m=+19497.742418290 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:20:05.651529568 -0700 PDT m=+19497.745506971 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:20:15.649186342 -0700 PDT m=+19507.743163856 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])         util.go:618: 2018-08-05 17:20:15.65237111 -0700 PDT m=+19507.746348940 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])         util.go:618: 2018-08-05 17:20:25.649427121 -0700 PDT m=+19517.743404600 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000 test-couchbase-brwrm-0001 test-couchbase-brwrm-0002])         util.go:618: 2018-08-05 17:20:25.651360846 -0700 PDT m=+19517.745338312 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000 test-couchbase-59mtb-0001 test-couchbase-59mtb-0002])         supportability_test.go:595: Collecting logs from single cluster         supportability_test.go:599: Returned: Wrote cluster information to cbopinfo-20180805T172025-0700.tar.gz         validation_test.go:59: Failure 1: File cbopinfo-20180805T172025-0700/default/deployment/couchbase-operator/events.yaml              Error: not found         validation_test.go:59: Failure 2: File cbopinfo-20180805T172025-0700/default/endpoints/couchbase-operator/couchbase-operator.yaml              Error: not found         validation_test.go:59: Failure 3: File cbopinfo-20180805T172025-0700/default/endpoints/kubernetes/kubernetes.yaml              Error: not found         validation_test.go:59: Failure 4: File cbopinfo-20180805T172025-0700/default/service/kubernetes/kubernetes.yaml              Error: not found         supportability_test.go:625: Collecting logs from cluster1 and cluster3         supportability_test.go:629: Returned: Wrote cluster information to cbopinfo-20180805T172029-0700.tar.gz         supportability_test.go:650: Collecting logs from all available cluster         supportability_test.go:654: Returned: Wrote cluster information to cbopinfo-20180805T172033-0700.tar.gz         supportability_test.go:677: Log Verification for kube-system         supportability_test.go:685: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns             Wrote cluster information to cbopinfo-20180805T172038-0700.tar.gz         supportability_test.go:715: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns             Wrote cluster information to cbopinfo-20180805T172100-0700.tar.gz         supportability_test.go:747: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns             Wrote cluster information to cbopinfo-20180805T172123-0700.tar.gz         supportability_test.go:780: Returned: Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0002:/tmp/cbinfo-default-test-couchbase-59mtb-0002-20180805T172153-0700.zip .             Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0000:/tmp/cbinfo-default-test-couchbase-59mtb-0000-20180805T172153-0700.zip .             Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0001:/tmp/cbinfo-default-test-couchbase-59mtb-0001-20180805T172153-0700.zip .             Wrote cluster information to cbopinfo-20180805T172153-0700.tar.gz         supportability_test.go:809: Test failed

Simon Murray added a comment - 06/Aug/18 4:19 AM For cbopinfo-20180805T172441-0700.tar.gz: Deployment creation time: 2018-08-05T21:20:27Z Logs collection time: 2018-08-06T00:32:48Z Again this is 3 hours later, hence no events exist for the object.

Simon Murray added a comment - 06/Aug/18 4:23 AM If you re-deploy the operator just before running this test I can guarantee it will fix the issue

Ashwin Govindarajulu added a comment - 06/Aug/18 4:25 AM So why the event existence is with respect to time ? We should be able to get the events at any point of time to know about the last event occurred in operator. In real time, we should be able to get the events in the longer run.

Simon Murray added a comment - 06/Aug/18 4:59 AM Kubernetes does not work that way, nothing we can do about it. If they collected every event ever their database would eventually fill up and the cluster will die. Likewise if we recorded every event we ever raised we'd run out of memory/disk eventually and be killed.

Simon Murray added a comment - 06/Aug/18 5:02 AM See https://github.com/kubernetes/kubernetes/issues/52521 K8S will not fix it.  We have everything we need for support in the log files already.

Ashwin Govindarajulu added a comment - 08/Aug/18 4:59 AM Closing this since it works as expected with k8s platform.