Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.0.0
Description
cbopinfo-20180803T190517-0700/default/deployment/couchbase-operator/events.yaml file missing in the collected logs
Attachments
Activity
On reflection there are events, and they are being collected in other dumps so this fail needs some more investigation.
The deployment was created at: 2018-08-03T22:51:56
The collection happened at: 2018-08-04T02:17:31Z
Events only last 1 hour (in this case there would be a scaling up event right at the beginning), so this isn't a bug.
These logs are collected within the test case execution.
You can see that during the test execution, cbopinfo-20180805T172025-0700.tar.gz file was created to validate log files, but failed.
So we again took a log file backup (cbopinfo-20180805T172441-0700.tar.gz) for debugging purpose.
Attaching the same file for reference.
--- FAIL: TestOperator/TestLogCollectUsingClusterNameAndNamespace (415.88s)
|
crd_util.go:41: creating couchbase cluster: test-couchbase-brwrm
|
crd_util.go:41: creating couchbase cluster: test-couchbase-mqbl2
|
crd_util.go:41: creating couchbase cluster: test-couchbase-59mtb
|
util.go:618: 2018-08-05 17:17:55.649842059 -0700 PDT m=+19367.743819416 waiting size (3), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:17:55.652834917 -0700 PDT m=+19367.746812223 waiting size (3), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:17:55.654801077 -0700 PDT m=+19367.748778400 waiting size (1), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:05.64973113 -0700 PDT m=+19377.743709070 waiting size (3), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:05.654312114 -0700 PDT m=+19377.748289483 waiting size (3), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:05.655982837 -0700 PDT m=+19377.749960073 waiting size (1), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:15.652465163 -0700 PDT m=+19387.746442632 waiting size (3), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:15.653731612 -0700 PDT m=+19387.747709084 waiting size (3), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:15.654143178 -0700 PDT m=+19387.748120625 waiting size (1), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:25.649049338 -0700 PDT m=+19397.743026770 waiting size (3), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:25.65452265 -0700 PDT m=+19397.748500050 waiting size (1), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:25.655606628 -0700 PDT m=+19397.749584132 waiting size (3), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:35.654415834 -0700 PDT m=+19407.748393330 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
|
util.go:618: 2018-08-05 17:18:35.655824481 -0700 PDT m=+19407.749801962 waiting size (3), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:35.656431154 -0700 PDT m=+19407.750408476 waiting size (1), healthy couchbase members: names ([])
|
util.go:618: 2018-08-05 17:18:45.649518779 -0700 PDT m=+19417.743496294 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
|
util.go:618: 2018-08-05 17:18:45.652279663 -0700 PDT m=+19417.746257166 waiting size (1), healthy couchbase members: names ([test-couchbase-mqbl2-0000])
|
util.go:618: 2018-08-05 17:18:45.653115674 -0700 PDT m=+19417.747093581 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
|
util.go:618: 2018-08-05 17:18:55.652561373 -0700 PDT m=+19427.746538906 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
|
util.go:618: 2018-08-05 17:18:55.654079763 -0700 PDT m=+19427.748057097 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
|
util.go:618: 2018-08-05 17:19:05.649128831 -0700 PDT m=+19437.743106177 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
|
util.go:618: 2018-08-05 17:19:05.653148592 -0700 PDT m=+19437.747125982 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
|
util.go:618: 2018-08-05 17:19:15.650989544 -0700 PDT m=+19447.744967135 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
|
util.go:618: 2018-08-05 17:19:15.653369465 -0700 PDT m=+19447.747346877 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
|
util.go:618: 2018-08-05 17:19:25.649412034 -0700 PDT m=+19457.743389602 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
|
util.go:618: 2018-08-05 17:19:25.655267605 -0700 PDT m=+19457.749245130 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
|
util.go:618: 2018-08-05 17:19:35.649775328 -0700 PDT m=+19467.743752696 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
|
util.go:618: 2018-08-05 17:19:35.653190808 -0700 PDT m=+19467.747168101 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
|
util.go:618: 2018-08-05 17:19:45.649123102 -0700 PDT m=+19477.743100382 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
|
util.go:618: 2018-08-05 17:19:45.652334279 -0700 PDT m=+19477.746311691 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
|
util.go:618: 2018-08-05 17:19:55.648846548 -0700 PDT m=+19487.742824030 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
|
util.go:618: 2018-08-05 17:19:55.651356324 -0700 PDT m=+19487.745333800 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
|
util.go:618: 2018-08-05 17:20:05.648440785 -0700 PDT m=+19497.742418290 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
|
util.go:618: 2018-08-05 17:20:05.651529568 -0700 PDT m=+19497.745506971 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
|
util.go:618: 2018-08-05 17:20:15.649186342 -0700 PDT m=+19507.743163856 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000])
|
util.go:618: 2018-08-05 17:20:15.65237111 -0700 PDT m=+19507.746348940 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000])
|
util.go:618: 2018-08-05 17:20:25.649427121 -0700 PDT m=+19517.743404600 waiting size (3), healthy couchbase members: names ([test-couchbase-brwrm-0000 test-couchbase-brwrm-0001 test-couchbase-brwrm-0002])
|
util.go:618: 2018-08-05 17:20:25.651360846 -0700 PDT m=+19517.745338312 waiting size (3), healthy couchbase members: names ([test-couchbase-59mtb-0000 test-couchbase-59mtb-0001 test-couchbase-59mtb-0002])
|
supportability_test.go:595: Collecting logs from single cluster
|
supportability_test.go:599: Returned: Wrote cluster information to cbopinfo-20180805T172025-0700.tar.gz
|
validation_test.go:59: Failure 1: File cbopinfo-20180805T172025-0700/default/deployment/couchbase-operator/events.yaml
|
Error: not found
|
validation_test.go:59: Failure 2: File cbopinfo-20180805T172025-0700/default/endpoints/couchbase-operator/couchbase-operator.yaml
|
Error: not found
|
validation_test.go:59: Failure 3: File cbopinfo-20180805T172025-0700/default/endpoints/kubernetes/kubernetes.yaml
|
Error: not found
|
validation_test.go:59: Failure 4: File cbopinfo-20180805T172025-0700/default/service/kubernetes/kubernetes.yaml
|
Error: not found
|
supportability_test.go:625: Collecting logs from cluster1 and cluster3
|
supportability_test.go:629: Returned: Wrote cluster information to cbopinfo-20180805T172029-0700.tar.gz
|
supportability_test.go:650: Collecting logs from all available cluster
|
supportability_test.go:654: Returned: Wrote cluster information to cbopinfo-20180805T172033-0700.tar.gz
|
supportability_test.go:677: Log Verification for kube-system
|
supportability_test.go:685: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns
|
Wrote cluster information to cbopinfo-20180805T172038-0700.tar.gz
|
supportability_test.go:715: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns
|
Wrote cluster information to cbopinfo-20180805T172100-0700.tar.gz
|
supportability_test.go:747: Returned: unable to fetch Logs for type Deployment name kube-dns: No pods delected for Deployment kube-dns
|
Wrote cluster information to cbopinfo-20180805T172123-0700.tar.gz
|
supportability_test.go:780: Returned: Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0002:/tmp/cbinfo-default-test-couchbase-59mtb-0002-20180805T172153-0700.zip .
|
Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0000:/tmp/cbinfo-default-test-couchbase-59mtb-0000-20180805T172153-0700.zip .
|
Server logs accessible via: kubectl cp default/test-couchbase-59mtb-0001:/tmp/cbinfo-default-test-couchbase-59mtb-0001-20180805T172153-0700.zip .
|
Wrote cluster information to cbopinfo-20180805T172153-0700.tar.gz
|
supportability_test.go:809: Test failed
|
For cbopinfo-20180805T172441-0700.tar.gz:
Deployment creation time: 2018-08-05T21:20:27Z
Logs collection time: 2018-08-06T00:32:48Z
Again this is 3 hours later, hence no events exist for the object.
If you re-deploy the operator just before running this test I can guarantee it will fix the issue
So why the event existence is with respect to time ?
We should be able to get the events at any point of time to know about the last event occurred in operator.
In real time, we should be able to get the events in the longer run.
Kubernetes does not work that way, nothing we can do about it.
If they collected every event ever their database would eventually fill up and the cluster will die.
Likewise if we recorded every event we ever raised we'd run out of memory/disk eventually and be killed.
See https://github.com/kubernetes/kubernetes/issues/52521 K8S will not fix it. We have everything we need for support in the log files already.
The operator doesn't create events for itself, only clusters