Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2210

[Fluent-bit] Cannot set an ownerRef on a resource you can't delete

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.2.0
    • documentation, logging, operator
    • None
    • 20: PE/Docs/Cleanup, 22: Docs/Cleanup/CI
    • 1

    Description

      Job: http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-oc-pipeline/114/

      Platform: OCP v4.4

      Logging Image: registry.connect.redhat.com/couchbase/fluent-bit:1.0.1-1

      Operator Image: registry.gitlab.com/cb-rhcc/operator:latest  {build 229}

      TestCase: TestLoggingAndAuditingDefaults 

      BackTrace:

      03:56:18     util.go:1288: timeout: size does not match, wanted 1, got 0
      03:56:18     util.go:1289: goroutine 1135 [running]:
      03:56:18         runtime/debug.Stack(0x1f09914, 0xc0012fc558, 0xc0012fc550)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/go/src/runtime/debug/stack.go:24 +0xab
      03:56:18         github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc0009ec780, 0x3525440, 0xc0006717e0)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/util.go:1284 +0x34
      03:56:18         github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustWaitClusterStatusHealthy(0xc0009ec780, 0xc000592380, 0xc000a88000, 0xd18c2e2800)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/wait_util.go:580 +0x94
      03:56:18         github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustNewClusterFromSpec(0xc0009ec780, 0xc000592380, 0xc000b02500, 0x11)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/util.go:88 +0x9d
      03:56:18         github.com/couchbase/couchbase-operator/test/e2e.TestLoggingAndAuditingDefaults(0xc0009ec780)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/logging_test.go:72 +0xbe5
      03:56:18         testing.tRunner(0xc0009ec780, 0x2b1c328)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/go/src/testing/testing.go:1193 +0x203
      03:56:18         created by testing.(*T).Run
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/go/src/testing/testing.go:1238 +0x5d8
      03:56:18         
      03:56:18 time="2021-05-24T03:56:12-07:00" level=info msg="TestOperator/TestLoggingAndAuditingDefaults ✗" 

      Error:

      "error":"secrets \"fluent-bit-config\" is forbidden: cannot set an ownerRef on a resource you can't delete: 

      (cbopinfo attached)

      The same test was tested with

      Logging Image: registry.gitlab.com/cb-rhcc/fluent-bit:1.0.3-103

      It failed with this version as well with similar error as described above.

      Job: http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-oc-pipeline/115/console

      (cbopinfo attached, having 1.0.3 in the name)

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            The three tests the use default log configuration are:

            1. TestChangeLogShipperImage
            2. TestRebalanceLogProcessing
            3. TestLoggingAndAuditingDefaults

            Of these, only the failing TestLoggingAndAuditingDefaults creates a dodgy configuration first prior to creating the cluster to demonstrate that it is updated during reconcile.

            patrick.stephens Patrick Stephens (Inactive) added a comment - The three tests the use default log configuration are: TestChangeLogShipperImage TestRebalanceLogProcessing TestLoggingAndAuditingDefaults Of these, only the failing TestLoggingAndAuditingDefaults creates a dodgy configuration first prior to creating the cluster to demonstrate that it is updated during reconcile.

            The fix was tested with Operator Image: cb-rhcc/operator:_2.2.0-232 and Logging Image: _registry.connect.redhat.com/couchbase/fluent-bit:1.0.1-1 on Openshift v4.4. All Logging tests pass.

            23:20:11 --- PASS: TestOperator (0.00s)
            23:20:11     --- PASS: TestOperator/TestNoLogOrAuditConfig (43.82s)
            23:20:11     --- PASS: TestOperator/TestLoggingDynamicConfigReload (139.93s)
            23:20:11     --- PASS: TestOperator/TestLoggingAndAuditingDefaults (97.76s)
            23:20:11     --- PASS: TestOperator/TestRebalanceLogProcessing (179.19s)
            23:20:11     --- PASS: TestOperator/TestInflightLogRedaction (214.61s)
            23:20:11     --- PASS: TestOperator/TestChangeLogShipperImage (75.27s)
            23:20:11     --- PASS: TestOperator/TestAuditingNoLogging (46.64s)
            23:20:11     --- PASS: TestOperator/TestCustomLogging (131.37s)
            23:20:11 PASS
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="Test Summary"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   1: TestNoLogOrAuditConfig ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   2: TestLoggingAndAuditingDefaults ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   3: TestAuditingNoLogging ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   4: TestCustomLogging ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   5: TestChangeLogShipperImage ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   6: TestInflightLogRedaction ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   7: TestRebalanceLogProcessing ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   8: TestLoggingDynamicConfigReload ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="Suite Summary (logging)"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg=" ✔ Passes: 8 (100.00%)" 

            prateek.kumar Prateek Kumar added a comment - The fix was tested with Operator Image: cb-rhcc/operator:_2.2.0-232 and Logging Image: _registry.connect.redhat.com/couchbase/fluent-bit:1.0.1-1 on Openshift v4.4. All Logging tests pass. 23 : 20 : 11 --- PASS: TestOperator ( 0 .00s) 23 : 20 : 11 --- PASS: TestOperator/TestNoLogOrAuditConfig ( 43 .82s) 23 : 20 : 11 --- PASS: TestOperator/TestLoggingDynamicConfigReload ( 139 .93s) 23 : 20 : 11 --- PASS: TestOperator/TestLoggingAndAuditingDefaults ( 97 .76s) 23 : 20 : 11 --- PASS: TestOperator/TestRebalanceLogProcessing ( 179 .19s) 23 : 20 : 11 --- PASS: TestOperator/TestInflightLogRedaction ( 214 .61s) 23 : 20 : 11 --- PASS: TestOperator/TestChangeLogShipperImage ( 75 .27s) 23 : 20 : 11 --- PASS: TestOperator/TestAuditingNoLogging ( 46 .64s) 23 : 20 : 11 --- PASS: TestOperator/TestCustomLogging ( 131 .37s) 23 : 20 : 11 PASS 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= "Test Summary" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 1: TestNoLogOrAuditConfig ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 2: TestLoggingAndAuditingDefaults ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 3: TestAuditingNoLogging ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 4: TestCustomLogging ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 5: TestChangeLogShipperImage ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 6: TestInflightLogRedaction ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 7: TestRebalanceLogProcessing ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 8: TestLoggingDynamicConfigReload ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= "Suite Summary (logging)" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " ✔ Passes: 8 (100.00%)"

            Thanks Prateek Kumar, I'll update the RBAC docs for this change and then resolve this issue as complete.

            patrick.stephens Patrick Stephens (Inactive) added a comment - Thanks Prateek Kumar , I'll update the RBAC docs for this change and then resolve this issue as complete.

            After discussion with Simon Murray I've reworked this so we can remove the extra privileges with some caveats that are fully documented.

            patrick.stephens Patrick Stephens (Inactive) added a comment - After discussion with Simon Murray  I've reworked this so we can remove the extra privileges with some caveats that are fully documented.

            The new fix was tested on local and the failing test passes with it.

             === RUN   TestOperator
                === RUN   TestOperator/TestLoggingAndAuditingDefaults
                === PAUSE TestOperator/TestLoggingAndAuditingDefaults
                === CONT  TestOperator/TestLoggingAndAuditingDefaults
                time="2021-05-26T11:21:16+05:30" level=info msg="TestOperator/TestLoggingAndAuditingDefaults ✔"
                --- PASS: TestOperator (0.00s)
                    --- PASS: TestOperator/TestLoggingAndAuditingDefaults (163.13s)
                        crd_util.go:44: creating couchbase cluster: test-couchbase-z8xx9
                PASS
                time="2021-05-26T11:21:18+05:30" level=info msg="Test Summary"
                time="2021-05-26T11:21:18+05:30" level=info msg="   1: TestLoggingAndAuditingDefaults ✔"
                time="2021-05-26T11:21:18+05:30" level=info msg="Suite Summary (custom)"
                time="2021-05-26T11:21:18+05:30" level=info msg=" ✔ Passes: 1 (100.00%)"
                ok  	github.com/couchbase/couchbase-operator/test/e2e	222.436s 

            prateek.kumar Prateek Kumar added a comment - The new fix was tested on local and the failing test passes with it. === RUN TestOperator === RUN TestOperator/TestLoggingAndAuditingDefaults === PAUSE TestOperator/TestLoggingAndAuditingDefaults === CONT TestOperator/TestLoggingAndAuditingDefaults time= "2021-05-26T11:21:16+05:30" level=info msg= "TestOperator/TestLoggingAndAuditingDefaults ✔" --- PASS: TestOperator ( 0 .00s) --- PASS: TestOperator/TestLoggingAndAuditingDefaults ( 163 .13s) crd_util.go: 44 : creating couchbase cluster: test-couchbase-z8xx9 PASS time= "2021-05-26T11:21:18+05:30" level=info msg= "Test Summary" time= "2021-05-26T11:21:18+05:30" level=info msg= " 1: TestLoggingAndAuditingDefaults ✔" time= "2021-05-26T11:21:18+05:30" level=info msg= "Suite Summary (custom)" time= "2021-05-26T11:21:18+05:30" level=info msg= " ✔ Passes: 1 (100.00%)" ok github.com/couchbase/couchbase-operator/test/e2e 222 .436s

            The fix was tested with official fluent-bit image on OCP and the logging tests pass.

            prateek.kumar Prateek Kumar added a comment - The fix was tested with official fluent-bit image on OCP and the logging tests pass.

            People

              prateek.kumar Prateek Kumar
              prateek.kumar Prateek Kumar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty