Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2210

[Fluent-bit] Cannot set an ownerRef on a resource you can't delete

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.2.0
    • documentation, logging, operator
    • None
    • 20: PE/Docs/Cleanup, 22: Docs/Cleanup/CI
    • 1

    Description

      Job: http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-oc-pipeline/114/

      Platform: OCP v4.4

      Logging Image: registry.connect.redhat.com/couchbase/fluent-bit:1.0.1-1

      Operator Image: registry.gitlab.com/cb-rhcc/operator:latest  {build 229}

      TestCase: TestLoggingAndAuditingDefaults 

      BackTrace:

      03:56:18     util.go:1288: timeout: size does not match, wanted 1, got 0
      03:56:18     util.go:1289: goroutine 1135 [running]:
      03:56:18         runtime/debug.Stack(0x1f09914, 0xc0012fc558, 0xc0012fc550)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/go/src/runtime/debug/stack.go:24 +0xab
      03:56:18         github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc0009ec780, 0x3525440, 0xc0006717e0)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/util.go:1284 +0x34
      03:56:18         github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustWaitClusterStatusHealthy(0xc0009ec780, 0xc000592380, 0xc000a88000, 0xd18c2e2800)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/wait_util.go:580 +0x94
      03:56:18         github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustNewClusterFromSpec(0xc0009ec780, 0xc000592380, 0xc000b02500, 0x11)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/util.go:88 +0x9d
      03:56:18         github.com/couchbase/couchbase-operator/test/e2e.TestLoggingAndAuditingDefaults(0xc0009ec780)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/logging_test.go:72 +0xbe5
      03:56:18         testing.tRunner(0xc0009ec780, 0x2b1c328)
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/go/src/testing/testing.go:1193 +0x203
      03:56:18         created by testing.(*T).Run
      03:56:18         	/jenkins/workspace/k8s-cbop-oc-pipeline/go/src/testing/testing.go:1238 +0x5d8
      03:56:18         
      03:56:18 time="2021-05-24T03:56:12-07:00" level=info msg="TestOperator/TestLoggingAndAuditingDefaults ✗" 

      Error:

      "error":"secrets \"fluent-bit-config\" is forbidden: cannot set an ownerRef on a resource you can't delete: 

      (cbopinfo attached)

      The same test was tested with

      Logging Image: registry.gitlab.com/cb-rhcc/fluent-bit:1.0.3-103

      It failed with this version as well with similar error as described above.

      Job: http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-oc-pipeline/115/console

      (cbopinfo attached, having 1.0.3 in the name)

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            prateek.kumar Prateek Kumar created issue -
            prateek.kumar Prateek Kumar made changes -
            Field Original Value New Value
            Description *Job*: [http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-oc-pipeline/114/]

            *Logging Image*: registry.connect.redhat.com/couchbase/fluent-bit:1.0.1-1

            *Operator Image*: registry.gitlab.com/cb-rhcc/operator:latest  \{build 229}

            *TestCase*: TestLoggingAndAuditingDefaults 

            BackTrace:
            {code:java}
            03:56:18 util.go:1288: timeout: size does not match, wanted 1, got 0
            03:56:18 util.go:1289: goroutine 1135 [running]:
            03:56:18 runtime/debug.Stack(0x1f09914, 0xc0012fc558, 0xc0012fc550)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/go/src/runtime/debug/stack.go:24 +0xab
            03:56:18 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc0009ec780, 0x3525440, 0xc0006717e0)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/util.go:1284 +0x34
            03:56:18 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustWaitClusterStatusHealthy(0xc0009ec780, 0xc000592380, 0xc000a88000, 0xd18c2e2800)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/wait_util.go:580 +0x94
            03:56:18 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustNewClusterFromSpec(0xc0009ec780, 0xc000592380, 0xc000b02500, 0x11)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/util.go:88 +0x9d
            03:56:18 github.com/couchbase/couchbase-operator/test/e2e.TestLoggingAndAuditingDefaults(0xc0009ec780)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/logging_test.go:72 +0xbe5
            03:56:18 testing.tRunner(0xc0009ec780, 0x2b1c328)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/go/src/testing/testing.go:1193 +0x203
            03:56:18 created by testing.(*T).Run
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/go/src/testing/testing.go:1238 +0x5d8
            03:56:18
            03:56:18 time="2021-05-24T03:56:12-07:00" level=info msg="TestOperator/TestLoggingAndAuditingDefaults ✗" {code}
            Error:
            {code:java}
            "error":"secrets \"fluent-bit-config\" is forbidden: cannot set an ownerRef on a resource you can't delete: {code}
            (cbopinfo attached)

            The same test was tested with

            *Logging Image*: registry.gitlab.com/cb-rhcc/fluent-bit:1.0.3-103

            It failed with this version as well with similar error as described above.

            *Job*: [http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-oc-pipeline/115/console]

            (cbopinfo attached, having 1.0.3 in the name)
            *Job*: [http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-oc-pipeline/114/]

            *Platform*: OCP v4.4

            *Logging Image*: registry.connect.redhat.com/couchbase/fluent-bit:1.0.1-1

            *Operator Image*: registry.gitlab.com/cb-rhcc/operator:latest  \{build 229}

            *TestCase*: TestLoggingAndAuditingDefaults 

            BackTrace:
            {code:java}03:56:18 util.go:1288: timeout: size does not match, wanted 1, got 0
            03:56:18 util.go:1289: goroutine 1135 [running]:
            03:56:18 runtime/debug.Stack(0x1f09914, 0xc0012fc558, 0xc0012fc550)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/go/src/runtime/debug/stack.go:24 +0xab
            03:56:18 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.Die(0xc0009ec780, 0x3525440, 0xc0006717e0)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/util.go:1284 +0x34
            03:56:18 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustWaitClusterStatusHealthy(0xc0009ec780, 0xc000592380, 0xc000a88000, 0xd18c2e2800)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/wait_util.go:580 +0x94
            03:56:18 github.com/couchbase/couchbase-operator/test/e2e/e2eutil.MustNewClusterFromSpec(0xc0009ec780, 0xc000592380, 0xc000b02500, 0x11)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/e2eutil/util.go:88 +0x9d
            03:56:18 github.com/couchbase/couchbase-operator/test/e2e.TestLoggingAndAuditingDefaults(0xc0009ec780)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/test/e2e/logging_test.go:72 +0xbe5
            03:56:18 testing.tRunner(0xc0009ec780, 0x2b1c328)
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/go/src/testing/testing.go:1193 +0x203
            03:56:18 created by testing.(*T).Run
            03:56:18 /jenkins/workspace/k8s-cbop-oc-pipeline/go/src/testing/testing.go:1238 +0x5d8
            03:56:18
            03:56:18 time="2021-05-24T03:56:12-07:00" level=info msg="TestOperator/TestLoggingAndAuditingDefaults ✗" {code}
            Error:
            {code:java}"error":"secrets \"fluent-bit-config\" is forbidden: cannot set an ownerRef on a resource you can't delete: {code}
            (cbopinfo attached)

            The same test was tested with

            *Logging Image*: registry.gitlab.com/cb-rhcc/fluent-bit:1.0.3-103

            It failed with this version as well with similar error as described above.

            *Job*: [http://qa.sc.couchbase.com/view/Cloud/job/k8s-cbop-oc-pipeline/115/console]

            (cbopinfo attached, having 1.0.3 in the name)
            prateek.kumar Prateek Kumar made changes -

            The three tests the use default log configuration are:

            1. TestChangeLogShipperImage
            2. TestRebalanceLogProcessing
            3. TestLoggingAndAuditingDefaults

            Of these, only the failing TestLoggingAndAuditingDefaults creates a dodgy configuration first prior to creating the cluster to demonstrate that it is updated during reconcile.

            patrick.stephens Patrick Stephens (Inactive) added a comment - The three tests the use default log configuration are: TestChangeLogShipperImage TestRebalanceLogProcessing TestLoggingAndAuditingDefaults Of these, only the failing TestLoggingAndAuditingDefaults creates a dodgy configuration first prior to creating the cluster to demonstrate that it is updated during reconcile.
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Component/s logging [ 16330 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Sprint 20: PE/Docs/Cleanup [ 1595 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Rank Ranked higher
            prateek.kumar Prateek Kumar made changes -

            The fix was tested with Operator Image: cb-rhcc/operator:_2.2.0-232 and Logging Image: _registry.connect.redhat.com/couchbase/fluent-bit:1.0.1-1 on Openshift v4.4. All Logging tests pass.

            23:20:11 --- PASS: TestOperator (0.00s)
            23:20:11     --- PASS: TestOperator/TestNoLogOrAuditConfig (43.82s)
            23:20:11     --- PASS: TestOperator/TestLoggingDynamicConfigReload (139.93s)
            23:20:11     --- PASS: TestOperator/TestLoggingAndAuditingDefaults (97.76s)
            23:20:11     --- PASS: TestOperator/TestRebalanceLogProcessing (179.19s)
            23:20:11     --- PASS: TestOperator/TestInflightLogRedaction (214.61s)
            23:20:11     --- PASS: TestOperator/TestChangeLogShipperImage (75.27s)
            23:20:11     --- PASS: TestOperator/TestAuditingNoLogging (46.64s)
            23:20:11     --- PASS: TestOperator/TestCustomLogging (131.37s)
            23:20:11 PASS
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="Test Summary"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   1: TestNoLogOrAuditConfig ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   2: TestLoggingAndAuditingDefaults ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   3: TestAuditingNoLogging ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   4: TestCustomLogging ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   5: TestChangeLogShipperImage ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   6: TestInflightLogRedaction ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   7: TestRebalanceLogProcessing ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="   8: TestLoggingDynamicConfigReload ✔"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg="Suite Summary (logging)"
            23:20:11 time="2021-05-24T23:20:08-07:00" level=info msg=" ✔ Passes: 8 (100.00%)" 

            prateek.kumar Prateek Kumar added a comment - The fix was tested with Operator Image: cb-rhcc/operator:_2.2.0-232 and Logging Image: _registry.connect.redhat.com/couchbase/fluent-bit:1.0.1-1 on Openshift v4.4. All Logging tests pass. 23 : 20 : 11 --- PASS: TestOperator ( 0 .00s) 23 : 20 : 11 --- PASS: TestOperator/TestNoLogOrAuditConfig ( 43 .82s) 23 : 20 : 11 --- PASS: TestOperator/TestLoggingDynamicConfigReload ( 139 .93s) 23 : 20 : 11 --- PASS: TestOperator/TestLoggingAndAuditingDefaults ( 97 .76s) 23 : 20 : 11 --- PASS: TestOperator/TestRebalanceLogProcessing ( 179 .19s) 23 : 20 : 11 --- PASS: TestOperator/TestInflightLogRedaction ( 214 .61s) 23 : 20 : 11 --- PASS: TestOperator/TestChangeLogShipperImage ( 75 .27s) 23 : 20 : 11 --- PASS: TestOperator/TestAuditingNoLogging ( 46 .64s) 23 : 20 : 11 --- PASS: TestOperator/TestCustomLogging ( 131 .37s) 23 : 20 : 11 PASS 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= "Test Summary" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 1: TestNoLogOrAuditConfig ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 2: TestLoggingAndAuditingDefaults ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 3: TestAuditingNoLogging ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 4: TestCustomLogging ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 5: TestChangeLogShipperImage ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 6: TestInflightLogRedaction ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 7: TestRebalanceLogProcessing ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " 8: TestLoggingDynamicConfigReload ✔" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= "Suite Summary (logging)" 23 : 20 : 11 time= "2021-05-24T23:20:08-07:00" level=info msg= " ✔ Passes: 8 (100.00%)"

            Thanks Prateek Kumar, I'll update the RBAC docs for this change and then resolve this issue as complete.

            patrick.stephens Patrick Stephens (Inactive) added a comment - Thanks Prateek Kumar , I'll update the RBAC docs for this change and then resolve this issue as complete.
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Component/s documentation [ 14113 ]

            After discussion with Simon Murray I've reworked this so we can remove the extra privileges with some caveats that are fully documented.

            patrick.stephens Patrick Stephens (Inactive) added a comment - After discussion with Simon Murray  I've reworked this so we can remove the extra privileges with some caveats that are fully documented.
            ingenthr Matt Ingenthron made changes -
            Sprint 20: PE/Docs/Cleanup [ 1595 ] 20: PE/Docs/Cleanup, 22: Docs/Cleanup/CI [ 1595, 1610 ]

            The new fix was tested on local and the failing test passes with it.

             === RUN   TestOperator
                === RUN   TestOperator/TestLoggingAndAuditingDefaults
                === PAUSE TestOperator/TestLoggingAndAuditingDefaults
                === CONT  TestOperator/TestLoggingAndAuditingDefaults
                time="2021-05-26T11:21:16+05:30" level=info msg="TestOperator/TestLoggingAndAuditingDefaults ✔"
                --- PASS: TestOperator (0.00s)
                    --- PASS: TestOperator/TestLoggingAndAuditingDefaults (163.13s)
                        crd_util.go:44: creating couchbase cluster: test-couchbase-z8xx9
                PASS
                time="2021-05-26T11:21:18+05:30" level=info msg="Test Summary"
                time="2021-05-26T11:21:18+05:30" level=info msg="   1: TestLoggingAndAuditingDefaults ✔"
                time="2021-05-26T11:21:18+05:30" level=info msg="Suite Summary (custom)"
                time="2021-05-26T11:21:18+05:30" level=info msg=" ✔ Passes: 1 (100.00%)"
                ok  	github.com/couchbase/couchbase-operator/test/e2e	222.436s 

            prateek.kumar Prateek Kumar added a comment - The new fix was tested on local and the failing test passes with it. === RUN TestOperator === RUN TestOperator/TestLoggingAndAuditingDefaults === PAUSE TestOperator/TestLoggingAndAuditingDefaults === CONT TestOperator/TestLoggingAndAuditingDefaults time= "2021-05-26T11:21:16+05:30" level=info msg= "TestOperator/TestLoggingAndAuditingDefaults ✔" --- PASS: TestOperator ( 0 .00s) --- PASS: TestOperator/TestLoggingAndAuditingDefaults ( 163 .13s) crd_util.go: 44 : creating couchbase cluster: test-couchbase-z8xx9 PASS time= "2021-05-26T11:21:18+05:30" level=info msg= "Test Summary" time= "2021-05-26T11:21:18+05:30" level=info msg= " 1: TestLoggingAndAuditingDefaults ✔" time= "2021-05-26T11:21:18+05:30" level=info msg= "Suite Summary (custom)" time= "2021-05-26T11:21:18+05:30" level=info msg= " ✔ Passes: 1 (100.00%)" ok github.com/couchbase/couchbase-operator/test/e2e 222 .436s
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            simon.murray Simon Murray made changes -
            Rank Ranked higher

            The fix was tested with official fluent-bit image on OCP and the logging tests pass.

            prateek.kumar Prateek Kumar added a comment - The fix was tested with official fluent-bit image on OCP and the logging tests pass.
            prateek.kumar Prateek Kumar made changes -
            Assignee Patrick Stephens [ JIRAUSER25332 ] Prateek Kumar [ prateek.kumar ]
            Status Resolved [ 5 ] Closed [ 6 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Link This issue relates to K8S-1734 [ K8S-1734 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Link This issue relates to K8S-2322 [ K8S-2322 ]

            People

              prateek.kumar Prateek Kumar
              prateek.kumar Prateek Kumar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty