Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2810

Fluent-bit watcher crashing

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.3.2
    • logging
    • None
    • 30: Maintenance, CMOS, ARM
    • 1

    Description

      Appears to be a regression in fluent bit where we've changed the base image to improve security (K8S-2804)

      There appears to be a different opening log from standard base image followed by crash:

      "level":"info","ts":1660178169.4961982,"logger":"couchbase-watcher","caller":"common/common.go:273","msg":"No environment variable so defaulting","environmentVariable":"STORAGE_BUFFER_PATH","defaultValue":"/tmp/buffer"}{"level":"info","ts":1660178169.4962957,"logger":"couchbase-watcher","caller":"common/common.go:143","msg":"Loaded environment files”}
       
      ...
       
       
      ","stacktrace":"github.com/couchbase/fluent-bit/pkg/fluent.Start\n\tgithub.com/couchbase/fluent-bit/pkg/fluent/watcher.go:122
      github.com/couchbase/fluent-bit/pkg/fluent.addFluentBitWatcher.func1	github.com/couchbase/fluent-bit/pkg/fluent/watcher.go:234\ngithub.com/oklog/run.(*Group).Run.func1	github.com/oklog/run@v1.1.0/group.go:38"}{"level":"info","ts":1660170655.0074291,"logger":"couchbase-watcher","caller":"fluent/watcher.go:237","msg":"Detected exit of Fluent Bit so backoff”}

       

      As a result fail with the following error because the expected output is non-existent:

      util.go:1390: timeout: logs do not contain "[0] couchbase.log." for pod "test-couchbase-jg22t-0000" 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          With QE folks out, Tommie McAfee is going to take on running the tests and verifying Alex's fix. Please add any detail and move to closed when done Tommie McAfee.

          ingenthr Matt Ingenthron added a comment - With QE folks out, Tommie McAfee is going to take on running the tests and verifying Alex's fix. Please add any detail and move to closed when done Tommie McAfee .

          Build couchbase-fluent-bit-1.2.1-103 contains couchbase-fluent-bit commit d35c016 with commit message:
          K8S-2810: fluent-bit watcher crashes

          build-team Couchbase Build Team added a comment - Build couchbase-fluent-bit-1.2.1-103 contains couchbase-fluent-bit commit d35c016 with commit message: K8S-2810 : fluent-bit watcher crashes

          Verified that 1.2.1-103 fixes this issue, testing now passes

          INFO[0199] TestOperator/TestInflightLogRedaction ✔
          --- PASS: TestOperator (0.00s)
              --- PASS: TestOperator/TestInflightLogRedaction (163.46s) 

           

          tommie Tommie McAfee added a comment - Verified that 1.2.1-103 fixes this issue, testing now passes INFO[ 0199 ] TestOperator/TestInflightLogRedaction ✔ --- PASS: TestOperator ( 0 .00s)     --- PASS: TestOperator/TestInflightLogRedaction ( 163 .46s)  

          People

            tommie Tommie McAfee
            tommie Tommie McAfee
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty