Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2493

Update log forwarding tutorial for Loki simplification

    XMLWordPrintable

Details

    • Page
    • Status: Resolved
    • Minor
    • Resolution: Resolved
    • None
    • 2.3.0
    • documentation
    • None
    • 5: Helm, backup, Marketplace, 1: Recovery to productivity, 3: SBEE, Multi-Cert
    • 1

    Description

      With Fluent Bit 1.1.2, there is no need to use a custom secret to control forwarding to Loki - this can be achieved by updating the pod template spec in the cluster config to provide some special annotations: https://docs-staging.couchbase.com/operator/current/tutorial-couchbase-log-forwarding.html#example-loki-stack

      As a simple example: https://github.com/couchbase/couchbase-fluent-bit/blob/3c10f93ef47443957edd738bfff72eb9e25ceaef/tools/loki-k8s-stack/values.yaml#L12-L18

            pod:
              metadata:
                annotations:
                  # Match all logs
                  fluentbit.couchbase.com/loki_match: "*"
                  # Send to this SVC
                  fluentbit.couchbase.com/loki_host: loki.monitoring
      

      This basically says match all logs (change that wildcard to just what you want, e.g. couchbase.log.audit) and send to that hostname (in cluster but separate namespace here).

      Here is an example that creates everything from scratch including the Grafana-Loki stack in a separate namespace: https://github.com/couchbase/couchbase-fluent-bit/tree/main/tools/loki-k8s-stack

      You can of course still do a full custom configuration via a K8S Secret. However this approach works with the default configuration and means people don’t need to manage Secrets. It also means we can easily integrate CMOS by just setting those annotations – potentially we could even do this when we deploy CMOS to update the pod specs automatically.

      We should highlight this example more heavily across the board as it will be the best way to support CMOS integration as well.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            patrick.stephens Patrick Stephens (Inactive) created issue -
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Field Original Value New Value
            Link This issue depends on CBD-4444 [ CBD-4444 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Description With Fluent Bit 1.1.2, there is no need to use a custom secret to control forwarding to Loki - this can be achieved by updating the pod template spec in the cluster config to provide some special annotations:

             
            With Fluent Bit 1.1.2, there is no need to use a custom secret to control forwarding to Loki - this can be achieved by updating the pod template spec in the cluster config to provide some special annotations: https://docs-staging.couchbase.com/operator/current/tutorial-couchbase-log-forwarding.html#example-loki-stack

            As a simple example: [https://github.com/couchbase/couchbase-fluent-bit/blob/3c10f93ef47443957edd738bfff72eb9e25ceaef/tools/loki-k8s-stack/values.yaml#L12-L18]

            {code:yaml}
                  pod:
                    metadata:
                      annotations:
                        # Match all logs
                        fluentbit.couchbase.com/loki_match: "*"
                        # Send to this SVC
                        fluentbit.couchbase.com/loki_host: loki.monitoring
            {code}

            This basically says match all logs (change that wildcard to just what you want, e.g. couchbase.log.audit) and send to that hostname (in cluster but separate namespace here).

            Here is an example that creates everything from scratch including the Grafana-Loki stack in a separate namespace: [https://github.com/couchbase/couchbase-fluent-bit/tree/main/tools/loki-k8s-stack]

            You can of course still do a full custom configuration via a K8S Secret. However this approach works with the default configuration and means people don’t need to manage Secrets. It also means we can easily integrate CMOS by just setting those annotations – potentially we could even do this when we deploy CMOS to update the pod specs automatically.

            We should highlight this example more heavily across the board as it will be the best way to support CMOS integration as well.
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Link This issue relates to K8S-2494 [ K8S-2494 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Assignee Patrick Stephens [ JIRAUSER25332 ] Alex Emery [ JIRAUSER26037 ]
            simon.murray Simon Murray made changes -
            Component/s logging [ 16330 ]
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            ingenthr Matt Ingenthron made changes -
            Sprint 0 [ 1891 ]
            ingenthr Matt Ingenthron made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Sprint 1: Recovery to productivity [ 1891 ] 5: Automation, upgrades 2, 1: Recovery to productivity [ 1534, 1891 ]
            simon.murray Simon Murray made changes -
            Sprint 5: Automation, upgrades 2, 1: Recovery to productivity [ 1534, 1891 ] 1: Recovery to productivity, 3: SBEE, Multi-Cert [ 1891, 1951 ]
            simon.murray Simon Murray made changes -
            Rank Ranked lower
            Alex.emery Alex Emery made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            ingenthr Matt Ingenthron made changes -
            Sprint 1: Recovery to productivity, 3: SBEE, Multi-Cert [ 1891, 1951 ] 5: Automation, upgrades 2, 1: Recovery to productivity, 3: SBEE, Multi-Cert [ 1534, 1891, 1951 ]
            simon.murray Simon Murray made changes -
            Rank Ranked lower

            Build couchbase-operator-2.3.0-247 contains couchbase-operator commit 8c988f5 with commit message:
            K8S-2493: Logging docs update

            build-team Couchbase Build Team added a comment - Build couchbase-operator-2.3.0-247 contains couchbase-operator commit 8c988f5 with commit message: K8S-2493 : Logging docs update
            Alex.emery Alex Emery made changes -
            Resolution Resolved [ 10400 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            arunkumar Arunkumar Senthilnathan (Inactive) made changes -
            Assignee Alex Emery [ JIRAUSER26037 ] Roo Thorp [ JIRAUSER25108 ]

            Roo Thorp  please verify and close this

            arunkumar Arunkumar Senthilnathan (Inactive) added a comment - Roo Thorp   please verify and close this

            People

              roo.thorp Roo Thorp
              patrick.stephens Patrick Stephens (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty