Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2171

Add worker thread and LUA filter for common info

    XMLWordPrintable

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 2.3.0
    • logging
    • None
    • 1

    Description

      Based on the recommendations and feedback from FluentCon 2021, make the following updates:

      Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput. In fact indicate this is a recommendation for all output plugins (it helps to quickly drop the out-of-order retries in Loki for example). For Loki, probably want to set `retry_limit 0` as well to immediately drop: https://docs.fluentbit.io/manual/administration/scheduling-and-retries#configuring-retries 

      Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.

      Attachments

        Issue Links

          For Gerrit Dashboard: K8S-2171
          # Subject Branch Project Status CR V

          Activity

            patrick.stephens Patrick Stephens (Inactive) created issue -
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Field Original Value New Value
            Link This issue relates to K8S-2108 [ K8S-2108 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Description Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing.
            Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Link This issue relates to K8S-2172 [ K8S-2172 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Description Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.
            Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput. In fact indicate this is a recommendation for all output plugins (it helps to quickly drop the out-of-order retries in Loki for example).

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Description Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput. In fact indicate this is a recommendation for all output plugins (it helps to quickly drop the out-of-order retries in Loki for example).

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.
            Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput. In fact indicate this is a recommendation for all output plugins (it helps to quickly drop the out-of-order retries in Loki for example). For Loki, probably want to set `retry_limit 0` as well to immediately drop: [https://docs.fluentbit.io/manual/administration/scheduling-and-retries#configuring-retries

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.
            simon.murray Simon Murray made changes -
            Fix Version/s not-targeted [ 16613 ]
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Fix Version/s Cthulu [ 17459 ]
            Fix Version/s not-targeted [ 16613 ]
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked lower

            No need for LUA and actually LUA does not give you an easy way to get a list of all environment variables: you can get named ones with a simple call but to get the full environment requires some C-shims which is not ideal particularly for a container.

            The `record_modifier` filter actually allows you to specify a superset of those variables you want: like the `modify` one it reports an error for any missing but carries on working rather than triggering an exit. This seems a better option as long as we document that the errors for missing variables can be ignored when they're not defined (e.g. when run outside kubernetes or for those configurations missing some of the services on a node - as per best practice).

            patrick.stephens Patrick Stephens (Inactive) added a comment - No need for LUA and actually LUA does not give you an easy way to get a list of all environment variables: you can get named ones with a simple call but to get the full environment requires some C-shims which is not ideal particularly for a container. The `record_modifier` filter actually allows you to specify a superset of those variables you want: like the `modify` one it reports an error for any missing but carries on working rather than triggering an exit. This seems a better option as long as we document that the errors for missing variables can be ignored when they're not defined (e.g. when run outside kubernetes or for those configurations missing some of the services on a node - as per best practice).
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]

            Update default image used by operator to latest fluent bit release.

            patrick.stephens Patrick Stephens (Inactive) added a comment - Update default image used by operator to latest fluent bit release.
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]

            Released in couchbase/fluent-bit:1.0.4

            patrick.stephens Patrick Stephens (Inactive) added a comment - Released in couchbase/fluent-bit:1.0.4
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            simon.murray Simon Murray made changes -
            Fix Version/s 2.3.0 [ 17600 ]
            Fix Version/s Cthulhu [ 17459 ]

            People

              patrick.stephens Patrick Stephens (Inactive)
              patrick.stephens Patrick Stephens (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty