Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2171

Add worker thread and LUA filter for common info

    XMLWordPrintable

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 2.3.0
    • logging
    • None
    • 1

    Description

      Based on the recommendations and feedback from FluentCon 2021, make the following updates:

      Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput. In fact indicate this is a recommendation for all output plugins (it helps to quickly drop the out-of-order retries in Loki for example). For Loki, probably want to set `retry_limit 0` as well to immediately drop: https://docs.fluentbit.io/manual/administration/scheduling-and-retries#configuring-retries 

      Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            patrick.stephens Patrick Stephens (Inactive) created issue -
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Field Original Value New Value
            Link This issue relates to K8S-2108 [ K8S-2108 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Description Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing.
            Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Link This issue relates to K8S-2172 [ K8S-2172 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Description Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.
            Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput. In fact indicate this is a recommendation for all output plugins (it helps to quickly drop the out-of-order retries in Loki for example).

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Description Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput. In fact indicate this is a recommendation for all output plugins (it helps to quickly drop the out-of-order retries in Loki for example).

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.
            Based on the recommendations and feedback from FluentCon 2021, make the following updates:

            Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput. In fact indicate this is a recommendation for all output plugins (it helps to quickly drop the out-of-order retries in Loki for example). For Loki, probably want to set `retry_limit 0` as well to immediately drop: [https://docs.fluentbit.io/manual/administration/scheduling-and-retries#configuring-retries

            Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.
            simon.murray Simon Murray made changes -
            Fix Version/s not-targeted [ 16613 ]
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Fix Version/s Cthulu [ 17459 ]
            Fix Version/s not-targeted [ 16613 ]
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked higher
            simon.murray Simon Murray made changes -
            Rank Ranked lower
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            patrick.stephens Patrick Stephens (Inactive) made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            simon.murray Simon Murray made changes -
            Fix Version/s 2.3.0 [ 17600 ]
            Fix Version/s Cthulhu [ 17459 ]

            People

              patrick.stephens Patrick Stephens (Inactive)
              patrick.stephens Patrick Stephens (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty