Description
Based on the recommendations and feedback from FluentCon 2021, make the following updates:
Add a dedicated worker for the `stdout` plugin to ensure we maximise throughput. In fact indicate this is a recommendation for all output plugins (it helps to quickly drop the out-of-order retries in Loki for example). For Loki, probably want to set `retry_limit 0` as well to immediately drop: https://docs.fluentbit.io/manual/administration/scheduling-and-retries#configuring-retries
Use a LUA filter to pick up any and all environment variables matching annotations/labels applied by CAO. Current approach uses a modify-nest filter pipeline but modify only works for valid variables (if not present it exits as a failure) and we just want to add any/all available (if not available then not an issue) plus ensure future-proofing. All the feedback from yesterday shows LUA is both powerful and performant although we likely want to monitor as well.
Attachments
Issue Links
For Gerrit Dashboard: K8S-2171 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
155168,4 | K8S-2171: Resolve issues with missing variables in non CAO use cases | main | couchbase-fluent-bit | Status: MERGED | +2 | +1 |
155183,4 | K8S-2171: Extend tests to check for extra info | master | couchbase-operator | Status: MERGED | +2 | +1 |