Description
Various improvements found that may be useful during testing:
Include cluster name when enriching the log dataReduce the tail refresh interval (currently 60 seconds) so it picks up logs sooner - the container starts quickly but if the log directory or rebalance is not present then can take a while once server starts so we may lose logs on a quick failure.Provide full integration tests with CIDefault Loki output - need to make sure no impact on customer usage, ideally a simple method to enable during testing but with a managed config. Relates toK8S-2112Look to see if we can provide counters for various errors and/or prometheus metrics (optionally): coming soon in FB but also see https://github.com/neiman-marcus/fluent-bit-out-prometheus-metricsAdd Docker-compose stack as an example for local usage.Reduce cyclometric complexity and refactor watcher to simplify, standardise logging as per operator too.Add unit tests for watcher functionality - all covered by integration tests currently so shift left if possible.Document GKE set up - issues with Autopilot and Promtail. Stalling of loki input. Working now it seems without PV: https://github.com/patrick-stephens/couchbase-gitops/tree/main/gkeRewrite the 4 letter level names for java logs. Good example of sorting case out as well: https://github.com/sassoftware/viya4-monitoring-kubernetes/blob/eaaf0498f835cbabbcf9f55715ddeafae2d68ca5/logging/fb/fluent-bit_config.configmap_open.yaml#L731- Ensure we test mount path changes, i.e. that we pick up the config from there/mount it in and can watch for changes:
K8S-2324
Attachments
Issue Links
- relates to
-
K8S-2324 Changing mount path for logging sidecar prevents dynamic reload
-
- Resolved
-
-
-
- Resolved
-
-
K8S-2112 Fluent bit parsing error with json_rpc logs
-
- Resolved
-
-
K8S-2172 Add Mem_Buf_Limit setting for each input
-
- In Progress
-
-
-
- Closed
-
| For Gerrit Dashboard: K8S-2108 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 150856,7 | K8S-2108: Refactor and unit tests | main | couchbase-fluent-bit | Status: MERGED | +2 | +1 |
| 151037,2 | K8S-2108: support updates to log streaming sidecar | master | couchbase-operator | Status: MERGED | +2 | +1 |
| 151122,3 | K8S-2108: Add information for release tagging and branching | main | couchbase-fluent-bit | Status: MERGED | +2 | +1 |
| 151522,3 | K8S-2108: Reduce refresh interval | main | couchbase-fluent-bit | Status: MERGED | +2 | +1 |
| 151530,2 | K8S-2108: Hadolint fixes | main | couchbase-fluent-bit | Status: MERGED | +2 | +1 |
| 151531,8 | K8S-2108: Jenkins integration for Docker | main | couchbase-fluent-bit | Status: MERGED | +2 | +1 |
| 151988,7 | K8S-2108: Resolve log level issues and improve flexibility | main | couchbase-fluent-bit | Status: MERGED | +2 | +1 |
| 152215,4 | K8S-2108: Addition of parser suggestion and cluster name to sidecar | master | couchbase-operator | Status: MERGED | +2 | +1 |
| 152236,8 | K8s_2108: Insert cluster name into record | main | couchbase-fluent-bit | Status: MERGED | +2 | +1 |
| 152582,9 | K8S-2108: Add CI support for Jenkins with operator repo | main | couchbase-fluent-bit | Status: MERGED | +2 | +1 |
Activity
Be good to provide a parser configuration that works for the daemonset as well (or PromTail). This should be more straightforward as we're just capturing the JSON output from stdout.
Possible option to mangle timestamps for out-of-order issues with Loki: https://github.com/fluent/fluent-bit/issues/2015#issuecomment-627547843
Fluent bit 1.8.2 now drops retrying a chunk as soon as it gets an out-of-order response from Loki.
Hopefully this will be resolved at some point on the Loki side: https://github.com/grafana/loki/issues/1544
It would be nice to be able to move certain bits of the user documentation to the couchbase-fluent-bit repo. This would Couchbase-specific documentation (i.e., not directed at the open source audience) that would be pulled into the Operator docs. It will likely be mostly reference-type documentation, such as compatibility tables, marked-up snippets of config files, etc.
Our docs site generator — Antora — supports using tags instead of branches for version control of the docs. So if we want to move some doc content to the fluent bit repo, then we need to make sure that the docs are considered when tagging a release. We can control how the semver appears as in the docs, but we need to make sure that docs updates go in before the release is tagged, since there will eventually be releases that are incompatible with older/newer versions of the Operator, and we need to be able to pull relevant docs into each version of the Operator docs.