Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2112

Fluent bit parsing error with json_rpc logs

    XMLWordPrintable

Details

    • Task
    • Resolution: Not a Bug
    • Minor
    • 2.2.0
    • None
    • operator
    • None
    • 1

    Description

      It looks like the json_rpc.log is causing some issues with its multiline parsing. Occasionally this seems to get out of sync so we keep ending up with stdout lines that are from a multiline message but just as individual randoms.

      It may also be related to embedding new line characters in the log message as well, ideally we should remove these.

      This was discovered when deploying the Loki stack which allowed us to filter the logs and visualise them a lot more: https://grafana.com/docs/loki/latest/installation/helm/#deploy-loki-stack-loki-fluent-bit-grafana-prometheus

      helm repo add grafana https://grafana.github.io/helm-charts
      kubectl create namespace logging
      helm upgrade --install loki --namespace=logging grafana/loki-stack   --set fluent-bit.enabled=false,promtail.enabled=true,grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false
      

      Attachments

        1. Screenshot 2021-04-08 at 16.52.30.png
          Screenshot 2021-04-08 at 16.52.30.png
          141 kB
        2. json_rpc.log
          761 kB
        3. Grafana-logs-dashboard.json
          6 kB
        4. all-logs.tgz
          689 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              patrick.stephens Patrick Stephens (Inactive)
              patrick.stephens Patrick Stephens (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty