Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2112

Fluent bit parsing error with json_rpc logs

    XMLWordPrintable

Details

    • Task
    • Status: Resolved
    • Minor
    • Resolution: Not a Bug
    • None
    • 2.2.0
    • operator
    • None
    • 1

    Description

      It looks like the json_rpc.log is causing some issues with its multiline parsing. Occasionally this seems to get out of sync so we keep ending up with stdout lines that are from a multiline message but just as individual randoms.

      It may also be related to embedding new line characters in the log message as well, ideally we should remove these.

      This was discovered when deploying the Loki stack which allowed us to filter the logs and visualise them a lot more: https://grafana.com/docs/loki/latest/installation/helm/#deploy-loki-stack-loki-fluent-bit-grafana-prometheus

      helm repo add grafana https://grafana.github.io/helm-charts
      kubectl create namespace logging
      helm upgrade --install loki --namespace=logging grafana/loki-stack   --set fluent-bit.enabled=false,promtail.enabled=true,grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false
      

      Attachments

        1. all-logs.tgz
          689 kB
        2. Grafana-logs-dashboard.json
          6 kB
        3. json_rpc.log
          761 kB
        4. Screenshot 2021-04-08 at 16.52.30.png
          Screenshot 2021-04-08 at 16.52.30.png
          141 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Actually looking at this, it might be down to the PromTail configuration not coping with multiline output. Testing shows that Fluent Bit seems to be ok but it has to output over multiple lines as per the original message although it is encapsulated within a single FB record.

            See https://stackoverflow.com/a/66686871 

            [1441] couchbase.log.json_rpc: [1617895514.962000000, {"filename"=>"/opt/couchbase/var/couchbase/logs/json_rpc.log", "logger"=>"json_rpc", "level"=>"debug", "timestamp"=>"2021-04-08T15:25:14.962Z", "message"=>"ns_1@cb-example-0002.cb-example.default.svc:json_rpc_connection-index-service_api<0.1467.0>:json_rpc_connection:handle_call:158]sending jsonrpc call:{[{jsonrpc,<<"2.0">>},
                                   {id,151},
                                   {method,<<"ServiceAPI.GetTaskList">>},
                                   {params,[{[{rev,<<"AAAAAAAAAAI=">>},
                                              {timeout,30000}]}]}]}", "pod"=>"47232f7c3803", "logshipper"=>"couchbase.sidecar.fluentbit"}]
            [1442] couchbase.log.json_rpc: [1617895514.965000000, {"filename"=>"/opt/couchbase/var/couchbase/logs/json_rpc.log", "logger"=>"json_rpc", "level"=>"debug", "timestamp"=>"2021-04-08T15:25:14.965Z", "message"=>"ns_1@cb-example-0002.cb-example.default.svc:json_rpc_connection-index-service_api<0.1467.0>:json_rpc_connection:handle_info:94]got response: [{<<"id">>,150},
                           {<<"result">>,
                            {[{<<"rev">>,<<"AAAAAAAAAAI=">>},
                              {<<"nodes">>,
                               [<<"c772ddec75b1d5cadc6288eb5409ec98">>,
                                <<"a5444727f282c01b6ef1bb0ec43a4ef0">>,
                                <<"91e4269cb6a425eab9358e99151a3357">>]},
                              {<<"isBalanced">>,true}]}},
                           {<<"error">>,null}]", "pod"=>"47232f7c3803", "logshipper"=>"couchbase.sidecar.fluentbit"}]
            [1443] couchbase.log.json_rpc
            

            patrick.stephens Patrick Stephens (Inactive) added a comment - - edited Actually looking at this, it might be down to the PromTail configuration not coping with multiline output. Testing shows that Fluent Bit seems to be ok but it has to output over multiple lines as per the original message although it is encapsulated within a single FB record. See https://stackoverflow.com/a/66686871   [ 1441 ] couchbase.log.json_rpc: [ 1617895514.962000000 , { "filename" => "/opt/couchbase/var/couchbase/logs/json_rpc.log" , "logger" => "json_rpc" , "level" => "debug" , "timestamp" => "2021-04-08T15:25:14.962Z" , "message" => "ns_1@cb-example-0002.cb-example.default.svc:json_rpc_connection-index-service_api<0.1467.0>:json_rpc_connection:handle_call:158]sending jsonrpc call:{[{jsonrpc,<<" 2.0 ">>},                        {id, 151 },                        {method,<< "ServiceAPI.GetTaskList" >>},                        {params,[{[{rev,<< "AAAAAAAAAAI=" >>},                                   {timeout, 30000 }]}]}]} ", " pod "=>" 47232f7c3803 ", " logshipper "=>" couchbase.sidecar.fluentbit"}] [ 1442 ] couchbase.log.json_rpc: [ 1617895514.965000000 , { "filename" => "/opt/couchbase/var/couchbase/logs/json_rpc.log" , "logger" => "json_rpc" , "level" => "debug" , "timestamp" => "2021-04-08T15:25:14.965Z" , "message" => "ns_1@cb-example-0002.cb-example.default.svc:json_rpc_connection-index-service_api<0.1467.0>:json_rpc_connection:handle_info:94]got response: [{<<" id">>, 150 },                {<< "result" >>,                 {[{<< "rev" >>,<< "AAAAAAAAAAI=" >>},                   {<< "nodes" >>,                    [<< "c772ddec75b1d5cadc6288eb5409ec98" >>,                     << "a5444727f282c01b6ef1bb0ec43a4ef0" >>,                     << "91e4269cb6a425eab9358e99151a3357" >>]},                   {<< "isBalanced" >>, true }]}},                {<< "error" >>, null }] ", " pod "=>" 47232f7c3803 ", " logshipper "=>" couchbase.sidecar.fluentbit"}] [ 1443 ] couchbase.log.json_rpc

            Downgrading this as effectively it is a config/doc task now - recommendation should be to use a custom configuration to stream direct to loki and avoid streaming to stdout particularly for multi-line.

            Eric Schneider you may be interested in this.

            patrick.stephens Patrick Stephens (Inactive) added a comment - - edited Downgrading this as effectively it is a config/doc task now - recommendation should be to use a custom configuration to stream direct to loki and avoid streaming to stdout particularly for multi-line. Eric Schneider  you may be interested in this.

            Need to configure PromTail to support multi-line if you want this.

            patrick.stephens Patrick Stephens (Inactive) added a comment - Need to configure PromTail to support multi-line if you want this.

            People

              patrick.stephens Patrick Stephens (Inactive)
              patrick.stephens Patrick Stephens (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty