Major Description
From a basic review of the logging when trying out eventing, it seems that it is incorrectly tagging user data in log messages.
From the log redaction PRD (https://docs.google.com/document/d/1rbQsiDpKspJpzZspAMxjvU19cHEtvEFeGQ_jLmFJ4bM/edit#heading=h.q6wzxxq9l7kg) the following data is supposed to be tagged user data:
- Key and value pairs in JSON documents, or the key exclusively
- Application/Admin usernames that identify the human person
- Query statements included in the log file collected by support that leak the document fields (Select floor_price from stock).
- Names and email addresses asked during product registration and alerting
- Usernames
- Document xattrs
—
Based on this I can see quite a few log messages where data is being incorrectly tagged as user data:
2018-03-14T14:07:05.395+00:00 [Debug] SMCO Got eventing nodes: <ud>[]string{"127.0.0.1:8096"}</ud>
|
These are node names, which are included under 'System Data', not user data.
2018-03-14T14:07:05.396+00:00 [Debug] Allowing access to <ud>/getLocallyDeployedApps</ud>
|
Here a built-in Couchbase REST endpoint is being redacted, which is classed as 'Metadata'.
2018-03-14T14:07:05.156+00:00 [Debug] Allowing access to <ud>/api/v1/stats</ud>
|
As above, a built-in REST endpoint is being redacted, this should instead be 'Metadata'.
2018-03-14T14:06:34.512+00:00 [Info] Inserting event: 1 opcode: 2 partition: 424 metadata: <ud>{"cas":1521035991125458944,"id":"route_5274","expiration":0,"flags":33554432,"vb":424,"seq":31}</ud>
|
This is kind of an interesting one, here the whole metadata is being tagged as 'userdata' when really it should just be the key. From a support's perspective we need to be able to easily see metadata when debugging issues.
2018-03-14T14:06:34.947+00:00 [Info] Cluster wide deployed app status: <ud>&map[127.0.0.1:8096:map[]]</ud>
|
It looks like node names (i.e. metadata) has been tagged as 'user data' here.
018-03-14T14:25:23.884+00:00 [Trace] Audit event <ud>32771</ud> with context <ud><nil></ud> on request <ud>&{GET /getDeployedApps HTTP/1.1 1 1 map[User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0] Ns-Server-Auth-Token:[d3e330bdc9c0dfbda0ee5e5e92683123] Cache-Control:[no-cache] Connection:[keep-alive] Cookie:[ui-auth-10.142.180.101%3A8091=d3e330bdc9c0dfbda0ee5e5e92683123] Invalid-Auth-Response:[on] Menelaus-Auth-Domain:[admin] Accept:[application/json, text/plain, */*] Accept-Encoding:[gzip, deflate] Menelaus-Auth-User:[Administrator] Referer:[http://10.142.180.101:8091/ui/index.html] Accept-Language:[en-US,en;q=0.5] Menelaus-Auth-Token:[d3e330bdc9c0dfbda0ee5e5e92683123] Ns-Server-Ui:[yes yes] Pragma:[no-cache]] 0x14cfb90 0 [] false 10.142.180.101:8091 map[] map[] <nil> map[] 127.0.0.1:35982 /getDeployedApps <nil> <nil> <nil> 0xc422976e80}</ud>
|
This log message looks like it is being over-redacted. Based on the code to redact the http_access.log in http://review.couchbase.org/#/c/89878/ it seems that we only need to redact CERTAIN urls and the username, rather than the entire message. I definitely don't think the audit event ID needs to be redacted.
—
These were just the messages that I saw in my logs but I took a look at all of the redacted log messages on http://src.couchbase.org/source/search?project=trunk&q=%22%25r%22&defs=&refs=&path=eventing&hist=&type= and it looks like there are quite a few more which may have been incorrectly tagged.
Could we please review what we have currently got tagged as user data in eventing and ensure that it is consistent with the log redaction specification?
This will make Support's life much easier, especially because this will be a new feature in Vulcan.