Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
6.6.0
-
Both windows and linux
-
Untriaged
-
1
-
No
Description
What's the issue?
There is a couple of places where 'cbbackupmgr' may implicitly scrape sensitive information where it shouldn't. For example:
1) 'cbbackupmgr' logs the S3 access keys at the beginning of running a sub-command (6.6.x+ only)
2) Some of the (platform specific) commands run when collecting system information may collect the command line arguments of other processes on the system.
Steps to reproduce #1
1) Install Couchbase server 6.6.0-7897 on a windows server 2016
2) Run backup to S3.
3) Run collect-logs in S3 using cbbackupmgr, logs collect ok but it display all S3 credentials in raw text.
|
|
-c localhost -u <ud>Administrator</ud> -p ******** -r backup -a s3://bkrepo --obj-access-key-id AKIAJP --obj-secret-access-key xzsNfaTXZWBf --obj-staging-dir /root/bk-staging --obj-region us-west-2
|
2020-08-04T22:48:59.413+00:00 (Cmd) mounted archive with id: 2b4c1837-86c1-4275-8934-9d138b2f7709
|
2020-08-04T22:48:59.415+00:00 (Rest) GET http://localhost:8091/pools 200
|
2020-08-04T22:48:59.419+00:00 (Rest) GET http://localhost:8091/pools/default 200
|
2020-08-04T22:48:59.424+00:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
|
2020-08-04T22:48:59.468+00:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
|
2020-08-04T22:48:59.473+00:00 (Rest) GET http://localhost:8091/pools/default/buckets/travel-sample 200
|
2020-08-04T22:48:59.475+00:00 (Rest) GET http://localhost:8091/pools 200
|
2020-08-04T22:48:59.475+00:00 (Cmd) Backing up cluster 759547ebd21e733e4173ad953bb0b196
|
2020-08-04T22:48:59.476+00:00 (Stats) Starting stat gathering - stat timestamp: 1596581339
|
2020-08-04T22:48:59.476+00:00 (Plan) Transferring cluster configuration
|
2020-08-04T22:48:59.477+00:00 (Rest)
|
We need to hide it as in password
This issue prevent upload logs to MB-40764
Attachments
Activity
| Field | Original Value | New Value |
|---|---|---|
| Assignee | Patrick Varley [ pvarley ] | James Lee [ james.lee ] |
Hi Thuan Nguyen,
This is a simple logical mistake that happened during the object store related argument renaming e.g. '--s3-access-key' -> '--obj-access-key'. The fix is simple, and is something that is unit tested in master, however, it's currently testing the wrong behavior.
| Summary | [CBM] cbbackupmgr collect-logs should not display raw access-key and secret-key in cbbackupmgr log file | cbbackupmgr 'cmdLineArgsToString' does not correctly mask object store credentials |
| Fix Version/s | 6.6.0 [ 16787 ] |
| Due Date | 05/Aug/20 |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Link | This issue blocks MB-38724 [ MB-38724 ] |
| Labels | approved-for-6.6.0 |
| Resolution | Fixed [ 1 ] | |
| Status | In Progress [ 3 ] | Resolved [ 5 ] |
| Assignee | James Lee [ james.lee ] | Thuan Nguyen [ thuan ] |
Build couchbase-server-6.6.0-7903 contains backup commit 1b96331 with commit message:
MB-40765 Correctly mask sensitive object store flags
When run collect-logs in build 6.6.0-7905,
in system_info file, it still shows plain text access and secret key
terSystem EC2AMAZ-A513PGD NisSrv.exe 6788 171 312500 NisSrv.exe Win32_OperatingSystem Microsoft Windows Server 2016 Datacenter|C:\Windows|\Device\Harddisk0\Partition1 2473 1411293 4723 4328 596 5752 2199108046848 10800 8 4431872 6788 10 128 40 128 0 0 0 8 625000 2199098613760 10.0.14393 10366976 0 0
|
|
|
cbbackupmgr.exe .\cbbackupmgr collect-logs -o /root -a s3://bkrepo --obj-access-key-id AKIAJPG7 --obj-secret-access-key xzsNfaTXZWBfB
|
| Resolution | Fixed [ 1 ] | |
| Status | Resolved [ 5 ] | Reopened [ 4 ] |
| Attachment | ss 2020-08-05 at 5.23.30 PM.png [ 103181 ] |
This system_info is generated in both linux and windows when collect-logs using cloud bucket.
| Assignee | Thuan Nguyen [ thuan ] | James Lee [ james.lee ] |
Build couchbase-server-7.0.0-2772 contains backup commit 1b96331 with commit message:
MB-40765 Correctly mask sensitive object store flags
Hi Thuan Nguyen,
This is a slightly different issue to the one described initially in this MB, however, it ultimately results in the same issue (leaking credentials). When running 'collect-logs' we will collect some information about the host system (this is platform depended) on Linux we will scrape the output of 'ps' (including the command used to execute) and on Windows we'll scrape the output of 'wmic process' (which includes the command used to execute). When providing credentials with the static credentials flags, they will be implicitly scraped and placed into the 'system_info' file.
We could perform a trivial string replace on the output to remove the credentials, however, there are some additional security concerns e.g. we could be implicitly scraping then storing passwords/credentials provided as flags for commands other than our own. This is a slightly more difficult problem to solve, the easiest immediate fix for this issue to change the commands that we run slightly so that they don't include the command.
For Linux this will be be a case of changing the 'ps' command from
ps -Aww -o user,pid,lwp,ppid,nlwp,pcpu,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command
|
into
ps -Aww -o user,pid,lwp,ppid,nlwp,pcpu,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime
|
For windows, the 'wmic process' command offers a 'brief' flag which won't include the extra information (including the command), however, it will still show all the relevant information that we would like to collect.
| Link | This issue relates to MB-40783 [ MB-40783 ] |
Build couchbase-server-6.6.0-7906 contains backup commit f97e82b with commit message:
MB-40765 Don't collect sensitive information when collecting logs
Build couchbase-server-6.6.0-7907 contains backup commit 269d6ec with commit message:
MB-40765 Add 'comm' to 'ps' call to 'exec.Command'
Build couchbase-server-6.6.0-7907 contains backup commit 13925be with commit message:
MB-40765 Display the command without arguments in 'ps' output
'cbbackupmgr' should no longer leak S3 credentials in the logs or when running collect-logs. I have double checked on Linux/Mac and will resolve the issue once I've checked on Windows (waiting on the build).
Steps taken to prove this is the case:
1) Configure an archive in S3 using static credentials
2) Run 'collect-logs' on the newly configured archive
3) Examine the logs; we see that the values for '-obj-access-key-id' and '-obj-secret-access-key' are both obscured.
4) Examine the 'system_info' file; we see that the output from 'ps' no longer contains the running commands arguments meaning we don't implicitly collect the S3 credentials.
I've attached both the Linux/Mac OS collected logs used in this verification. I will update with the Windows logs once I've got them.
| Attachment | mac-os-cbbackupmgr-collectinfo-archive-2020-08-06T164507.zip [ 103255 ] |
| Attachment | cbbackupmgr-collectinfo-couchbase-archive-2020-08-06T170212.zip [ 103256 ] |
Build couchbase-server-6.6.0-7908 contains backup commit 5e545ae with commit message:
MB-40765 Don't show verbose task information using 'tasklist'
| Attachment | windows-cbbackupmgr-collectinfo-couchbase-archive-2020-08-06T122944.zip [ 103301 ] |
| Assignee | James Lee [ james.lee ] | Thuan Nguyen [ thuan ] |
| Resolution | Fixed [ 1 ] | |
| Status | Reopened [ 4 ] | Resolved [ 5 ] |
Verified on Windows server 2016 with build 6.6.0-7908
PS C:\root\cbbackupmgr-collectinfo-bk-staging-2020-08-06T215908\cbbackupmgr-collectinfo-bk-staging-2020-08-06T215908> cat .\logs\backup-0.* | Select-String 'access-key'
|
|
|
2020-08-06T21:50:59.022+00:00 (Cmd) config -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/bk-staging --obj-region us-west-2
|
2020-08-06T21:51:14.736+00:00 (Cmd) backup -c localhost -u <ud>Administrator</ud> -p ***** -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/bk-staging --obj-region us-west-2
|
2020-08-06T21:59:08.952+00:00 (Cmd) collect-logs -o /root -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/bk-staging --obj-region us-west-2
|
2020-08-06T21:50:59.022+00:00 (Cmd) config -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/bk-staging --obj-region us-west-2
|
2020-08-06T21:51:14.736+00:00 (Cmd) backup -c localhost -u <ud>Administrator</ud> -p ***** -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/bk-staging --obj-region us-west-2
|
|
|
|
|
PS C:\root\cbbackupmgr-collectinfo-bk-staging-2020-08-06T215908\cbbackupmgr-collectinfo-bk-staging-2020-08-06T215908> cat .\system_info.log | Select-String 'access-key'
|
|
|
PS C:\root\cbbackupmgr-collectinfo-bk-staging-2020-08-06T215908\cbbackupmgr-collectinfo-bk-staging-2020-08-06T215908>
|
Verified on Windows server2016 with build 6.6.0-7908 with --redact flag
|
|
PS C:\root\redacted-cbbackupmgr-collectinfo-bk-staging-2020-08-06T221129\redacted-cbbackupmgr-collectinfo-bk-staging-2020-08-06T221129\cbbackupmgr-collectinfo-bk-staging-2020-08
|
-06T221129> cat .\system_info.log | Select-String 'access-key'
|
PS C:\root\redacted-cbbackupmgr-collectinfo-bk-staging-2020-08-06T221129\redacted-cbbackupmgr-collectinfo-bk-staging-2020-08-06T221129\cbbackupmgr-collectinfo-bk-staging-2020-08
|
-06T221129> cat .\logs\backup-0.* | Select-String 'access-key'
|
|
|
2020-08-06T21:50:59.022+00:00 (Cmd) config -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/bk-staging --obj-region
|
us-west-2
|
2020-08-06T21:51:14.736+00:00 (Cmd) backup -c localhost -u <ud>22fde3ef0e935f43371673ce0a44869798d6b152</ud> -p ***** -r backup -a s3://bkrepo --obj-access-key-id *****
|
--obj-secret-access-key ***** --obj-staging-dir /root/bk-staging --obj-region us-west-2
|
2020-08-06T21:59:08.952+00:00 (Cmd) collect-logs -o /root -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/bk-staging
|
--obj-region us-west-2
|
2020-08-06T22:11:29.744+00:00 (Cmd) collect-logs -o /root -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/bk-staging
|
--obj-region us-west-2 --redact
|
2020-08-06T21:50:59.022+00:00 (Cmd) config -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/bk-staging --obj-region
|
us-west-2
|
2020-08-06T21:51:14.736+00:00 (Cmd) backup -c localhost -u <ud>22fde3ef0e935f43371673ce0a44869798d6b152</ud> -p ***** -r backup -a s3://bkrepo --obj-access-key-id *****
|
--obj-secret-access-key ***** --obj-staging-dir /root/bk-staging --obj-region us-west-2
|
2020-08-06T21:59:08.952+00:00 (Cmd) collect-logs -o /root -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/bk-staging
|
--obj-region us-west-2
|
|
|
|
|
PS C:\root\redacted-cbbackupmgr-collectinfo-bk-staging-2020-08-06T221129\redacted-cbbackupmgr-collectinfo-bk-staging-2020-08-06T221129\cbbackupmgr-collectinfo-bk-staging-2020-08
|
-06T221129>
|
Verified on mac Mojave 10.14.6 with build 6.6.0-7908
:cbbackupmgr-collectinfo-tmp-2020-08-06T163248 thuan$
|
Saigon:cbbackupmgr-collectinfo-tmp-2020-08-06T163248 thuan$ grep access-key system_info.log
|
Saigon:cbbackupmgr-collectinfo-tmp-2020-08-06T163248 thuan$ grep access-key ./*
|
grep: ./backup: Is a directory
|
grep: ./hello: Is a directory
|
grep: ./logs: Is a directory
|
Saigon:cbbackupmgr-collectinfo-tmp-2020-08-06T163248 thuan$ grep access-key ./*/*
|
grep: ./backup/2020-08-06T16_28_35.073324-07_00: Is a directory
|
./logs/backup-0.log:2020-08-06T16:23:22.824-07:00 (Cmd) config -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2
|
./logs/backup-0.log:2020-08-06T16:28:35.041-07:00 (Cmd) backup -c localhost -u <ud>Administrator</ud> -p ***** -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2
|
./logs/backup-0.log:2020-08-06T16:32:48.296-07:00 (Cmd) collect-logs -o /trtmp/tmp/ -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2
|
./logs/backup-0.staged.log:2020-08-06T16:23:22.824-07:00 (Cmd) config -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2
|
./logs/backup-0.staged.log:2020-08-06T16:28:35.041-07:00 (Cmd) backup -c localhost -u <ud>Administrator</ud> -p ***** -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2
|
./logs/backup-0.staged.log:2020-08-06T16:29:57.515-07:00 (Cmd) info --all -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2 -r backup
|
grep: ./logs/stats: Is a directory
|
Saigon:cbbackupmgr-collectinfo-tmp-2020-08-06T163248 thuan$
|
Verified on mac Mojave 10.14.6 with build 6.6.0-7908 with --redact flag
thuan$ grep access-key ./*
|
grep: ./backup: Is a directory
|
grep: ./logs: Is a directory
|
Saigon:cbbackupmgr-collectinfo-tmp-2020-08-06T164431 thuan$ grep access-key ./*/*
|
grep: ./backup/2020-08-06T16_28_35.073324-07_00: Is a directory
|
./logs/backup-0.log:2020-08-06T16:23:22.824-07:00 (Cmd) config -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2
|
./logs/backup-0.log:2020-08-06T16:28:35.041-07:00 (Cmd) backup -c localhost -u <ud>22fde3ef0e935f43371673ce0a44869798d6b152</ud> -p ***** -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2
|
./logs/backup-0.log:2020-08-06T16:32:48.296-07:00 (Cmd) collect-logs -o /trtmp/tmp/ -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2
|
./logs/backup-0.log:2020-08-06T16:44:31.496-07:00 (Cmd) collect-logs -o /trtmp/tmp/ -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2 --redact
|
./logs/backup-0.staged.log:2020-08-06T16:23:22.824-07:00 (Cmd) config -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2
|
./logs/backup-0.staged.log:2020-08-06T16:28:35.041-07:00 (Cmd) backup -c localhost -u <ud>22fde3ef0e935f43371673ce0a44869798d6b152</ud> -p ***** -r backup -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2
|
./logs/backup-0.staged.log:2020-08-06T16:32:48.296-07:00 (Cmd) collect-logs -o /trtmp/tmp/ -a s3://bkrepo --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /trtmp/tmp/ --obj-region us-west-2
|
grep: ./logs/stats: Is a directory
|
Saigon:cbbackupmgr-collectinfo-tmp-2020-08-06T164431 thuan$
|
Verified in centos7:
[root@node1-mad-hatter-testing-centos7 logs]# cd cbbackupmgr-collectinfo-tests3-2020-08-06T172446
|
[root@node1-mad-hatter-testing-centos7 cbbackupmgr-collectinfo-tests3-2020-08-06T172446]# grep access-key system_info.log
|
[root@node1-mad-hatter-testing-centos7 cbbackupmgr-collectinfo-tests3-2020-08-06T172446]# grep access-key ./*
|
grep: ./backup: Is a directory
|
grep: ./logs: Is a directory
|
[root@node1-mad-hatter-testing-centos7 cbbackupmgr-collectinfo-tests3-2020-08-06T172446]# grep access-key ./*/*
|
grep: ./backup/2020-08-06T17_18_09.165566263-07_00: Is a directory
|
./logs/backup-0.log:2020-08-06T17:17:44.326-07:00 (Cmd) config -r backup -a s3://mb40765 --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/tests3 --obj-region us-west-1
|
./logs/backup-0.log:2020-08-06T17:18:09.156-07:00 (Cmd) backup -a s3://mb40765 -r backup --obj-access-key-id ***** --obj-secret-access-key ***** -c 10.112.194.101 -u <ud>Administrator</ud> -p ***** --obj-staging-dir /root/tests3 -obj-region us-west-1
|
./logs/backup-0.log:2020-08-06T17:24:46.411-07:00 (Cmd) collect-logs -a s3://mb40765 --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/tests3 -obj-region us-west-1
|
./logs/backup-0.staged.log:2020-08-06T17:17:44.326-07:00 (Cmd) config -r backup -a s3://mb40765 --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/tests3 --obj-region us-west-1
|
./logs/backup-0.staged.log:2020-08-06T17:18:09.156-07:00 (Cmd) backup -a s3://mb40765 -r backup --obj-access-key-id ***** --obj-secret-access-key ***** -c 10.112.194.101 -u <ud>Administrator</ud> -p ***** --obj-staging-dir /root/tests3 -obj-region us-west-1
|
grep: ./logs/stats: Is a directory
|
[root@node1-mad-hatter-testing-centos7 cbbackupmgr-collectinfo-tests3-2020-08-06T172446]# cd ../
|
[root@node1-mad-hatter-testing-centos7 logs]# ls
|
backup-0.log backup-0.staged.log cbbackupmgr-collectinfo-tests3-2020-08-06T172446 cbbackupmgr-collectinfo-tests3-2020-08-06T172446.zip cbbackupmgr-collectinfo-tests3-2020-08-06T172503.zip redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503.zip stats
|
[root@node1-mad-hatter-testing-centos7 logs]# unzip redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503.zip
|
Archive: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503.zip
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/system_info.log
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/info.json
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/logs/backup-0.log
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/logs/backup-0.staged.log
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/logs/stats/cpu/backup-backup-1596759489
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/logs/stats/disk/backup-backup-1596759489
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/logs/stats/net/backup-backup-1596759489
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/backup/backup-meta.json
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/backup/2020-08-06T17_18_09.165566263-07_00/plan.json
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/backup/2020-08-06T17_18_09.165566263-07_00/info.json
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/backup/2020-08-06T17_18_09.165566263-07_00/travel-sample-f286a0b41467f3132ebb57db77a44db3/range.json
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/backup/2020-08-06T17_18_09.165566263-07_00/travel-sample-f286a0b41467f3132ebb57db77a44db3/restrictions.json
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/backup/2020-08-06T17_18_09.165566263-07_00/travel-sample-f286a0b41467f3132ebb57db77a44db3/data.json
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/backup/2020-08-06T17_18_09.165566263-07_00/travel-sample-f286a0b41467f3132ebb57db77a44db3/data/snapshots.zip
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/backup/2020-08-06T17_18_09.165566263-07_00/travel-sample-f286a0b41467f3132ebb57db77a44db3/data/failoverlogs.zip
|
inflating: redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503/cbbackupmgr-collectinfo-tests3-2020-08-06T172503/backup/2020-08-06T17_18_09.165566263-07_00/travel-sample-f286a0b41467f3132ebb57db77a44db3/data/stats.zip
|
[root@node1-mad-hatter-testing-centos7 logs]# cd redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503
|
[root@node1-mad-hatter-testing-centos7 redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503]# grep access-key system_info.log
|
grep: system_info.log: No such file or directory
|
[root@node1-mad-hatter-testing-centos7 redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503]# grep access-key ./*
|
grep: ./cbbackupmgr-collectinfo-tests3-2020-08-06T172503: Is a directory
|
[root@node1-mad-hatter-testing-centos7 redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503]# grep access-key ./*/*
|
grep: ./cbbackupmgr-collectinfo-tests3-2020-08-06T172503/backup: Is a directory
|
grep: ./cbbackupmgr-collectinfo-tests3-2020-08-06T172503/logs: Is a directory
|
[root@node1-mad-hatter-testing-centos7 redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503]# ls
|
cbbackupmgr-collectinfo-tests3-2020-08-06T172503
|
[root@node1-mad-hatter-testing-centos7 redacted-cbbackupmgr-collectinfo-tests3-2020-08-06T172503]# cd cbbackupmgr-collectinfo-tests3-2020-08-06T172503/
|
[root@node1-mad-hatter-testing-centos7 cbbackupmgr-collectinfo-tests3-2020-08-06T172503]# grep access-key system_info.log
|
[root@node1-mad-hatter-testing-centos7 cbbackupmgr-collectinfo-tests3-2020-08-06T172503]# grep access-key ./*
|
grep: ./backup: Is a directory
|
grep: ./logs: Is a directory
|
[root@node1-mad-hatter-testing-centos7 cbbackupmgr-collectinfo-tests3-2020-08-06T172503]# grep access-key ./*/*
|
grep: ./backup/2020-08-06T17_18_09.165566263-07_00: Is a directory
|
./logs/backup-0.log:2020-08-06T17:17:44.326-07:00 (Cmd) config -r backup -a s3://mb40765 --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/tests3 --obj-region us-west-1
|
./logs/backup-0.log:2020-08-06T17:18:09.156-07:00 (Cmd) backup -a s3://mb40765 -r backup --obj-access-key-id ***** --obj-secret-access-key ***** -c 10.112.194.101 -u <ud>5c0bf01eafe80aa7b699e71e097488b25190d8ad</ud> -p ***** --obj-staging-dir /root/tests3 -obj-region us-west-1
|
./logs/backup-0.log:2020-08-06T17:24:46.411-07:00 (Cmd) collect-logs -a s3://mb40765 --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/tests3 -obj-region us-west-1
|
./logs/backup-0.log:2020-08-06T17:25:03.465-07:00 (Cmd) collect-logs -a s3://mb40765 --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/tests3 -obj-region us-west-1 --redact
|
./logs/backup-0.staged.log:2020-08-06T17:17:44.326-07:00 (Cmd) config -r backup -a s3://mb40765 --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/tests3 --obj-region us-west-1
|
./logs/backup-0.staged.log:2020-08-06T17:18:09.156-07:00 (Cmd) backup -a s3://mb40765 -r backup --obj-access-key-id ***** --obj-secret-access-key ***** -c 10.112.194.101 -u <ud>5c0bf01eafe80aa7b699e71e097488b25190d8ad</ud> -p ***** --obj-staging-dir /root/tests3 -obj-region us-west-1
|
./logs/backup-0.staged.log:2020-08-06T17:24:46.411-07:00 (Cmd) collect-logs -a s3://mb40765 --obj-access-key-id ***** --obj-secret-access-key ***** --obj-staging-dir /root/tests3 -obj-region us-west-1
|
grep: ./logs/stats: Is a directory
|
[root@node1-mad-hatter-testing-centos7 cbbackupmgr-collectinfo-tests3-2020-08-06T172503]#
|
| Status | Resolved [ 5 ] | Closed [ 6 ] |
Build couchbase-server-7.0.0-2787 contains backup commit 5e545ae with commit message:
MB-40765 Don't show verbose task information using 'tasklist'
Build couchbase-server-7.0.0-2787 contains backup commit 269d6ec with commit message:
MB-40765 Add 'comm' to 'ps' call to 'exec.Command'
Build couchbase-server-7.0.0-2787 contains backup commit 13925be with commit message:
MB-40765 Display the command without arguments in 'ps' output
Build couchbase-server-7.0.0-2787 contains backup commit f97e82b with commit message:
MB-40765 Don't collect sensitive information when collecting logs
| Link | This issue relates to CBD-3571 [ CBD-3571 ] |
Reopening because patches under this MB need to be backported to the 6.5.2. release
| Assignee | Thuan Nguyen [ thuan ] | James Lee [ james.lee ] |
| Resolution | Fixed [ 1 ] | |
| Status | Closed [ 6 ] | Reopened [ 4 ] |
| Fix Version/s | 6.5.2 [ 17223 ] |
| Link | This issue blocks MB-42583 [ MB-42583 ] |
| Labels | approved-for-6.6.0 | approved-for-6.5.2 approved-for-6.6.0 |
James Lee did you mean 6.6.2? this is only for backup to s3 right? which is not there in 6.5.x
Arunkumar Senthilnathan, whilst fixing the 'cmdLineArgsToString' function, some other related security issues were fixed; for example some of the 'collect-logs' related 'ps' and 'wmic' invocations which would implicitly scrape the cloud credentials. It's those fixes which need to be backported into 6.5.2 since it's also possible to scrape other sensitive information; I'll update the MB to be slightly more clear.
| Summary | cbbackupmgr 'cmdLineArgsToString' does not correctly mask object store credentials | cbbackupmgr shouldn't log/collect sensitive information |
| Description |
Install Couchbase server 6.6.0-7897 on a windows server 2016
Run backup to S3. Run collect-logs in S3 using cbbackupmgr, logs collect ok but it display all S3 credentials in raw text. {noformat} -c localhost -u <ud>Administrator</ud> -p ******** -r backup -a s3://bkrepo --obj-access-key-id AKIAJP --obj-secret-access-key xzsNfaTXZWBf --obj-staging-dir /root/bk-staging --obj-region us-west-2 2020-08-04T22:48:59.413+00:00 (Cmd) mounted archive with id: 2b4c1837-86c1-4275-8934-9d138b2f7709 2020-08-04T22:48:59.415+00:00 (Rest) GET http://localhost:8091/pools 200 2020-08-04T22:48:59.419+00:00 (Rest) GET http://localhost:8091/pools/default 200 2020-08-04T22:48:59.424+00:00 (Rest) GET http://localhost:8091/pools/default/buckets 200 2020-08-04T22:48:59.468+00:00 (Rest) GET http://localhost:8091/pools/default/buckets 200 2020-08-04T22:48:59.473+00:00 (Rest) GET http://localhost:8091/pools/default/buckets/travel-sample 200 2020-08-04T22:48:59.475+00:00 (Rest) GET http://localhost:8091/pools 200 2020-08-04T22:48:59.475+00:00 (Cmd) Backing up cluster 759547ebd21e733e4173ad953bb0b196 2020-08-04T22:48:59.476+00:00 (Stats) Starting stat gathering - stat timestamp: 1596581339 2020-08-04T22:48:59.476+00:00 (Plan) Transferring cluster configuration 2020-08-04T22:48:59.477+00:00 (Rest) {noformat} We need to hide it as in password This issue prevent upload logs to |
+What's the issue?+
There is a couple of places where '{{cbbackupmgr}}' may implicitly scrape sensitive information where it shouldn't. For example: 1) '{{cbbackupmgr}}' logs the S3 access keys at the beginning of running a sub-command (6.6.x+ only) 2) Some of the (platform specific) commands run when collecting system information may collect the command line arguments of other processes on the system. +Steps to reproduce #1+ 1) Install Couchbase server 6.6.0-7897 on a windows server 2016 2) Run backup to S3. 3) Run collect-logs in S3 using cbbackupmgr, logs collect ok but it display all S3 credentials in raw text. {noformat} -c localhost -u <ud>Administrator</ud> -p ******** -r backup -a s3://bkrepo --obj-access-key-id AKIAJP --obj-secret-access-key xzsNfaTXZWBf --obj-staging-dir /root/bk-staging --obj-region us-west-2 2020-08-04T22:48:59.413+00:00 (Cmd) mounted archive with id: 2b4c1837-86c1-4275-8934-9d138b2f7709 2020-08-04T22:48:59.415+00:00 (Rest) GET http://localhost:8091/pools 200 2020-08-04T22:48:59.419+00:00 (Rest) GET http://localhost:8091/pools/default 200 2020-08-04T22:48:59.424+00:00 (Rest) GET http://localhost:8091/pools/default/buckets 200 2020-08-04T22:48:59.468+00:00 (Rest) GET http://localhost:8091/pools/default/buckets 200 2020-08-04T22:48:59.473+00:00 (Rest) GET http://localhost:8091/pools/default/buckets/travel-sample 200 2020-08-04T22:48:59.475+00:00 (Rest) GET http://localhost:8091/pools 200 2020-08-04T22:48:59.475+00:00 (Cmd) Backing up cluster 759547ebd21e733e4173ad953bb0b196 2020-08-04T22:48:59.476+00:00 (Stats) Starting stat gathering - stat timestamp: 1596581339 2020-08-04T22:48:59.476+00:00 (Plan) Transferring cluster configuration 2020-08-04T22:48:59.477+00:00 (Rest) {noformat} We need to hide it as in password This issue prevent upload logs to |
| Status | Reopened [ 4 ] | In Progress [ 3 ] |
| Assignee | James Lee [ james.lee ] | Thuan Nguyen [ thuan ] |
| Resolution | Fixed [ 1 ] | |
| Status | In Progress [ 3 ] | Resolved [ 5 ] |
| Assignee | Thuan Nguyen [ thuan ] | James Lee [ james.lee ] |
| Resolution | Fixed [ 1 ] | |
| Status | Resolved [ 5 ] | Reopened [ 4 ] |
An update on progress: security fixes have been merged into 6.5.2, the issue remains open whilst we merge the changes forward into mad-hatter and cheshire-cat (note that theses are noop merges i.e. these issue have already been fixed in MH and CC).
| Assignee | James Lee [ james.lee ] | Thuan Nguyen [ thuan ] |
| Resolution | Fixed [ 1 ] | |
| Status | Reopened [ 4 ] | Resolved [ 5 ] |
Build couchbase-server-6.6.2-9452 contains backup commit dd571dc with commit message:
MB-40765 Merge branch '6.5.2' into mad-hatter
Build couchbase-server-6.6.2-9452 contains backup commit 09258c5 with commit message:
MB-40765 [BP] Don't show verbose task information using 'tasklist'
Build couchbase-server-6.6.2-9452 contains backup commit f7a2eaa with commit message:
MB-40765 [BP] Add 'comm' to 'ps' call to 'exec.Command'
Build couchbase-server-6.6.2-9452 contains backup commit 123066a with commit message:
MB-40765 [BP] Display the command without arguments in 'ps' output
Build couchbase-server-6.6.2-9452 contains backup commit db275f5 with commit message:
MB-40765 [BP] Don't collect sensitive information when collecting logs
Build couchbase-server-7.0.0-4300 contains backup commit af8df59 with commit message:
MB-40765 Merge branch 'mad-hatter' into master
Build couchbase-server-7.0.0-4300 contains backup commit dd571dc with commit message:
MB-40765 Merge branch '6.5.2' into mad-hatter
Build couchbase-server-7.0.0-4300 contains backup commit 09258c5 with commit message:
MB-40765 [BP] Don't show verbose task information using 'tasklist'
Build couchbase-server-7.0.0-4300 contains backup commit f7a2eaa with commit message:
MB-40765 [BP] Add 'comm' to 'ps' call to 'exec.Command'
Build couchbase-server-7.0.0-4300 contains backup commit 123066a with commit message:
MB-40765 [BP] Display the command without arguments in 'ps' output
Build couchbase-server-7.0.0-4300 contains backup commit db275f5 with commit message:
MB-40765 [BP] Don't collect sensitive information when collecting logs
Build couchbase-server-6.5.2-6622 contains backup commit 09258c5 with commit message:
MB-40765 [BP] Don't show verbose task information using 'tasklist'
Build couchbase-server-6.5.2-6622 contains backup commit f7a2eaa with commit message:
MB-40765 [BP] Add 'comm' to 'ps' call to 'exec.Command'
Build couchbase-server-6.5.2-6622 contains backup commit 123066a with commit message:
MB-40765 [BP] Display the command without arguments in 'ps' output
Build couchbase-server-6.5.2-6622 contains backup commit db275f5 with commit message:
MB-40765 [BP] Don't collect sensitive information when collecting logs
| Attachment | cbbackupmgr-collectinfo-my_archive-2021-01-28T182737.zip [ 124273 ] |
| Attachment | cbbackupmgr-collectinfo-my_archive-2021-01-28T182737 2.zip [ 124274 ] |
Verification for 6.5.2-6624 on Windows:
This ticket seems to discuss two related issues.
1. The first issue is that cbbackupmgr logged AWS credentials, this only affects versions 6.6.x and above.
Cbbackupmgr in version 6.5.2 cannot backup to s3 so it is not affected by this issue.
2. The second issue is that process arguments were being logged revealing sensitive information.
We can check that this is fixed by running various cbbackupmgr sub-commands and checking if process arguments are leaked and if any sensitive information is revealed.
I created a backup archive, performed 3 backups and used the info command to check which backups exist.
I then used the collect-log sub command to produce the following logs:
cbbackupmgr-collectinfo-my_archive-2021-01-28T182737.zip
I performed a case-case-insensistive grep for 'administrator' and 'password'. In addition, I do not seem to see any process arguments.
(It's definitely worth if someone could take a second look at these).
An additional collection where I also performed the restore command.
cbbackupmgr-collectinfo-my_archive-2021-01-28T185045.zip
The grep results:
sadzaidi@EM224 cbbackupmgr-collectinfo-my_archive-2021-01-28T185045 % grep -ri 'administrator' ./
|
asadzaidi@EM224 cbbackupmgr-collectinfo-my_archive-2021-01-28T185045 % grep -ri 'password' ./
|
.//system_info.log:AdminPasswordStatus AutomaticManagedPagefile AutomaticResetBootOption AutomaticResetCapability BootOptionOnLimit BootOptionOnWatchDog BootROMSupported BootStatus BootupState Caption ChassisBootupState ChassisSKUNumber CreationClassName CurrentTimeZone DaylightInEffect Description DNSHostName Domain DomainRole EnableDaylightSavingsTime FrontPanelResetStatus HypervisorPresent InfraredSupported InitialLoadInfo InstallDate KeyboardPasswordStatus LastLoadInfo Manufacturer Model Name NameFormat NetworkServerModeEnabled NumberOfLogicalProcessors NumberOfProcessors OEMLogoBitmap OEMStringArray PartOfDomain PauseAfterReset PCSystemType PCSystemTypeEx PowerManagementCapabilities PowerManagementSupported PowerOnPasswordStatus PowerState PowerSupplyState PrimaryOwnerContact PrimaryOwnerName ResetCapability ResetCount ResetLimit Roles Status SupportContactDescription SystemFamily SystemSKUNumber SystemStartupDelay SystemStartupOptions SystemStartupSetting SystemType ThermalState TotalPhysicalMemory UserName WakeUpType Workgroup
|
asadzaidi@EM224 cbbackupmgr-collectinfo-my_archive-2021-01-28T185045 %
|
| Attachment | cbbackupmgr-collectinfo-my_archive-2021-01-28T185045.zip [ 124275 ] |
| Status | Resolved [ 5 ] | Closed [ 6 ] |
I tried again with --redact flag, collect-logs still has raw access key and secret key in log.