Details
-
Bug
-
Resolution: Fixed
-
Major
-
2.8.6
-
None
-
None
-
1
Description
I need to do some load testing of a cluster using TLS. I setup a loadtest bucket and a loadtest user with a password. Then tried using cbc-pillowfight as follows:
[bweir@ltx1-app19835 ~]$ cbc-pillowfight -U couchbases://ltx1-app31067.prod.linkedin.com/loadtest -u loadtest -P <redacted> --truststorepath /etc/riddler/ca-bundle.crt -v
|
Running. Press Ctrl-C to terminate...
|
0ms [I8f1eda8c] {26187} [INFO] (instance - L:466) Version=2.8.7, Changeset=081e8b16b991bf706eb77f8243935c6fba31b895
|
0ms [I8f1eda8c] {26187} [INFO] (instance - L:467) Effective connection string: couchbases://ltx1-app31067.prod.linkedin.com/loadtest?truststorepath=/etc/riddler/ca-bundle.crt&username=loadtest&console_log_level=2&. Bucket=loadtest
|
8ms [I8f1eda8c] {26187} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
|
8ms [I8f1eda8c] {26187} [INFO] (cccp - L:151) Requesting connection to node ltx1-app31067.prod.linkedin.com:11207 for CCCP configuration
|
8ms [I8f1eda8c] {26187} [INFO] (connection - L:474) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=86df6471614e68b7) Starting. Timeout=2000000us
|
9ms [I8f1eda8c] {26187} [INFO] (connection - L:147) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=86df6471614e68b7) Connected established
|
12ms [I8f1eda8c] {26187} [ERROR] (SSL - L:152) error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
|
13ms [I8f1eda8c] {26187} [ERROR] (negotiation - L:154) <ltx1-app31067.prod.linkedin.com:11207> (CTX=0x172ed70,sasl,SASLREQ=0x172dd40) Error: 0x37, IO Error
|
13ms [I8f1eda8c] {26187} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_SSL_CANTVERIFY (0x37)
|
13ms [I8f1eda8c] {26187} [INFO] (confmon - L:185) Provider 'CCCP' failed
|
14ms [I8f1eda8c] {26187} [INFO] (connection - L:474) <ltx1-app31067.prod.linkedin.com:18091> (SOCK=7571b4a16ff99a7e) Starting. Timeout=2000000us
|
14ms [I8f1eda8c] {26187} [INFO] (connection - L:147) <ltx1-app31067.prod.linkedin.com:18091> (SOCK=7571b4a16ff99a7e) Connected established
|
17ms [I8f1eda8c] {26187} [ERROR] (SSL - L:152) error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
|
17ms [I8f1eda8c] {26187} [INFO] (confmon - L:185) Provider 'HTTP' failed
|
17ms [I8f1eda8c] {26187} [ERROR] (bootstrap - L:170) Failed to bootstrap client=0x170e8a0. Error=LCB_SSL_CANTVERIFY (0x37), Message=No more bootstrap providers remainFailed to connect: Client could not verify server's certificate
|
|
|
|
As seen above, the client failed to verify server's cert even though I had specified --truststorepath on the commandline
When I switch from password auth to certificate auth, then it works:
[bweir@ltx1-app19835 ~]$ cbc-pillowfight -U couchbases://ltx1-app31067.prod.linkedin.com/loadtest --truststorepath /etc/riddler/ca-bundle.crt --keypath ~/identity.key --certpath ~/identity.cert -v
|
Running. Press Ctrl-C to terminate...
|
0ms [I51df37c1] {27608} [INFO] (instance - L:466) Version=2.8.7, Changeset=081e8b16b991bf706eb77f8243935c6fba31b895
|
0ms [I51df37c1] {27608} [INFO] (instance - L:467) Effective connection string: couchbases://ltx1-app31067.prod.linkedin.com/loadtest?truststorepath=/etc/riddler/ca-bundle.crt&certpath=/export/home/bweir/identity.cert&keypath=/export/home/bweir/identity.key&console_log_level=2&. Bucket=loadtest
|
8ms [I51df37c1] {27608} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
|
8ms [I51df37c1] {27608} [INFO] (cccp - L:151) Requesting connection to node ltx1-app31067.prod.linkedin.com:11207 for CCCP configuration
|
8ms [I51df37c1] {27608} [INFO] (connection - L:474) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=1ca76b23d8385b32) Starting. Timeout=2000000us
|
9ms [I51df37c1] {27608} [INFO] (connection - L:147) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=1ca76b23d8385b32) Connected established
|
29ms [I51df37c1] {27608} [INFO] (lcbio_mgr - L:498) <ltx1-app31067.prod.linkedin.com:11207> (HE=0x90ce10) Placing socket back into the pool. I=0x90cfd0,C=0x919390
|
30ms [I51df37c1] {27608} [INFO] (confmon - L:160) Setting new configuration. Received via CCCP
|
32ms [I51df37c1] {27613} [INFO] (connection - L:474) <ltx1-app25928.prod.linkedin.com:11207> (SOCK=1693e6671fd4128e) Starting. Timeout=2500000us
|
32ms [I51df37c1] {27613} [INFO] (connection - L:474) <ltx1-app26263.prod.linkedin.com:11207> (SOCK=35b81b54b0c2e2b7) Starting. Timeout=2500000us
|
32ms [I51df37c1] {27613} [INFO] (connection - L:474) <ltx1-app26336.prod.linkedin.com:11207> (SOCK=592b3ca88a613a1a) Starting. Timeout=2500000us
|
32ms [I51df37c1] {27613} [INFO] (connection - L:474) <ltx1-app26347.prod.linkedin.com:11207> (SOCK=17e70313b01d51dc) Starting. Timeout=2500000us
|
...
|
For completeness, here's specifying certificate auth but leaving out --truststorepath which leads to a "Client could not verify server's certificate" error as expected:
[bweir@ltx1-app19835 ~]$ cbc-pillowfight -U couchbases://ltx1-app31067.prod.linkedin.com/loadtest --keypath ~/identity.key --certpath ~/identity.cert -v
|
Running. Press Ctrl-C to terminate...
|
0ms [I109af43d] {29293} [INFO] (instance - L:466) Version=2.8.7, Changeset=081e8b16b991bf706eb77f8243935c6fba31b895
|
0ms [I109af43d] {29293} [INFO] (instance - L:467) Effective connection string: couchbases://ltx1-app31067.prod.linkedin.com/loadtest?certpath=/export/home/bweir/identity.cert&keypath=/export/home/bweir/identity.key&console_log_level=2&. Bucket=loadtest
|
6ms [I109af43d] {29293} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
|
6ms [I109af43d] {29293} [INFO] (cccp - L:151) Requesting connection to node ltx1-app31067.prod.linkedin.com:11207 for CCCP configuration
|
6ms [I109af43d] {29293} [INFO] (connection - L:474) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=d5b7caf75fc13082) Starting. Timeout=2000000us
|
7ms [I109af43d] {29293} [INFO] (connection - L:147) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=d5b7caf75fc13082) Connected established
|
12ms [I109af43d] {29293} [ERROR] (SSL - L:152) error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
|
13ms [I109af43d] {29293} [ERROR] (negotiation - L:154) <ltx1-app31067.prod.linkedin.com:11207> (CTX=0x1a6f2b0,sasl,SASLREQ=0x1a6e300) Error: 0x37, IO Error
|
13ms [I109af43d] {29293} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_SSL_CANTVERIFY (0x37)
|
13ms [I109af43d] {29293} [INFO] (confmon - L:185) Provider 'CCCP' failed
|
13ms [I109af43d] {29293} [INFO] (connection - L:474) <ltx1-app31067.prod.linkedin.com:18091> (SOCK=94c40a19ea55c802) Starting. Timeout=2000000us
|
13ms [I109af43d] {29293} [INFO] (connection - L:147) <ltx1-app31067.prod.linkedin.com:18091> (SOCK=94c40a19ea55c802) Connected established
|
16ms [I109af43d] {29293} [ERROR] (SSL - L:152) error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
|
17ms [I109af43d] {29293} [INFO] (confmon - L:185) Provider 'HTTP' failed
|
17ms [I109af43d] {29293} [ERROR] (bootstrap - L:170) Failed to bootstrap client=0x1a4a8d0. Error=LCB_SSL_CANTVERIFY (0x37), Message=No more bootstrap providers remainFailed to connect: Client could not verify server's certificate
|
|
My expectation is that the --truststorepath parameter would be used any time a TLS connection is requested, regardless of whether certificate or password auth is being used.
I have verified using strace that my trust store file is never being opened in the first case.
Attachments
| For Gerrit Dashboard: CCBC-1007 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 102888,2 | CCBC-1007: allow using trusted store path without key file | master | libcouchbase | Status: MERGED | +2 | +1 |