Uploaded image for project: 'Couchbase C client library libcouchbase'
  1. Couchbase C client library libcouchbase
  2. CCBC-1007

pillowfight ignores --truststorepath when using password auth

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.6
    • Fix Version/s: 2.10.3
    • Component/s: None
    • Labels:
      None

      Description

      I need to do some load testing of a cluster using TLS. I setup a loadtest bucket and a loadtest user with a password. Then tried using cbc-pillowfight as follows:

      [bweir@ltx1-app19835 ~]$ cbc-pillowfight -U couchbases://ltx1-app31067.prod.linkedin.com/loadtest -u loadtest -P <redacted> --truststorepath /etc/riddler/ca-bundle.crt -v
      Running. Press Ctrl-C to terminate...
      0ms [I8f1eda8c] {26187} [INFO] (instance - L:466) Version=2.8.7, Changeset=081e8b16b991bf706eb77f8243935c6fba31b895
      0ms [I8f1eda8c] {26187} [INFO] (instance - L:467) Effective connection string: couchbases://ltx1-app31067.prod.linkedin.com/loadtest?truststorepath=/etc/riddler/ca-bundle.crt&username=loadtest&console_log_level=2&. Bucket=loadtest
      8ms [I8f1eda8c] {26187} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
      8ms [I8f1eda8c] {26187} [INFO] (cccp - L:151) Requesting connection to node ltx1-app31067.prod.linkedin.com:11207 for CCCP configuration
      8ms [I8f1eda8c] {26187} [INFO] (connection - L:474) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=86df6471614e68b7) Starting. Timeout=2000000us
      9ms [I8f1eda8c] {26187} [INFO] (connection - L:147) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=86df6471614e68b7) Connected established
      12ms [I8f1eda8c] {26187} [ERROR] (SSL - L:152) error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
      13ms [I8f1eda8c] {26187} [ERROR] (negotiation - L:154) <ltx1-app31067.prod.linkedin.com:11207> (CTX=0x172ed70,sasl,SASLREQ=0x172dd40) Error: 0x37, IO Error
      13ms [I8f1eda8c] {26187} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_SSL_CANTVERIFY (0x37)
      13ms [I8f1eda8c] {26187} [INFO] (confmon - L:185) Provider 'CCCP' failed
      14ms [I8f1eda8c] {26187} [INFO] (connection - L:474) <ltx1-app31067.prod.linkedin.com:18091> (SOCK=7571b4a16ff99a7e) Starting. Timeout=2000000us
      14ms [I8f1eda8c] {26187} [INFO] (connection - L:147) <ltx1-app31067.prod.linkedin.com:18091> (SOCK=7571b4a16ff99a7e) Connected established
      17ms [I8f1eda8c] {26187} [ERROR] (SSL - L:152) error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
      17ms [I8f1eda8c] {26187} [INFO] (confmon - L:185) Provider 'HTTP' failed
      17ms [I8f1eda8c] {26187} [ERROR] (bootstrap - L:170) Failed to bootstrap client=0x170e8a0. Error=LCB_SSL_CANTVERIFY (0x37), Message=No more bootstrap providers remainFailed to connect: Client could not verify server's certificate
       
       

       

      As seen above, the client failed to verify server's cert even though I had specified --truststorepath on the commandline

       

      When I switch from password auth to certificate auth, then it works:

      [bweir@ltx1-app19835 ~]$ cbc-pillowfight -U couchbases://ltx1-app31067.prod.linkedin.com/loadtest --truststorepath /etc/riddler/ca-bundle.crt --keypath ~/identity.key --certpath ~/identity.cert -v
      Running. Press Ctrl-C to terminate...
      0ms [I51df37c1] {27608} [INFO] (instance - L:466) Version=2.8.7, Changeset=081e8b16b991bf706eb77f8243935c6fba31b895
      0ms [I51df37c1] {27608} [INFO] (instance - L:467) Effective connection string: couchbases://ltx1-app31067.prod.linkedin.com/loadtest?truststorepath=/etc/riddler/ca-bundle.crt&certpath=/export/home/bweir/identity.cert&keypath=/export/home/bweir/identity.key&console_log_level=2&. Bucket=loadtest
      8ms [I51df37c1] {27608} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
      8ms [I51df37c1] {27608} [INFO] (cccp - L:151) Requesting connection to node ltx1-app31067.prod.linkedin.com:11207 for CCCP configuration
      8ms [I51df37c1] {27608} [INFO] (connection - L:474) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=1ca76b23d8385b32) Starting. Timeout=2000000us
      9ms [I51df37c1] {27608} [INFO] (connection - L:147) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=1ca76b23d8385b32) Connected established
      29ms [I51df37c1] {27608} [INFO] (lcbio_mgr - L:498) <ltx1-app31067.prod.linkedin.com:11207> (HE=0x90ce10) Placing socket back into the pool. I=0x90cfd0,C=0x919390
      30ms [I51df37c1] {27608} [INFO] (confmon - L:160) Setting new configuration. Received via CCCP
      32ms [I51df37c1] {27613} [INFO] (connection - L:474) <ltx1-app25928.prod.linkedin.com:11207> (SOCK=1693e6671fd4128e) Starting. Timeout=2500000us
      32ms [I51df37c1] {27613} [INFO] (connection - L:474) <ltx1-app26263.prod.linkedin.com:11207> (SOCK=35b81b54b0c2e2b7) Starting. Timeout=2500000us
      32ms [I51df37c1] {27613} [INFO] (connection - L:474) <ltx1-app26336.prod.linkedin.com:11207> (SOCK=592b3ca88a613a1a) Starting. Timeout=2500000us
      32ms [I51df37c1] {27613} [INFO] (connection - L:474) <ltx1-app26347.prod.linkedin.com:11207> (SOCK=17e70313b01d51dc) Starting. Timeout=2500000us
      ... 

       

      For completeness, here's specifying certificate auth but leaving out --truststorepath which leads to a "Client could not verify server's certificate" error as expected:

      [bweir@ltx1-app19835 ~]$ cbc-pillowfight -U couchbases://ltx1-app31067.prod.linkedin.com/loadtest --keypath ~/identity.key --certpath ~/identity.cert -v
      Running. Press Ctrl-C to terminate...
      0ms [I109af43d] {29293} [INFO] (instance - L:466) Version=2.8.7, Changeset=081e8b16b991bf706eb77f8243935c6fba31b895
      0ms [I109af43d] {29293} [INFO] (instance - L:467) Effective connection string: couchbases://ltx1-app31067.prod.linkedin.com/loadtest?certpath=/export/home/bweir/identity.cert&keypath=/export/home/bweir/identity.key&console_log_level=2&. Bucket=loadtest
      6ms [I109af43d] {29293} [INFO] (instance - L:146) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
      6ms [I109af43d] {29293} [INFO] (cccp - L:151) Requesting connection to node ltx1-app31067.prod.linkedin.com:11207 for CCCP configuration
      6ms [I109af43d] {29293} [INFO] (connection - L:474) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=d5b7caf75fc13082) Starting. Timeout=2000000us
      7ms [I109af43d] {29293} [INFO] (connection - L:147) <ltx1-app31067.prod.linkedin.com:11207> (SOCK=d5b7caf75fc13082) Connected established
      12ms [I109af43d] {29293} [ERROR] (SSL - L:152) error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
      13ms [I109af43d] {29293} [ERROR] (negotiation - L:154) <ltx1-app31067.prod.linkedin.com:11207> (CTX=0x1a6f2b0,sasl,SASLREQ=0x1a6e300) Error: 0x37, IO Error
      13ms [I109af43d] {29293} [ERROR] (cccp - L:165) <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_SSL_CANTVERIFY (0x37)
      13ms [I109af43d] {29293} [INFO] (confmon - L:185) Provider 'CCCP' failed
      13ms [I109af43d] {29293} [INFO] (connection - L:474) <ltx1-app31067.prod.linkedin.com:18091> (SOCK=94c40a19ea55c802) Starting. Timeout=2000000us
      13ms [I109af43d] {29293} [INFO] (connection - L:147) <ltx1-app31067.prod.linkedin.com:18091> (SOCK=94c40a19ea55c802) Connected established
      16ms [I109af43d] {29293} [ERROR] (SSL - L:152) error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
      17ms [I109af43d] {29293} [INFO] (confmon - L:185) Provider 'HTTP' failed
      17ms [I109af43d] {29293} [ERROR] (bootstrap - L:170) Failed to bootstrap client=0x1a4a8d0. Error=LCB_SSL_CANTVERIFY (0x37), Message=No more bootstrap providers remainFailed to connect: Client could not verify server's certificate
       

      My expectation is that the --truststorepath parameter would be used any time a TLS connection is requested, regardless of whether certificate or password auth is being used.

      I have verified using strace that my trust store file is never being opened in the first case.

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          avsej Sergey Avseyev added a comment -

          Thanks for report, bweir. I will fix it and publish with the next release. Current workaround would be to use -certpath for trusted store in case when you don't have -keypath.

          https://github.com/couchbase/libcouchbase/blob/c3bdd2d6f8e74118df546bbeb3efccba60c833f5/src/ssl/ssl_common.c#L284-L289

          Show
          avsej Sergey Avseyev added a comment - Thanks for report, bweir . I will fix it and publish with the next release. Current workaround would be to use - certpath for trusted store in case when you don't have -keypath . https://github.com/couchbase/libcouchbase/blob/c3bdd2d6f8e74118df546bbeb3efccba60c833f5/src/ssl/ssl_common.c#L284-L289
          Hide
          build-team Couchbase Build Team added a comment -

          Build libcouchbase-2.8.5-319 contains libcouchbase commit b445a11 with commit message:
          CCBC-1007: allow using trusted store path without key file

          Show
          build-team Couchbase Build Team added a comment - Build libcouchbase-2.8.5-319 contains libcouchbase commit b445a11 with commit message: CCBC-1007 : allow using trusted store path without key file
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-6.5.0-1891 contains libcouchbase commit b445a11 with commit message:
          CCBC-1007: allow using trusted store path without key file

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1891 contains libcouchbase commit b445a11 with commit message: CCBC-1007 : allow using trusted store path without key file

            People

            • Assignee:
              avsej Sergey Avseyev
              Reporter:
              bweir bweir
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.