Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-63001

Client certificate authentication for cluster admin failing in 7.2.6/7.6.3

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Yes

    Description

      In a recent regression, client certificate authentication has started to fail (always returning 401) against ns_server REST endpoints on 7.6.3.

      Example scenario:

      1. Create a couchbase cluster, configure admin user as couchbase
      2. Configure cluster for client auth via post to /settings/clientCertAuth of following 

        {"state":"enable","prefixes":[{"path":"subject.cn","prefix":"","delimiter":""}]}

        , ensuring 202 is returned

      3. Generate client certificate signed by cluster certificate
      4. Attempt to present certificate against ns_server REST api (e.g. /pools/default/nodeServices, resulting in 401.
      5. Observe following in debug log 

        [ns_server:debug,2024-07-30T01:01:07.492Z,ns_1@n1.couchbase.host:ns_audit<0.708.0>:ns_audit:handle_call:168]Audit auth_failure: [{local,{[{ip,<<"192.168.176.3">>},{port,18091}]}},
        		 
                             {remote,{[{ip,<<"192.168.176.2">>},{port,40278}]}},
        		 
                             {timestamp,<<"2024-07-30T01:01:07.492Z">>},

      Output from test client generating certs:

      2024-07-29T18:05:23.192-07:00 INFO DockerTestBase [main] Generating certificates for cluster 1
      		 
      2024-07-29T18:05:23.253-07:00 INFO DockerTestBase [main] Executing openssl command: genrsa -out cluster1/private/ca0.key 2048
      		 
      2024-07-29T18:05:23.335-07:00 INFO DockerTestBase [main] Executing openssl command: req -config /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-server/target/test-classes/tls/rootConfig.conf -new -x509 -days 90 -sha256 -key cluster1/private/ca0.key -out cluster1/public/ca0.pem -subj /C=UA/O=MyCompany/CN=RootCA
      		 
      2024-07-29T18:05:23.345-07:00 INFO DockerTestBase [main] Executing openssl command: genrsa -out cluster1/private/ca1.key 2048
      		 
      2024-07-29T18:05:23.484-07:00 INFO DockerTestBase [main] Executing openssl command: req -config /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-server/target/test-classes/tls/rootConfig.conf -new -x509 -days 90 -sha256 -key cluster1/private/ca1.key -out cluster1/public/ca1.pem -subj /C=UA/O=MyCompany/CN=RootCA
      		 
      2024-07-29T18:05:23.492-07:00 INFO DockerTestBase [main] Executing openssl command: req -config /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-server/target/test-classes/tls/rootConfig.conf -new -x509 -days 90 -sha256 -key cluster1/private/ca0.key -out cluster1/public/ca0.pem -subj /C=UA/O=MyCompany/CN=RootCA
      		 
      2024-07-29T18:05:23.503-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -text -noout -in cluster1/public/ca0.pem
      		 
      2024-07-29T18:05:23.516-07:00 INFO DockerTestBase [main] Executing openssl command: genrsa -out cluster1/private/int.key 1024
      		 
      2024-07-29T18:05:23.534-07:00 INFO DockerTestBase [main] Executing openssl command: req -new -days 90 -sha256 -key cluster1/private/int.key -out cluster1/requests/server-signing.csr -subj /C=UA/O=MyCompany/OU=Servers/CN=ServerSigningCA
      		 
      2024-07-29T18:05:23.541-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -CA cluster1/public/ca0.pem -CAkey cluster1/private/ca0.key -CAcreateserial -CAserial cluster1/public/server-signing.srl -days 90 -req -in cluster1/requests/server-signing.csr -out cluster1/public/int.pem -extfile /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-server/target/test-classes/tls/int.ext
      		 
      2024-07-29T18:05:23.560-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -text -noout -in cluster1/public/int.pem
      		 
      2024-07-29T18:05:23.574-07:00 INFO DockerTestBase [main] Executing openssl command: genrsa -out cluster1/private/intNonCA.key 1024
      		 
      2024-07-29T18:05:23.583-07:00 INFO DockerTestBase [main] Executing openssl command: req -new -days 90 -sha256 -key cluster1/private/intNonCA.key -out cluster1/requests/server-signing.csr -subj /C=UA/O=MyCompany/OU=Servers/CN=ServerSigningCA
      		 
      2024-07-29T18:05:23.589-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -CA cluster1/public/ca0.pem -CAkey cluster1/private/ca0.key -CAcreateserial -CAserial cluster1/public/server-signing.srl -days 90 -req -in cluster1/requests/server-signing.csr -out cluster1/public/intNonCA.pem -extfile /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-server/target/test-classes/tls/intNonCA.ext
      		 
      2024-07-29T18:05:23.605-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -text -noout -in cluster1/public/intNonCA.pem
      		 
      2024-07-29T18:05:23.618-07:00 INFO DockerTestBase [main] Executing openssl command: genrsa -out cluster1/private/couchbase.default.svc.n_1.key 2048
      		 
      2024-07-29T18:05:23.687-07:00 INFO DockerTestBase [main] Executing openssl command: req -new -sha256 -key cluster1/private/couchbase.default.svc.n_1.key -out cluster1/requests/couchbase.default.svc.n_1.csr -subj /CN=Couchbase Server
      		 
      2024-07-29T18:05:23.698-07:00 INFO DockerTestBase [main] Executing openssl command: req -text -noout -verify -in cluster1/requests/couchbase.default.svc.n_1.csr
      		 
      2024-07-29T18:05:23.705-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -CA cluster1/public/int.pem -CAkey cluster1/private/int.key -CAcreateserial -days 90 -req -sha256 -in cluster1/requests/couchbase.default.svc.n_1.csr -out cluster1/public/couchbase.default.svc.n_1.pem -extfile cluster1/server.ext
      		 
      2024-07-29T18:05:23.722-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -text -noout -in cluster1/public/couchbase.default.svc.n_1.pem
      		 
      2024-07-29T18:05:23.744-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host mkdir -p /opt/couchbase/var/lib/couchbase/inbox/CA
      		 
      2024-07-29T18:05:24.081-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host chmod a+wx /opt/couchbase/var/lib/couchbase/inbox/CA/ca0.pem
      		 
      2024-07-29T18:05:24.199-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host curl -u couchbase:couchbase -X POST http://127.0.0.1:8091/node/controller/loadTrustedCAs
      		 
      2024-07-29T18:05:24.311-07:00 INFO DockerTestBase [main+] >>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
      		 
      2024-07-29T18:05:24.312-07:00 INFO DockerTestBase [main+] >>                                  Dload  Upload   Total   Spent    Left  Speed
      		 
      2024-07-29T18:05:24.312-07:00 INFO DockerTestBase [main+] >> 
      2024-07-29T18:05:24.366-07:00 INFO DockerTestBase [main+] >>   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
      		 
      2024-07-29T18:05:24.367-07:00 INFO DockerTestBase [main+] >> 100  1634  100  1634    0     0  29211      0 --:--:-- --:--:-- --:--:-- 29709
      		 
      2024-07-29T18:05:24.372-07:00 INFO DockerTestBase [main+] >> [{"id":1,"loadTimestamp":"2024-07-30T01:05:24.000Z","subject":"C=UA, O=MyCompany, CN=RootCA","notBefore":"2024-07-30T01:05:23.000Z","notAfter":"2024-10-28T01:05:23.000Z","type":"uploaded","pem":"-----BEGIN CERTIFICATE-----\nMIIDnzCCAoegAwIBAgIUKaAiXEYqzddTUDufG+7eegM7uXUwDQYJKoZIhvcNAQEL\nBQAwMjELMAkGA1UEBhMCVUExEjAQBgNVBAoMCU15Q29tcGFueTEPMA0GA1UEAwwG\nUm9vdENBMB4XDTI0MDczMDAxMDUyM1oXDTI0MTAyODAxMDUyM1owMjELMAkGA1UE\nBhMCVUExEjAQBgNVBAoMCU15Q29tcGFueTEPMA0GA1UEAwwGUm9vdENBMIIBIjAN\nBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtouiSRx3uxYbzJZHio6N+LaVsHmn\nke8WbKxWkLG5woo1vAB/DUL8At9ESm9bFLBzA7h+Wb+WZ2QP/VesjfnVIgqj7l2H\naUIPIosoo/w5uqaTcAWsNISUHKxJSfu49V7FBUAsc4LMDFlskS8d5gWU/gMQgtB6\npKgFTxU8ru6Ql9v9P4rJY7TXU0WUOmmVYhevDY5lZgUlrJYx7vWFT8Kv+KK9sRS3\nT8l8/moBpU67cnJd2AwNnrcxWEbkNg4f9BsLyyHqoRBHc5p1NUWF3a9PzaX8FR5q\n0Yq1w5VcYeWxWbG8LimcBKx3ZqFohd0F5tPd5CW/3aKzJTrQT64Or8H/GQIDAQAB\no4GsMIGpMAwGA1UdEwQFMAMBAf8wHQYDVR0OBBYEFH61R/FcZlMFSvO+WyyTe4k8\nR6ixMG0GA1UdIwRmMGSAFH61R/FcZlMFSvO+WyyTe4k8R6ixoTakNDAyMQswCQYD\nVQQGEwJVQTESMBAGA1UECgwJTXlDb21wYW55MQ8wDQYDVQQDDAZSb290Q0GCFCmg\nIlxGKs3XU1A7nxvu3noDO7l1MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQsFAAOC\nAQEAhEu7XMcIM7m+sjtu8QN0Od5UdkxfOENsmi/iF+ySdWMSC6ZrfolgqJHYIZRP\nQHZzuZvoDl2ENQyTLyBEKWmTuA7WwDZlG4SYqoc6rsbAdv/a9GwkxeBfSIAGEWZN\nUIDWmkN5l+2IrX6TzruBKSYvxzyPFLE+wkzhFgaCfvgdqb+pnriXtWdMpVf3nN+t\nSCk3N7mcYLqurnr2fergZa5qyjymbLyppmY+GzWz6B721vFyAAsk/rRtbF+w6msf\nPRL9uVKcp5OnxaQgbMVkxWFsQ4lNSlbIex7Ou7stXLTtHwwkr/YjVyBq3blMa7ZN\n8nd9rjMplYEeEJftgfdmtK1Ukg==\n-----END CERTIFICATE-----\n\n","loadHost":"n1.couchbase.host","loadFile":"/opt/couchbase/var/lib/couchbase/inbox/CA/ca0.pem"}]
      		 
      2024-07-29T18:05:24.374-07:00 INFO DockerTestBase [main] Executing the command: bash -c chmod a+x /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteTLSCertificatesITD/testClientCertificateAuthEncryptedPrivateKey/nodeStaging15998739862237639520/inbox/chain.pem
      		 
      2024-07-29T18:05:24.379-07:00 INFO DockerTestBase [main] Executing the command: bash -c chmod a+x /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteTLSCertificatesITD/testClientCertificateAuthEncryptedPrivateKey/nodeStaging15998739862237639520/inbox/pkey.key
      		 
      2024-07-29T18:05:24.382-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host mkdir -p /opt/couchbase/var/lib/couchbase/inbox
      		 
      2024-07-29T18:05:24.645-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host chmod -R a+wx /opt/couchbase/var/lib/couchbase/inbox
      		 
      2024-07-29T18:05:24.741-07:00 INFO DockerTestBase [main] +http://172.17.0.1:37905/node/controller/reloadCertificate
      		 
      2024-07-29T18:05:24.783-07:00 INFO DockerTestBase [main] +http://172.17.0.1:37905/settings/clientCertAuth
      		 
      2024-07-29T18:05:24.815-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:38547]
      		 
      2024-07-29T18:05:24.816-07:00 INFO DockerTestBase [main] Executing openssl command: genrsa -aes128 -passout pass:emmW316rGuzRDX2rSu6 -out cluster1/client/client_couchbase_enc.key 2048
      		 
      2024-07-29T18:05:24.891-07:00 INFO DockerTestBase [main] Executing openssl command: req -new -key cluster1/client/client_couchbase_enc.key -passin pass:emmW316rGuzRDX2rSu6 -out cluster1/client/client_couchbase_enc.csr -subj /CN=couchbase
      		 
      2024-07-29T18:05:24.899-07:00 INFO DockerTestBase [main] Executing openssl command: req -text -noout -verify -in cluster1/client/client_couchbase_enc.csr
      		 
      2024-07-29T18:05:24.905-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -CA cluster1/public/int.pem -CAkey cluster1/private/int.key -CAcreateserial -days 90 -req -in cluster1/client/client_couchbase_enc.csr -out cluster1/client/client_couchbase_enc.pem -extfile /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-server/target/test-classes/tls/client.ext
      		 
      2024-07-29T18:05:24.921-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -text -noout -in cluster1/client/client_couchbase_enc_chain.pem
      		 
      2024-07-29T18:05:25.485-07:00 ERRO UpgradeTestBase [main] test failed due to java.lang.Exception: Invalid credentials for link Default.externalcb (in line 1, at column 1); will take cbcollect_info on container teardown

      Attachments

        1. ca0.key
          2 kB
        2. ca0.pem
          1 kB
        3. cbcollect_info.zip
          1.76 MB
        4. client_couchbase.key
          2 kB
        5. client_couchbase.pem
          1 kB
        6. client.ext
          0.2 kB

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MB-62413 [client certificate auth] Only mark the tuple a match if it contains an existing user

          • Major - Major loss of function.
          • Closed
          relates to

          Task - A task that needs to be done. MB-63003 Disable remote links client cert auth tests

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. MB-63146 ns_server returns 401 when mandatory client cert is used with Administrator creds

          • Major - Major loss of function.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              shaazin.sheikh Shaazin Sheikh
              michael.blow Michael Blow
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty