Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48165

[Eventing][n2n encryption + x509 cert]: REST calls fail after changing encryption level to "all"

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 7.1.0, 7.0.2
    • 7.1.0
    • eventing

    Description

      Build - 7.0.2 - 6558

      STEPS TO REPRODUCE

      • Generate x509 root, node, client cert on all servers of the cluster.
      • Upload root certs and client-cert settings on servers.
      • Uploads node certs on servers.
      • Disable n2n encryption.
      • Create and deploy handler, load docs into src bucket and verify mutations are processed or not.
      • Undeploy handler, enable n2n encryption, deploy handler, delete docs from src bucket and verify mutations are processed or not.
        No issues observed.
      • Undeploy handler, change encryption level to all, deploy handler, load docs into src bucket and verify mutations are processed or not.
        REST calls fail.

      On 172.23.106.67
      eventing.log

      2021-08-25T03:59:58.118-07:00 [Info] Updating node-to-node encryption level:
      {EncryptData:true DisableNonSSLPorts:false}
      2021-08-25T03:59:58.118-07:00 [Info] serviceChangeNotifier: received EncryptionLevelChangeNotification
      2021-08-25T03:59:58.134-07:00 [Info] ServiceMgr::functionsHandler REST Call: /api/v1/functions/Function_651451090_test_eventing_with_n2n_encryption_enabled/deploy POST
      2021-08-25T03:59:58.135-07:00 [Info] ServiceMgr::getTempStore Function: Function_651451090_test_eventing_with_n2n_encryption_enabled fetching function draft definitions
      2021-08-25T03:59:58.141-07:00 [Info] ServiceMgr::setSettings Function: Function_651451090_test_eventing_with_n2n_encryption_enabled save settings
      2021-08-25T03:59:58.141-07:00 [Info] ServiceMgr::getTempStore Function: Function_651451090_test_eventing_with_n2n_encryption_enabled fetching function draft definitions
      2021-08-25T03:59:58.148-07:00 [Info] ServiceMgr::setSettings Function: Function_651451090_test_eventing_with_n2n_encryption_enabled settings params: map[deployment_status:true processing_status:true]
      2021/08/25 03:59:58 http: TLS handshake error from 172.23.106.67:47890: remote error: tls: bad certificate
      2021-08-25T03:59:58.158-07:00 [Error] util::GetNodeUUIDs Failed to fetch node uuid from url: https://172.23.106.67:18096/uuid, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.158-07:00 [Error] ServiceMgr::getActiveNodeAddrs Failed to get eventing node uuids, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.158-07:00 [Error] ServiceMgr::compareEventingVersion failed to get active eventing nodes, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.159-07:00 [Info] ServiceMgr::getConfig Retrieving config from metakv: map[enable_debugger:false ram_quota:512]
      2021-08-25T03:59:58.163-07:00 [Error] util::CheckIfRebalanceOngoing Failed to gather rebalance status from url: https://172.23.106.67:18096/getRebalanceStatus, err: Get https://172.23.106.67:18096/getRebalanceStatus: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.163-07:00 [Error] ServiceMgr::checkRebalanceStatus Failed to grab correct rebalance or failover status from some/all Eventing nodes, err: Get https://172.23.106.67:18096/getRebalanceStatus: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.163-07:00 [Error] ServiceMgr:enableLifeCycleOpsDuringRebalance Failed to get rebalance or failover status from eventing nodes
      2021/08/25 03:59:58 http: TLS handshake error from 172.23.106.67:47892: remote error: tls: bad certificate
      2021-08-25T03:59:58.189-07:00 [Error] util::GetNodeUUIDs Failed to fetch node uuid from url: https://172.23.106.67:18096/uuid, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.189-07:00 [Error] ServiceMgr::getActiveNodeAddrs Failed to get eventing node uuids, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021/08/25 03:59:58 http: TLS handshake error from 172.23.106.67:47894: remote error: tls: bad certificate
      2021-08-25T03:59:58.189-07:00 [Warn] ServiceMgr::getAppList failed to fetch active Eventing nodes, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.189-07:00 [Warn] Unknown status code: 37
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Sujay Gad Please feel free to pick up the latest changes and test.

            In order to test this fix, you would need to refresh cluster / ca certs, change n2n encryption to all / strict and check whether REST calls work.

            Thanks,

            abhishek.jindal Abhishek Jindal added a comment - Sujay Gad Please feel free to pick up the latest changes and test. In order to test this fix, you would need to refresh cluster / ca certs, change n2n encryption to all / strict and check whether REST calls work. Thanks,

            Build couchbase-server-7.1.0-1236 contains eventing commit cd1f58e with commit message:
            MB-48165 : Restart TLS server with new certs instead of relying on KPR

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1236 contains eventing commit cd1f58e with commit message: MB-48165 : Restart TLS server with new certs instead of relying on KPR
            sujay.gad Sujay Gad added a comment -

            Validated the fix on Enterprise Edition 7.1.0 build 2081.

            sujay.gad Sujay Gad added a comment - Validated the fix on Enterprise Edition 7.1.0 build 2081.

            People

              sujay.gad Sujay Gad
              sujay.gad Sujay Gad
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty