Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48165

[Eventing][n2n encryption + x509 cert]: REST calls fail after changing encryption level to "all"

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 7.1.0, 7.0.2
    • 7.1.0
    • eventing

    Description

      Build - 7.0.2 - 6558

      STEPS TO REPRODUCE

      • Generate x509 root, node, client cert on all servers of the cluster.
      • Upload root certs and client-cert settings on servers.
      • Uploads node certs on servers.
      • Disable n2n encryption.
      • Create and deploy handler, load docs into src bucket and verify mutations are processed or not.
      • Undeploy handler, enable n2n encryption, deploy handler, delete docs from src bucket and verify mutations are processed or not.
        No issues observed.
      • Undeploy handler, change encryption level to all, deploy handler, load docs into src bucket and verify mutations are processed or not.
        REST calls fail.

      On 172.23.106.67
      eventing.log

      2021-08-25T03:59:58.118-07:00 [Info] Updating node-to-node encryption level:
      		 
      {EncryptData:true DisableNonSSLPorts:false}
      		 
      2021-08-25T03:59:58.118-07:00 [Info] serviceChangeNotifier: received EncryptionLevelChangeNotification
      		 
      2021-08-25T03:59:58.134-07:00 [Info] ServiceMgr::functionsHandler REST Call: /api/v1/functions/Function_651451090_test_eventing_with_n2n_encryption_enabled/deploy POST
      		 
      2021-08-25T03:59:58.135-07:00 [Info] ServiceMgr::getTempStore Function: Function_651451090_test_eventing_with_n2n_encryption_enabled fetching function draft definitions
      		 
      2021-08-25T03:59:58.141-07:00 [Info] ServiceMgr::setSettings Function: Function_651451090_test_eventing_with_n2n_encryption_enabled save settings
      		 
      2021-08-25T03:59:58.141-07:00 [Info] ServiceMgr::getTempStore Function: Function_651451090_test_eventing_with_n2n_encryption_enabled fetching function draft definitions
      		 
      2021-08-25T03:59:58.148-07:00 [Info] ServiceMgr::setSettings Function: Function_651451090_test_eventing_with_n2n_encryption_enabled settings params: map[deployment_status:true processing_status:true]
      		 
      2021/08/25 03:59:58 http: TLS handshake error from 172.23.106.67:47890: remote error: tls: bad certificate
      		 
      2021-08-25T03:59:58.158-07:00 [Error] util::GetNodeUUIDs Failed to fetch node uuid from url: https://172.23.106.67:18096/uuid, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      		 
      2021-08-25T03:59:58.158-07:00 [Error] ServiceMgr::getActiveNodeAddrs Failed to get eventing node uuids, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      		 
      2021-08-25T03:59:58.158-07:00 [Error] ServiceMgr::compareEventingVersion failed to get active eventing nodes, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      		 
      2021-08-25T03:59:58.159-07:00 [Info] ServiceMgr::getConfig Retrieving config from metakv: map[enable_debugger:false ram_quota:512]
      		 
      2021-08-25T03:59:58.163-07:00 [Error] util::CheckIfRebalanceOngoing Failed to gather rebalance status from url: https://172.23.106.67:18096/getRebalanceStatus, err: Get https://172.23.106.67:18096/getRebalanceStatus: x509: certificate signed by unknown authority
      		 
      2021-08-25T03:59:58.163-07:00 [Error] ServiceMgr::checkRebalanceStatus Failed to grab correct rebalance or failover status from some/all Eventing nodes, err: Get https://172.23.106.67:18096/getRebalanceStatus: x509: certificate signed by unknown authority
      		 
      2021-08-25T03:59:58.163-07:00 [Error] ServiceMgr:enableLifeCycleOpsDuringRebalance Failed to get rebalance or failover status from eventing nodes
      		 
      2021/08/25 03:59:58 http: TLS handshake error from 172.23.106.67:47892: remote error: tls: bad certificate
      		 
      2021-08-25T03:59:58.189-07:00 [Error] util::GetNodeUUIDs Failed to fetch node uuid from url: https://172.23.106.67:18096/uuid, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      		 
      2021-08-25T03:59:58.189-07:00 [Error] ServiceMgr::getActiveNodeAddrs Failed to get eventing node uuids, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      		 
      2021/08/25 03:59:58 http: TLS handshake error from 172.23.106.67:47894: remote error: tls: bad certificate
      		 
      2021-08-25T03:59:58.189-07:00 [Warn] ServiceMgr::getAppList failed to fetch active Eventing nodes, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      		 
      2021-08-25T03:59:58.189-07:00 [Warn] Unknown status code: 37

      Attachments

        Issue Links

          backports to

          Bug - A problem which impairs or prevents the functions of the product. MB-48195 [BP 7.0.2 MB-48165] - [Eventing][n2n encryption + x509 cert]: REST calls fail after changing encryption level to "all"

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ascending order - Click to sort in descending order
            sujay.gad Sujay Gad created issue -
            sujay.gad Sujay Gad made changes -
            Assignee Jeelan Poola [ jeelan.poola ]
            chanabasappa.ghali Chanabasappa Ghali made changes -
            Assignee Jeelan Poola [ jeelan.poola ] Ankit Prabhu [ ankit.prabhu ]
            chanabasappa.ghali Chanabasappa Ghali made changes -
            Assignee Ankit Prabhu [ ankit.prabhu ] Sujay Gad [ JIRAUSER25279 ]
            abhishek.jindal Abhishek Jindal made changes -
            Link This issue is cloned by MB-48195 [ MB-48195 ]
            sujay.gad Sujay Gad made changes -
            Assignee Sujay Gad [ JIRAUSER25279 ] Abhishek Jindal [ abhishek.jindal ]
            wayne Wayne Siu made changes -
            Link This issue backports to MB-48195 [ MB-48195 ]
            wayne Wayne Siu made changes -
            Link This issue is cloned by MB-48195 [ MB-48195 ]
            abhishek.jindal Abhishek Jindal added a comment -

            Sujay Gad Please feel free to pick up the latest changes and test.

            In order to test this fix, you would need to refresh cluster / ca certs, change n2n encryption to all / strict and check whether REST calls work.

            Thanks,

            abhishek.jindal Abhishek Jindal added a comment - Sujay Gad Please feel free to pick up the latest changes and test. In order to test this fix, you would need to refresh cluster / ca certs, change n2n encryption to all / strict and check whether REST calls work. Thanks,
            abhishek.jindal Abhishek Jindal made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Sujay Gad [ JIRAUSER25279 ]
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.1.0-1236 contains eventing commit cd1f58e with commit message:
            MB-48165 : Restart TLS server with new certs instead of relying on KPR

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1236 contains eventing commit cd1f58e with commit message: MB-48165 : Restart TLS server with new certs instead of relying on KPR
            sujay.gad Sujay Gad added a comment -

            Validated the fix on Enterprise Edition 7.1.0 build 2081.

            sujay.gad Sujay Gad added a comment - Validated the fix on Enterprise Edition 7.1.0 build 2081.
            sujay.gad Sujay Gad made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              sujay.gad Sujay Gad
              sujay.gad Sujay Gad
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty