Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48195

[BP 7.0.2 MB-48165] - [Eventing][n2n encryption + x509 cert]: REST calls fail after changing encryption level to "all"

    XMLWordPrintable

Details

    Description

      Build - 7.0.2 - 6558

      STEPS TO REPRODUCE

      • Generate x509 root, node, client cert on all servers of the cluster.
      • Upload root certs and client-cert settings on servers.
      • Uploads node certs on servers.
      • Disable n2n encryption.
      • Create and deploy handler, load docs into src bucket and verify mutations are processed or not.
      • Undeploy handler, enable n2n encryption, deploy handler, delete docs from src bucket and verify mutations are processed or not.
        No issues observed.
      • Undeploy handler, change encryption level to all, deploy handler, load docs into src bucket and verify mutations are processed or not.
        REST calls fail.

      On 172.23.106.67
      eventing.log

      2021-08-25T03:59:58.118-07:00 [Info] Updating node-to-node encryption level:
      {EncryptData:true DisableNonSSLPorts:false}
      2021-08-25T03:59:58.118-07:00 [Info] serviceChangeNotifier: received EncryptionLevelChangeNotification
      2021-08-25T03:59:58.134-07:00 [Info] ServiceMgr::functionsHandler REST Call: /api/v1/functions/Function_651451090_test_eventing_with_n2n_encryption_enabled/deploy POST
      2021-08-25T03:59:58.135-07:00 [Info] ServiceMgr::getTempStore Function: Function_651451090_test_eventing_with_n2n_encryption_enabled fetching function draft definitions
      2021-08-25T03:59:58.141-07:00 [Info] ServiceMgr::setSettings Function: Function_651451090_test_eventing_with_n2n_encryption_enabled save settings
      2021-08-25T03:59:58.141-07:00 [Info] ServiceMgr::getTempStore Function: Function_651451090_test_eventing_with_n2n_encryption_enabled fetching function draft definitions
      2021-08-25T03:59:58.148-07:00 [Info] ServiceMgr::setSettings Function: Function_651451090_test_eventing_with_n2n_encryption_enabled settings params: map[deployment_status:true processing_status:true]
      2021/08/25 03:59:58 http: TLS handshake error from 172.23.106.67:47890: remote error: tls: bad certificate
      2021-08-25T03:59:58.158-07:00 [Error] util::GetNodeUUIDs Failed to fetch node uuid from url: https://172.23.106.67:18096/uuid, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.158-07:00 [Error] ServiceMgr::getActiveNodeAddrs Failed to get eventing node uuids, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.158-07:00 [Error] ServiceMgr::compareEventingVersion failed to get active eventing nodes, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.159-07:00 [Info] ServiceMgr::getConfig Retrieving config from metakv: map[enable_debugger:false ram_quota:512]
      2021-08-25T03:59:58.163-07:00 [Error] util::CheckIfRebalanceOngoing Failed to gather rebalance status from url: https://172.23.106.67:18096/getRebalanceStatus, err: Get https://172.23.106.67:18096/getRebalanceStatus: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.163-07:00 [Error] ServiceMgr::checkRebalanceStatus Failed to grab correct rebalance or failover status from some/all Eventing nodes, err: Get https://172.23.106.67:18096/getRebalanceStatus: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.163-07:00 [Error] ServiceMgr:enableLifeCycleOpsDuringRebalance Failed to get rebalance or failover status from eventing nodes
      2021/08/25 03:59:58 http: TLS handshake error from 172.23.106.67:47892: remote error: tls: bad certificate
      2021-08-25T03:59:58.189-07:00 [Error] util::GetNodeUUIDs Failed to fetch node uuid from url: https://172.23.106.67:18096/uuid, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.189-07:00 [Error] ServiceMgr::getActiveNodeAddrs Failed to get eventing node uuids, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021/08/25 03:59:58 http: TLS handshake error from 172.23.106.67:47894: remote error: tls: bad certificate
      2021-08-25T03:59:58.189-07:00 [Warn] ServiceMgr::getAppList failed to fetch active Eventing nodes, err: Get https://172.23.106.67:18096/uuid: x509: certificate signed by unknown authority
      2021-08-25T03:59:58.189-07:00 [Warn] Unknown status code: 37
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            abhishek.jindal Abhishek Jindal created issue -
            abhishek.jindal Abhishek Jindal made changes -
            Field Original Value New Value
            Link This issue Clones MB-48165 [ MB-48165 ]
            abhishek.jindal Abhishek Jindal made changes -
            Fix Version/s 7.0.2 [ 18012 ]
            Fix Version/s Neo [ 17615 ]

            Under some conditions, we see that new certificate is not being loaded by the tls server. Needs to be added to 7.0.2. cc: Jeelan Poola

            abhishek.jindal Abhishek Jindal added a comment - Under some conditions, we see that new certificate is not being loaded by the tls server. Needs to be added to 7.0.2. cc: Jeelan Poola
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Sujay Gad [ JIRAUSER25279 ] Abhishek Jindal [ abhishek.jindal ]
            jeelan.poola Jeelan Poola made changes -
            Summary CLONE - [Eventing][n2n encryption + x509 cert]: REST calls fail after changing encryption level to "all" [BP 7.0.2 MB-48165] - [Eventing][n2n encryption + x509 cert]: REST calls fail after changing encryption level to "all"
            jeelan.poola Jeelan Poola made changes -
            Labels Encryption Encryption approved-for-7.0.2
            wayne Wayne Siu made changes -
            Link This issue is a backport of MB-48165 [ MB-48165 ]
            wayne Wayne Siu made changes -
            Link This issue Clones MB-48165 [ MB-48165 ]
            wayne Wayne Siu made changes -
            Link This issue blocks MB-46308 [ MB-46308 ]

            Resolving based on comments in MB-48165

            cc: Sujay Gad

            abhishek.jindal Abhishek Jindal added a comment - Resolving based on comments in MB-48165 cc: Sujay Gad
            abhishek.jindal Abhishek Jindal made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Sujay Gad [ JIRAUSER25279 ]

            Build couchbase-server-7.0.2-6618 contains eventing commit 78c8527 with commit message:
            MB-48195 : Restart TLS server with new certs instead of relying on KPR

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6618 contains eventing commit 78c8527 with commit message: MB-48195 : Restart TLS server with new certs instead of relying on KPR
            sujay.gad Sujay Gad added a comment -

            Verified using 7.0.2 - 6618.

            sujay.gad Sujay Gad added a comment - Verified using 7.0.2 - 6618.
            sujay.gad Sujay Gad made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              sujay.gad Sujay Gad
              abhishek.jindal Abhishek Jindal
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty