Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47707

Enforce TLS: Established plaintext connections are not closed when plaintext listening socket is closed

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • Neo, 7.0.2
    • Neo, 7.0.2
    • documentation, memcached
    • None

    Description

      Build - 7.0.1 - 5962

      STEPS

      1. Create and deploy eventing handler.
      2. Push docs to src bucket, mutations are processed.
      3. Enforce tls.
      4. Push some more docs to src bucket.
        Mutations are getting processed which is unexpected behaviour.

      EXPECTED BEHAVIOUR
      Try pushing some new documents to the source bucket. Expectation: The eventing handler will not be able to process those mutations as memcached has now restricted it’s plain text connections to localhost if enforce TLS has been enabled.
      (As per specification document mentioned below)

      SPECIFICATION - https://docs.google.com/document/d/1mm9xHvWsBeL0n3CpWmzOquttf71ogjImywNHNE9zVTk/

      From Abhishek Jindal

      Hi Sujay, this is a KV issue. Problem here is that we / any DCP consumer or application expects KV to shutdown all non-TLS client connections to non-TLS port. However, in this case from netstat I see KV still has these connections open. Please feel free to open up a bug and assign to KV
      3:51
      Because, if we think like this - Say you have an application / CB SDK running outside the cluster running on non-TLS. Now we switch over from enforce TLS.
      We won't expect that application to automatically close out the connection and initiate on TLS. User should face a disruption in service due to Data node closing out the non-TLS connections.
      3:51
      Same is true to eventing (thinking of it as an application consuming from KV)
      3:52
      Hence, we expect KV to close all ESTABLISHED connections hence preventing any further mutations from being streamed.

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            drigby Dave Rigby added a comment -

            Sujay Gad Which of the above node(s) is the data service node which you expected the plain (11210) port to be closed on?

            drigby Dave Rigby added a comment - Sujay Gad Which of the above node(s) is the data service node which you expected the plain (11210) port to be closed on?
            sujay.gad Sujay Gad added a comment -

            Dave Rigby Data service node - 172.23.106.64.

            sujay.gad Sujay Gad added a comment - Dave Rigby  Data service node - 172.23.106.64.
            drigby Dave Rigby added a comment -

            Thanks - looking at the logs...

            drigby Dave Rigby added a comment - Thanks - looking at the logs...
            ianmccloy Ian McCloy added a comment -

            Hey David, 

            This statement in the PRD is just saying that any configuration option must be persisted over a node restart.  There wasn't any implied requirement to restart a node to initially enable it. 

            All components of Couchbase must abide by this setting when started/re-started by the cluster manager.

            ianmccloy Ian McCloy added a comment - Hey David,  This statement in the PRD is just saying that any configuration option must be persisted over a node restart.  There wasn't any implied requirement to restart a node to initially enable it.  All components of Couchbase must abide by this setting when started/re-started by the cluster manager.
            abhishek.jindal Abhishek Jindal added a comment - - edited

            Dave Rigby Following is a brief timeline of events for your reference:

            Timeline:

            • An Eventing handler named "test" was created prior to enforcing TLS hence the DCP, gocb and libcouchbase connections were over non-TLS ports as described below.
            • Enforce TLS in this cluster was enabled at the following timestamp:

            2021-08-02T23:05:00.429-07:00 [Info] Updating node-to-node encryption level: {EncryptData:true DisableNonSSLPorts:true}
            

            * Now, lets say the user creates a new document in the source bucket. Observation is that memcached streams this one mutation to the eventing-producer process over the previously created non-TLS port.

            Following is a description of netstat from KV node .64 we see the following (cbcollect was captured after enforce TLS was enabled)

            tcp        0      0 172.23.106.64:11210     172.23.106.67:49368     ESTABLISHED 62963/memcached
            tcp        0      0 172.23.106.64:11210     172.23.106.67:49360     ESTABLISHED 62963/memcached
            tcp        0      0 172.23.106.64:11210     172.23.106.67:49372     ESTABLISHED 62963/memcached
            tcp        0      0 172.23.106.64:11210     172.23.106.67:49364     ESTABLISHED 62963/memcached
            tcp        0      0 172.23.106.64:11210     172.23.106.67:49176     ESTABLISHED 62963/memcached
            

            Here node 172.23.106.67 is hosting eventing producer and consumer processes:

            connections from eventing-producer -> memcached:

            tcp        0      0 172.23.106.67:49364     172.23.106.64:11210     ESTABLISHED 34776/eventing-prod
            tcp        0      0 172.23.106.67:49360     172.23.106.64:11210     ESTABLISHED 34776/eventing-prod
            tcp        0      0 172.23.106.67:49176     172.23.106.64:11210     ESTABLISHED 34776/eventing-prod
            

            Following 2 are the libcouchbase connections from eventing consumer / cpp process:

            tcp        0      0 172.23.106.67:49368     172.23.106.64:11210     ESTABLISHED 40253/eventing-cons
            tcp        0      0 172.23.106.67:49372     172.23.106.64:11210     ESTABLISHED 40253/eventing-cons
            

            Worth noting that these connections were established prior to enforce TLS being enabled.

            As mentioned in the Enforce TLS PRD:

            The cluster must never communicate over the network, on any port, in an insecure manner with plain text once the feature is enabled.

            It was understood that memached will tear down existing non-TLS connections to all applications and DCP consumers and hence no data will be streamed over network unencrypted to any app / DCP consumer. Eventing's implementation of enforce TLS has been built keeping this behaviour into consideration.

            As Sujay Gad mentioned, thinking more in a generic sense, if memcached is not required to tear down existing non-TLS established connections, how is a user's application that is communicating to memcached from outside the cluster supposed to behave?

            Thanks,
            Abhishek

            abhishek.jindal Abhishek Jindal added a comment - - edited Dave Rigby Following is a brief timeline of events for your reference: Timeline: An Eventing handler named "test" was created prior to enforcing TLS hence the DCP, gocb and libcouchbase connections were over non-TLS ports as described below. Enforce TLS in this cluster was enabled at the following timestamp: 2021-08-02T23:05:00.429-07:00 [Info] Updating node-to-node encryption level: {EncryptData:true DisableNonSSLPorts:true} * Now, lets say the user creates a new document in the source bucket. Observation is that memcached streams this one mutation to the eventing-producer process over the previously created non-TLS port . — Following is a description of netstat from KV node .64 we see the following (cbcollect was captured after enforce TLS was enabled) tcp 0 0 172.23.106.64:11210 172.23.106.67:49368 ESTABLISHED 62963/memcached tcp 0 0 172.23.106.64:11210 172.23.106.67:49360 ESTABLISHED 62963/memcached tcp 0 0 172.23.106.64:11210 172.23.106.67:49372 ESTABLISHED 62963/memcached tcp 0 0 172.23.106.64:11210 172.23.106.67:49364 ESTABLISHED 62963/memcached tcp 0 0 172.23.106.64:11210 172.23.106.67:49176 ESTABLISHED 62963/memcached Here node 172.23.106.67 is hosting eventing producer and consumer processes: connections from eventing-producer -> memcached: tcp 0 0 172.23.106.67:49364 172.23.106.64:11210 ESTABLISHED 34776/eventing-prod tcp 0 0 172.23.106.67:49360 172.23.106.64:11210 ESTABLISHED 34776/eventing-prod tcp 0 0 172.23.106.67:49176 172.23.106.64:11210 ESTABLISHED 34776/eventing-prod Following 2 are the libcouchbase connections from eventing consumer / cpp process: tcp 0 0 172.23.106.67:49368 172.23.106.64:11210 ESTABLISHED 40253/eventing-cons tcp 0 0 172.23.106.67:49372 172.23.106.64:11210 ESTABLISHED 40253/eventing-cons Worth noting that these connections were established prior to enforce TLS being enabled. — As mentioned in the Enforce TLS PRD: The cluster must never communicate over the network, on any port, in an insecure manner with plain text once the feature is enabled. It was understood that memached will tear down existing non-TLS connections to all applications and DCP consumers and hence no data will be streamed over network unencrypted to any app / DCP consumer. Eventing's implementation of enforce TLS has been built keeping this behaviour into consideration. As Sujay Gad mentioned, thinking more in a generic sense, if memcached is not required to tear down existing non-TLS established connections, how is a user's application that is communicating to memcached from outside the cluster supposed to behave? Thanks, Abhishek
            drigby Dave Rigby added a comment -

            Abhishek Jindal Thanks, the timeline makes sense.

            Essentially it seems that we overlooked the "close existing plaintext connections" aspect of the Enforce TLS feature - I suspect because KV-Engine has supported dynamic interface changes since around 5.0 and hence was assumed that was sufficient for Enforce TLS. (I can't be 100% sure as Trond who implemented it is on holiday atm).

            I'll look into what is required to close existing connections when the listening interface which it was created off is closed.

            drigby Dave Rigby added a comment - Abhishek Jindal Thanks, the timeline makes sense. Essentially it seems that we overlooked the "close existing plaintext connections" aspect of the Enforce TLS feature - I suspect because KV-Engine has supported dynamic interface changes since around 5.0 and hence was assumed that was sufficient for Enforce TLS. (I can't be 100% sure as Trond who implemented it is on holiday atm). I'll look into what is required to close existing connections when the listening interface which it was created off is closed.
            drigby Dave Rigby added a comment -

            Trond Norbye - Could you take a look at this (when you get back)?

            drigby Dave Rigby added a comment - Trond Norbye - Could you take a look at this (when you get back)?

            Trond Norbye I can speak for web server only, I don't know the answer for outgoing dcp connections if you mean them. So, for our web server: Ns server drops all the non TLS connections but those that handle some special http requests (like node addition, for example). Basically we don't immediately restart only the ones that currently are handling requests that lead to restart of the web server (otherwise those http requests would kill themselves). Instead we let these special requests finish, and only after that we drop the connection. So, eventually all the non tls web server connections should be dropped.

            timofey.barmin Timofey Barmin added a comment - Trond Norbye I can speak for web server only, I don't know the answer for outgoing dcp connections if you mean them. So, for our web server: Ns server drops all the non TLS connections but those that handle some special http requests (like node addition, for example). Basically we don't immediately restart only the ones that currently are handling requests that lead to restart of the web server (otherwise those http requests would kill themselves). Instead we let these special requests finish, and only after that we drop the connection. So, eventually all the non tls web server connections should be dropped.
            dfinlay Dave Finlay added a comment - - edited

            My view on this is that in general the clients should close existing connections when they find out that the server has moved to strict mode (or from encryption level "control" to "all"; or from IPvX to IPvX-only).

            The reason is the races that can happen otherwise. If memcached shuts down a connection after it has learned of the configuration change but before the client has learned of the configuration change, the client will attempt to reconnect on an address family or port that is now no longer available. Eventually things should resolve, but in the cluster goes through this period of unavailability.

            An argument that was given against this was that the SDKs don't automatically learn of the configuration change and reconnect. My view is that they should do this. I think the CCCP / pools/default/nodeServices payload should ideally include the encryption level / address family only settings and SDK clients should shut their connections and attempt to connect on a new connection when they see a change that requires reconnection.

            I think the above model is the principled way to approach this issue.

            Near term, if we want to avoid a lot of work in the SDKs perhaps we can do the following:

            • Have services detect the configuration change and reconnect to KV (or other connections such as query -> indexing etc)
            • Document that customers may need to restart their applications to remove existing unencrypted connections

            If everyone is OK with KV summarily closing client connections maybe we can go that route, but I have to say it makes me feel somewhat uncomfortable.

            We may need to get together and talk about it.

            dfinlay Dave Finlay added a comment - - edited My view on this is that in general the clients should close existing connections when they find out that the server has moved to strict mode (or from encryption level "control" to "all"; or from IPvX to IPvX-only). The reason is the races that can happen otherwise. If memcached shuts down a connection after it has learned of the configuration change but before the client has learned of the configuration change, the client will attempt to reconnect on an address family or port that is now no longer available. Eventually things should resolve, but in the cluster goes through this period of unavailability. An argument that was given against this was that the SDKs don't automatically learn of the configuration change and reconnect. My view is that they should do this. I think the CCCP / pools/default/nodeServices payload should ideally include the encryption level / address family only settings and SDK clients should shut their connections and attempt to connect on a new connection when they see a change that requires reconnection. I think the above model is the principled way to approach this issue. Near term, if we want to avoid a lot of work in the SDKs perhaps we can do the following: Have services detect the configuration change and reconnect to KV (or other connections such as query -> indexing etc) Document that customers may need to restart their applications to remove existing unencrypted connections If everyone is OK with KV summarily closing client connections maybe we can go that route, but I have to say it makes me feel somewhat uncomfortable. We may need to get together and talk about it.
            jeelan.poola Jeelan Poola added a comment -

            Ian McCloy No down time or zero impact (no in-flight KV/N1QL/FTS/Eventing/Client request processing lost) switch over to 'enforce-tls' requires graceful switch over to secure connections which essentially means close coordination across services & clients and significantly more work at this point in 7.0.1. And since enabling 'enforce-tls' is an administrative operation, it can be expected (or recommended) that it will be done during a maintenance window by our customers. In which case we can request a restart of couchbase server when 'enforce-tls' is enabled in a cluster. I believe the RoI on 'No down time' requirement is not high enough to justify the cost to make things work accurately across services & applications (testing will be costly as well). Hence I request & suggest we drop 'No down time' requirement altogether.

            Dave Finlay My view is that, in a typical client-server model, if things change on the server, it can/must initiate a necessary action (closing a connection in this case) forcing the client rediscover the server appropriately. But this becomes moot if the 'no downtime' requirement is dropped.

            jeelan.poola Jeelan Poola added a comment - Ian McCloy No down time or zero impact (no in-flight KV/N1QL/FTS/Eventing/Client request processing lost) switch over to 'enforce-tls' requires graceful switch over to secure connections which essentially means close coordination across services & clients and significantly more work at this point in 7.0.1. And since enabling 'enforce-tls' is an administrative operation, it can be expected (or recommended) that it will be done during a maintenance window by our customers. In which case we can request a restart of couchbase server when 'enforce-tls' is enabled in a cluster. I believe the RoI on 'No down time' requirement is not high enough to justify the cost to make things work accurately across services & applications (testing will be costly as well). Hence I request & suggest we drop 'No down time' requirement altogether. Dave Finlay My view is that, in a typical client-server model, if things change on the server, it can/must initiate a necessary action (closing a connection in this case) forcing the client rediscover the server appropriately. But this becomes moot if the 'no downtime' requirement is dropped.
            dfinlay Dave Finlay added a comment - - edited

            I met with Ian, Trond and Meni this morning, primarily to talk about requirements but also to discuss some aspects of the implementation. Hopefully this update should make the work tractable for folks in 7.0.2. These discussions were mostly in the context of TLS enforcement, but generally also apply to the "address family only" support.

            1. It must be the case that by some well-bounded interval after the encryption level is set to strict, there is no unencrypted communication happening over non-loopback interfaces in the cluster. This "well-bounded interval" can be as much as say an hour, but existing connections should not continue to survive potentially indefinitely.
              1. If there are clients that are communicating unencrypted with the cluster, they will be interrupted, possibly permanently (if they are set to bootstrap off unencrypted ports or if they are not set up with the right certs) when the change to strict happens. This is just a consequence of setting the encryption level to strict.
            2. It is permissible that there is some disruption to front-end traffic, replication, intra-cluster traffic during this change, though obviously the less the better.

            Given the situation we have currently where SDK clients are not aware of cluster settings around encryption level, requirement #1 means that servers must terminate unencrypted connections as clients will not terminate connections and reconnect on their own.

            It's my personal view that the most principled way to satisfy the requirements is the following:

            • Once servers learn of the configuration change they should allow some grace period (say 5 minutes) before terminating unencrypted connections.
            • During this time, if and when clients learn about the configuration change they should adjust in an attempt to minimize disruption.

            However, this is just my personal view and in terms of MVP, the following should be true:

            • servers can choose to immediately terminate all unencrypted connections and are not required to let currently executing requests run to completion (since interruption is permissible)
            • clients do not have to terminate their own connections - they can rely on the fact that servers will eventually terminate them. Of course clients can terminate their own connections if they would like to and it may result in some amount of chaos as connections get torn down and re-established only to get torn down again, but this is considered permissible disruption.

            In particular: the fact that the "well-bounded interval" can be quite long (an hour) means that short-lived requests such as almost all REST API requests can be allowed to run to completion. However, HTTP keep alive could pose a problem here and needs to be tested.

            Customers who need immediate compliance (i.e. no waiting for an hour) with an encryption level of strict will need to rolling restart the cluster (or some other technique like a series of swap rebalances if the desire is to make this happen with no downtime.)

            As mentioned, hopefully this makes the feature tractable for folks in 7.0.2. If not, let's continue to talk.

            dfinlay Dave Finlay added a comment - - edited I met with Ian, Trond and Meni this morning, primarily to talk about requirements but also to discuss some aspects of the implementation. Hopefully this update should make the work tractable for folks in 7.0.2. These discussions were mostly in the context of TLS enforcement, but generally also apply to the "address family only" support. It must be the case that by some well-bounded interval after the encryption level is set to strict , there is no unencrypted communication happening over non-loopback interfaces in the cluster. This "well-bounded interval" can be as much as say an hour, but existing connections should not continue to survive potentially indefinitely. If there are clients that are communicating unencrypted with the cluster, they will be interrupted, possibly permanently (if they are set to bootstrap off unencrypted ports or if they are not set up with the right certs) when the change to strict happens. This is just a consequence of setting the encryption level to strict . It is permissible that there is some disruption to front-end traffic, replication, intra-cluster traffic during this change, though obviously the less the better. Given the situation we have currently where SDK clients are not aware of cluster settings around encryption level, requirement #1 means that servers must terminate unencrypted connections as clients will not terminate connections and reconnect on their own. It's my personal view that the most principled way to satisfy the requirements is the following: Once servers learn of the configuration change they should allow some grace period (say 5 minutes) before terminating unencrypted connections. During this time, if and when clients learn about the configuration change they should adjust in an attempt to minimize disruption. However, this is just my personal view and in terms of MVP, the following should be true: servers can choose to immediately terminate all unencrypted connections and are not required to let currently executing requests run to completion (since interruption is permissible) clients do not have to terminate their own connections - they can rely on the fact that servers will eventually terminate them. Of course clients can terminate their own connections if they would like to and it may result in some amount of chaos as connections get torn down and re-established only to get torn down again, but this is considered permissible disruption. In particular: the fact that the "well-bounded interval" can be quite long (an hour) means that short-lived requests such as almost all REST API requests can be allowed to run to completion. However, HTTP keep alive could pose a problem here and needs to be tested. Customers who need immediate compliance (i.e. no waiting for an hour) with an encryption level of strict will need to rolling restart the cluster (or some other technique like a series of swap rebalances if the desire is to make this happen with no downtime.) As mentioned, hopefully this makes the feature tractable for folks in 7.0.2. If not, let's continue to talk.

            Build couchbase-server-7.0.1-6002 contains kv_engine commit af43b5b with commit message:
            MB-47707: Shut down connections when the parent port dies

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.1-6002 contains kv_engine commit af43b5b with commit message: MB-47707 : Shut down connections when the parent port dies
            jeelan.poola Jeelan Poola added a comment - - edited

            Dave FinlayIan McCloy I may be missing something, it appears this is still vague from other services, clients & customers PoV. Few Qs & comments,

            1. What happens after "well-bounded interval" (1 hour)? Will servers terminate connections or should clients terminate connections?
            2. What really happens when connections are terminated? If front-end traffic disruption is expected, will it be acceptable to customers?
            3. Why not simply ask customers to enable 'enforce-tls or address-family-only' during maintenance window with rolling restart of the nodes in the cluster? After all, these are infrequent serious admin ops.
            4. IMHO, customers would appreciate simple & clear messaging on what to do & expect when enabling 'enforce-tls or address-family-only'. Our proposed approach with "well-bounded interval" comes across as vague in the sense they don't know exactly when all connections will move over to TLS/IPvX family and what kind of traffic disruption customers should expect.

            I would like to get a better understanding of why [3] is not a preferred approach. We do ask for rolling-restarts when turning-on encryption in the cluster and 'enforce-tls' should be no different. IMO, We can expect serious security conscious customers to actually turn-on encryption & enforce-tls at the same time. But, it's possible I am missing something that was discussed yesterday. Thank you!

            jeelan.poola Jeelan Poola added a comment - - edited Dave Finlay Ian McCloy I may be missing something, it appears this is still vague from other services, clients & customers PoV. Few Qs & comments, 1. What happens after "well-bounded interval" (1 hour)? Will servers terminate connections or should clients terminate connections? 2. What really happens when connections are terminated? If front-end traffic disruption is expected, will it be acceptable to customers? 3. Why not simply ask customers to enable 'enforce-tls or address-family-only' during maintenance window with rolling restart of the nodes in the cluster? After all, these are infrequent serious admin ops. 4. IMHO, customers would appreciate simple & clear messaging on what to do & expect when enabling 'enforce-tls or address-family-only'. Our proposed approach with "well-bounded interval" comes across as vague in the sense they don't know exactly when all connections will move over to TLS/IPvX family and what kind of traffic disruption customers should expect. I would like to get a better understanding of why [3] is not a preferred approach. We do ask for rolling-restarts when turning-on encryption in the cluster and 'enforce-tls' should be no different. IMO, We can expect serious security conscious customers to actually turn-on encryption & enforce-tls at the same time. But, it's possible I am missing something that was discussed yesterday. Thank you!
            ianmccloy Ian McCloy added a comment -

            1.)  The expectation is that the clients (where possible) should terminate non-TLS connections, if that doesn't happen the server-side will terminate non-TLS connections once the threshold is reached. 

            2.)  The threshold is designed to allow existing operations to complete,  such as an ongoing N1QL requests with minimal interruption.  But eventually new requests must be made on encrypted connections only.

            3.)  The entire reason that many customers choose to use Couchbase vs other solutions is for "Always-Available, Always-On" use-cases.  Maintenance windows are a legacy concept and don't really exist in practice for many use-cases. In every possible situation Couchbase Server should be designed to meet the goal of uninterrupted operation.  We will document that customers who require immediate transition to TLS can optionally perform a cluster wide rebalance in/out of every node.  We do not want this to be the default option as it isn't a good user experience that puts unnecessary burden on the customer.  Additionally customers who are deployed in a secure fashion where all connections are already running over TLS will have limited or no perceivable disruption with implementing the feature.

            4.)  I think it's clear for customers.  Customers follow the documentation and ensure that their client applications and various scripts are using TLS as a first step, also they need to ensure that they have N2N already enabled and XDCR is using secure connections. Then they enable the enforce option and after the max threshold, say we decide on 1hr, there shouldn't be any non-TLS network connections to the nodes. Customers who do not want to wait for the 1hr threshold can optionally rebalance every node in/out. 

            You say that "We do ask for rolling-restarts when turning-on encryption". 

            That isn't something I'm aware of, AFAIK that shouldn't be required.  
            Here's the docs for enabling cluster side TLS: https://docs.couchbase.com/server/current/manage/manage-security/configure-server-certificates.html#root-and-node-certificates 
            And the docs for N2N  https://docs.couchbase.com/server/current/manage/manage-nodes/apply-node-to-node-encryption.html 

            ianmccloy Ian McCloy added a comment - 1.)  The expectation is that the clients (where possible) should terminate non-TLS connections, if that doesn't happen the server-side will terminate non-TLS connections once the threshold is reached.  2.)  The threshold is designed to allow existing operations to complete,  such as an ongoing N1QL requests with minimal interruption.  But eventually new requests must be made on encrypted connections only. 3.)  The entire reason that many customers choose to use Couchbase vs other solutions is for "Always-Available, Always-On" use-cases.  Maintenance windows are a legacy concept and don't really exist in practice for many use-cases. In every possible situation Couchbase Server should be designed to meet the goal of uninterrupted operation.  We will document that customers who require immediate transition to TLS can optionally perform a cluster wide rebalance in/out of every node.  We do not want this to be the default option as it isn't a good user experience that puts unnecessary burden on the customer.  Additionally customers who are deployed in a secure fashion where all connections are already running over TLS will have limited or no perceivable disruption with implementing the feature. 4.)  I think it's clear for customers.  Customers follow the documentation and ensure that their client applications and various scripts are using TLS as a first step, also they need to ensure that they have N2N already enabled and XDCR is using secure connections. Then they enable the enforce option and after the max threshold, say we decide on 1hr, there shouldn't be any non-TLS network connections to the nodes. Customers who do not want to wait for the 1hr threshold can optionally rebalance every node in/out.  You say that "We do ask for rolling-restarts when turning-on encryption".  That isn't something I'm aware of, AFAIK that shouldn't be required.   Here's the docs for enabling cluster side TLS: https://docs.couchbase.com/server/current/manage/manage-security/configure-server-certificates.html#root-and-node-certificates   And the docs for N2N  https://docs.couchbase.com/server/current/manage/manage-nodes/apply-node-to-node-encryption.html  
            jeelan.poola Jeelan Poola added a comment -

            Thank you Ian McCloy! However, I still do not see how we can achieve a non-disruptive & no-downtime switch over to TLS connections without additional work in 7.0.2. Consider below,

            [1] When we say, wherever possible, clients should terminate connections, are we expecting that customers would do rolling-restart of their running application instances in a way that some instances continue to serve operate over non-secure-connections while other instances slowly move over to secure connections? If yes, it is still a manual process (something needs a restart) while the non-disruptive-service-availability goal is achievable this way.

            [2] If customers do not follow the process above, when server closes connections after 1 hour, service disruption is expected.

            [3] If automatic switch over to TLS connections is expected, the SDKs need to be intelligent enough to detect 'enforce-tls' and gracefully switch over to TLS connections without requiring application restart. I believe this is not possible with 7.0.2.

            [4] While Couchbase services, like eventing, gsi, views must detect 'enforce-tls' and gracefully switch over to TLS connections. This requires additional work and can not be accommodated into current 7.0.2 timelines. The current design assumes that server closes connections and when we restart connections, we check for 'enforce-tls' and reconnect appropriately. Additionally, eventing functions need to be paused/resumed explicitly when 'enforce-tls' is enabled which we discussed earlier if you recall.

            [5] We found references to couchbase server restart requirement on enabling encryption on/off in n2n design specifications doc at https://docs.google.com/document/d/1gKtIi7aRK_TqlALT8PkQjUubrjR20hTeKiInzeF4iIM/edit#. Not sure, if this is still the case or we have removed the restriction along the way.

            jeelan.poola Jeelan Poola added a comment - Thank you Ian McCloy ! However, I still do not see how we can achieve a non-disruptive & no-downtime switch over to TLS connections without additional work in 7.0.2. Consider below, [1] When we say, wherever possible, clients should terminate connections, are we expecting that customers would do rolling-restart of their running application instances in a way that some instances continue to serve operate over non-secure-connections while other instances slowly move over to secure connections? If yes, it is still a manual process (something needs a restart) while the non-disruptive-service-availability goal is achievable this way. [2] If customers do not follow the process above, when server closes connections after 1 hour, service disruption is expected. [3] If automatic switch over to TLS connections is expected, the SDKs need to be intelligent enough to detect 'enforce-tls' and gracefully switch over to TLS connections without requiring application restart. I believe this is not possible with 7.0.2. [4] While Couchbase services, like eventing, gsi, views must detect 'enforce-tls' and gracefully switch over to TLS connections. This requires additional work and can not be accommodated into current 7.0.2 timelines. The current design assumes that server closes connections and when we restart connections, we check for 'enforce-tls' and reconnect appropriately. Additionally, eventing functions need to be paused/resumed explicitly when 'enforce-tls' is enabled which we discussed earlier if you recall. [5] We found references to couchbase server restart requirement on enabling encryption on/off in n2n design specifications doc at https://docs.google.com/document/d/1gKtIi7aRK_TqlALT8PkQjUubrjR20hTeKiInzeF4iIM/edit# . Not sure, if this is still the case or we have removed the restriction along the way.
            dfinlay Dave Finlay added a comment -

            Hi Jeelan Poola:

            As I mentioned in my earlier comment:

            1. it must be the case that by some well-bounded interval after the encryption level is set to strict, there is no unencrypted communication happening over non-loopback interfaces in the cluster. This "well-bounded interval" can be as much as say an hour, but existing connections should not continue to survive potentially indefinitely.
            2. It is permissible that there is some disruption to front-end traffic, replication, intra-cluster traffic during this change, though obviously the less the better.

            This means that (at least for now) you can:
            (1) expect memcached (and any other server you connect to) to terminate your connections; you are not required to do it yourself, though you can if you want to
            (2) it's OK if service is interrupted during this time - running queries can fail, temporary unavailability is OK, etc

            On your questions:
            (1) Clients may well already have converted their clients to use TLS and so may not need any kind of rolling restart. But if they have not, then they will need to do something.
            (2) Yes - this needs to be documented as a consequence of moving to strict
            (3) This isn't required of the SDKs (at this point) and needs to be documented
            (4) As long as you can consult the configuration when servers terminate your connections you satisfy what PM is asking for in 7.0.2.

            On (5), there was some back and forth on this during the implementation of node-to-node encryption support. You'll note that the uncrossed out requirements say:

            But no restart would be needed when there is a change in cluster encryption level (between control and all).

            And what we're talking about here is a change in encryption level.

            But in any case, does it work for you in 7.0.2 if memcached terminates your connections, and you consult the TLS config before reconnecting – or does it not work for you and you need to insist on a full cluster restart?

            dfinlay Dave Finlay added a comment - Hi Jeelan Poola : As I mentioned in my earlier comment: it must be the case that by some well-bounded interval after the encryption level is set to strict, there is no unencrypted communication happening over non-loopback interfaces in the cluster. This "well-bounded interval" can be as much as say an hour, but existing connections should not continue to survive potentially indefinitely. It is permissible that there is some disruption to front-end traffic, replication, intra-cluster traffic during this change, though obviously the less the better. This means that (at least for now) you can: (1) expect memcached (and any other server you connect to) to terminate your connections; you are not required to do it yourself, though you can if you want to (2) it's OK if service is interrupted during this time - running queries can fail, temporary unavailability is OK, etc On your questions: (1) Clients may well already have converted their clients to use TLS and so may not need any kind of rolling restart. But if they have not, then they will need to do something. (2) Yes - this needs to be documented as a consequence of moving to strict (3) This isn't required of the SDKs (at this point) and needs to be documented (4) As long as you can consult the configuration when servers terminate your connections you satisfy what PM is asking for in 7.0.2. On (5), there was some back and forth on this during the implementation of node-to-node encryption support. You'll note that the uncrossed out requirements say: But no restart would be needed when there is a change in cluster encryption level (between control and all). And what we're talking about here is a change in encryption level. But in any case, does it work for you in 7.0.2 if memcached terminates your connections, and you consult the TLS config before reconnecting – or does it not work for you and you need to insist on a full cluster restart?

            I would like to add one think for consideration (probably pretty obvious): If we are talking about dropping tens of thousands of connections, the correct way to do it is to drop them not simultaneously but with some delay, so they don't reconnect back all simultaneously, basically to avoid spikes of reconnection attempts.

            timofey.barmin Timofey Barmin added a comment - I would like to add one think for consideration (probably pretty obvious): If we are talking about dropping tens of thousands of connections, the correct way to do it is to drop them not simultaneously but with some delay, so they don't reconnect back all simultaneously, basically to avoid spikes of reconnection attempts.
            jon.strabala Jon Strabala added a comment - - edited

            Jeelan Poola Discussed the concerns brought up in the Eventing scrum Ian McCloy 1) lack to time/resources to get this feature done correctly and tested in Eventing in the near term 2) Issues with cascading logic not working correctly on a pause/resume (which we have anyway on a rebalance out/in).  Rebalance doesn't seem to be viable across 100's of nodes just to toggle this feature on. 

            I know I am late to the party on this, WRT Eventing there are aths forward to make the target release viable WRT Eventing Dave Finlay or others I am free to discuss :

            1. Don't allow the enforce TLS toggle on a cluster has any deployed (or a transitioning "deploying" "bootstrapping" state etc.) Eventing Functions on the cluster's Eventing nodes.  This would allow security minded customers to adopt Eventing and might be an easy path forward.  Obviously this would be a non-impact to clusters that don't use Eventing.
              (a) Warn the customer they are running Eventing functions and allow a yes/no choice for a hard "pause" of all Eventing functions and leave it to the customer to resume.
              (b) Warn the customer they are running Eventing functions and allow a yes/no choice for a hard "pause" of all Eventing functions and then auto resume like (3) below but with a warning.
            2. Allow Eventing to always run in mixed non-TLS and TLS modes where new Function deployments use the default. But provide a UI warning to the users that non-TLS functions are deployed and need a pause/resume to be compliant with the desired configuration. Also provide a status of same to /api/v1 for CI/CD discovery of non-compliance. The downside of this is that it might prevent adoption of Evening by some security minded customers (and not meet this requirement).
            3. Blindly stop processing checkpoint and pause all Eventing functions then restart without concern for order or breaking any processing chains, data inconsistencies, or duplicate messages. This is similar to the comment "if service is interrupted during this time - running queries can fail" but for Eventing would be more correct ""if service is interrupted during this time - eventing processing can fail and lead to an inconsistent state in non idempotent processing."  
              (a) We just document it is best to pause Eventing to avoid inconsistencies due to the ordering of functions or potential duplicate processing
              (b) I assume in Eventing we check that all Eventing Functions that have URL bindings for cURL will enforce https or not be deployable. 
              (c) Is the Eventing team overthinking "No down time or zero impact"
              (d) Since Functions are distributed across all nodes running the Eventing service can we even do a node by node enforce TLS?  Even if it worked say we have 4 eventing nodes we might have to pause/resume the Function four times if we toggle enforce TLS one node at a time. 
            4. Can we have a enforce TLS flag for Evening only that is cluster wide to prestage this.  Or can an Eventing function be configured to in it's definition to use TLS only, would this help by providing a path forward prior to hitting (1) above.
            jon.strabala Jon Strabala added a comment - - edited Jeelan Poola  Discussed the concerns brought up in the Eventing scrum Ian McCloy  1) lack to time/resources to get this feature done correctly and tested in Eventing in the near term 2) Issues with cascading logic not working correctly on a pause/resume (which we have anyway on a rebalance out/in).  Rebalance doesn't seem to be viable across 100's of nodes just to toggle this feature on.  I know I am late to the party on this, WRT Eventing there are aths forward to make the target release viable WRT Eventing Dave Finlay  or others I am free to discuss : Don't allow the enforce TLS toggle on a cluster has any deployed (or a transitioning "deploying" "bootstrapping" state etc.) Eventing Functions on the cluster's Eventing nodes.  This would allow security minded customers to adopt Eventing and might be an easy path forward.  Obviously this would be a non-impact to clusters that don't use Eventing. (a) Warn the customer they are running Eventing functions and allow a yes/no choice for a hard "pause" of all Eventing functions and leave it to the customer to resume. (b) Warn the customer they are running Eventing functions and allow a yes/no choice for a hard "pause" of all Eventing functions and then auto resume like (3) below but with a warning. Allow Eventing to always run in mixed non-TLS and TLS modes where new Function deployments use the default. But provide a UI warning to the users that non-TLS functions are deployed and need a pause/resume to be compliant with the desired configuration. Also provide a status of same to /api/v1 for CI/CD discovery of non-compliance. The downside of this is that it might prevent adoption of Evening by some security minded customers (and not meet this requirement). Blindly stop processing checkpoint and pause all Eventing functions then restart without concern for order or breaking any processing chains, data inconsistencies, or duplicate messages. This is similar to the comment " if service is interrupted during this time - running queries can fail " but for Eventing would be more correct "" if service is interrupted during this time - eventing processing can fail and lead to an inconsistent state in non idempotent processing."    (a) We just document it is best to pause Eventing to avoid inconsistencies due to the ordering of functions or potential duplicate processing (b) I assume in Eventing we check that all Eventing Functions that have URL bindings for cURL will enforce https or not be deployable.  (c) Is the Eventing team overthinking "No down time or zero impact" (d) Since Functions are distributed across all nodes running the Eventing service can we even do a node by node enforce TLS?  Even if it worked say we have 4 eventing nodes we might have to pause/resume the Function four times if we toggle enforce TLS one node at a time.  Can we have a enforce TLS flag for Evening only that is cluster wide to prestage this.  Or can an Eventing function be configured to in it's definition to use TLS only, would this help by providing a path forward prior to hitting (1) above.
            ianmccloy Ian McCloy added a comment -

            Jon Strabala and I discussed what we can do to move forward in the short 7.0.x timeframe and I think the most straightforward path we agreed on would be to have customers who use Eventing go ahead and pause their functions before making any changes to TLS enforcement or IP Address family.  We can document this as a part of the steps to making changes with these settings.

            ianmccloy Ian McCloy added a comment - Jon Strabala  and I discussed what we can do to move forward in the short 7.0.x timeframe and I think the most straightforward path we agreed on would be to have customers who use Eventing go ahead and pause their functions before making any changes to TLS enforcement or IP Address family.  We can document this as a part of the steps to making changes with these settings.

            Sounds good.

            jon.strabala Jon Strabala added a comment - Sounds good.
            dfinlay Dave Finlay added a comment -

            Awesome, thanks guys.

            dfinlay Dave Finlay added a comment - Awesome, thanks guys.
            jeelan.poola Jeelan Poola added a comment - - edited

            Thank you Jon Strabala Ian McCloy!

            Reading through all the thoughts on this topic thus far, it appears, when we say 'enforce-tls', we are saying couchbase server would stop accepting new connections on all non-tls ports across all services with immediate effect whereas existing non-tls connections would continue to work till some well-bounded-interval. This well-bounded-interval would give the user sufficient time to take necessary actions to move over existing non-tls connections over to TLS in some controlled fashion, for e.g, in the case of eventing, it will be to pause & resume of the deployed functions as per user's preference in terms of order & choice.

            This sounds reasonable & acceptable. Thanks a lot everyone for chiming in!

            jeelan.poola Jeelan Poola added a comment - - edited Thank you Jon Strabala Ian McCloy ! Reading through all the thoughts on this topic thus far, it appears, when we say 'enforce-tls', we are saying couchbase server would stop accepting new connections on all non-tls ports across all services with immediate effect whereas existing non-tls connections would continue to work till some well-bounded-interval . This well-bounded-interval would give the user sufficient time to take necessary actions to move over existing non-tls connections over to TLS in some controlled fashion, for e.g, in the case of eventing, it will be to pause & resume of the deployed functions as per user's preference in terms of order & choice. This sounds reasonable & acceptable. Thanks a lot everyone for chiming in!

            Same thing applies for IPvX Family-Only feature as well.

            Just wanted to clarify that there is no need to pause Eventing handlers after changing Address Family.
            As the "required' / "optional' flags are passed by ns_server via command-line at the time it starts / restarts the services, if a user is switching over from ipv4 only to ipv6 only the services would have to be restarted regardless, either automatically by ns_server or the user is expected to restart.

            Hence, for eventing this is just like any other KillAndRespawn or restart of eventing-producer process where handlers will be read back from metakv, checkpoints re-read from metadata bucket and fresh connections to KV will be established using whatever TLS / non-TLS protocol received via cbauth callback at that point.

            For the above reasons, there is no need to pause handlers when address family is changed.

            abhishek.jindal Abhishek Jindal added a comment - Same thing applies for IPvX Family-Only feature as well. Just wanted to clarify that there is no need to pause Eventing handlers after changing Address Family. As the "required' / "optional' flags are passed by ns_server via command-line at the time it starts / restarts the services, if a user is switching over from ipv4 only to ipv6 only the services would have to be restarted regardless, either automatically by ns_server or the user is expected to restart. Hence, for eventing this is just like any other KillAndRespawn or restart of eventing-producer process where handlers will be read back from metakv, checkpoints re-read from metadata bucket and fresh connections to KV will be established using whatever TLS / non-TLS protocol received via cbauth callback at that point. For the above reasons, there is no need to pause handlers when address family is changed.
            dfinlay Dave Finlay added a comment -

            Abhishek Jindal: indeed that's true for almost all of the services (everything except memcached.)

            And just to confirm:

            either automatically by ns_server or the user is expected to restart.

            If the command line ever changes, ns_server will detect this and restart the appropriate service. The user doesn't need to do it.

            dfinlay Dave Finlay added a comment - Abhishek Jindal : indeed that's true for almost all of the services (everything except memcached.) And just to confirm: either automatically by ns_server or the user is expected to restart. If the command line ever changes, ns_server will detect this and restart the appropriate service. The user doesn't need to do it.

            When memcached detects that the parent port dies all (external) connections to that port gets disconnected. All connections from the same host is left connected.

            trond Trond Norbye added a comment - When memcached detects that the parent port dies all (external) connections to that port gets disconnected. All connections from the same host is left connected.

            Build couchbase-server-7.1.0-1188 contains kv_engine commit af43b5b with commit message:
            MB-47707: Shut down connections when the parent port dies

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1188 contains kv_engine commit af43b5b with commit message: MB-47707 : Shut down connections when the parent port dies
            sujay.gad Sujay Gad added a comment -

            Verified using 7.0.2 - 6646, 7.1.0 - 1190.

            sujay.gad Sujay Gad added a comment - Verified using 7.0.2 - 6646, 7.1.0 - 1190.

            People

              sujay.gad Sujay Gad
              sujay.gad Sujay Gad
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty