Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32704

Replace per-port configuration of max connections with a single system setting

    XMLWordPrintable

Details

    • Task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 6.5.0
    • 6.5.0
    • memcached

    Description

      Problem

      Memcached currently have a “per port” setting of the maximum number of client which may be connected to the port. The motivation behind that was that we wanted to keep a pool of connections available to make sure that ns_server could connect to the system (via the 11209 port). Later on when we added support for SSL we didn’t have time to look at the overall model, we just copied the “per port” setting into the new SSL connection. This leads into the following “problem” with the current configuration:

      I can have 5000 connections to 11209 (plain, ipv4/6)
      I can have 30 000 connections to 11210 (plain, ipv4/6)
      I can have 30 000 connections to 11207 (SSL, ipv4/6)

      In a deployment which use a mix of SSL and plain clients one may use 60k clients connected to the system (30k of each type), but deployments which only use PLAIN connections may only use 30k connections (trying to use 30 001 would fail, even if we don’t use a single SSL connection).

      Solution

      The configuration file provided to memcached contains two new toplevel keys in the configuration:

      Key Description
      max_connections A number containing the maximum number of connections allowed to memcached (65k if we use the values in the example above)
      system_connections A number reserved for users authenticated as system users. (5k if we use the values in the example above)

      memcached will accept all sockets until we reach max_connections, at that time it'll close the connection immediately. As part of the command validation phase memcached determines if the connection should be disconnected by using the following logic:

       

      1. If the connection is represent a system-internal user, leave the connection alone and continue to execute the command.
      2. If we've exceeded the number of "user" connections (60k in the example above) and used more than half of the system connections the connection is disconnected if it is authenticated or the commad being executed isn't one of Hello, SaslListMech, SaslAuth or SaslStep.

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            trond Trond Norbye created issue -
            trond Trond Norbye made changes -
            Field Original Value New Value
            Description Problem:
            Memcached currently have a “per port” setting of the maximum number of client which may be connected to the port. The motivation behind that was that we wanted to keep a pool of connections available to make sure that ns_server could connect to the system (via the 11209 port). Later on when we added support for SSL we didn’t have time to look at the overall model, we just copied the “per port” setting into the new SSL connection. This leads into the following “problem” with the current configuration:

            I can have 5000 connections to 11209 (plain, ipv4/6)
            I can have 30 000 connections to 11210 (plain, ipv4/6)
            I can have 30 000 connections to 11207 (SSL, ipv4/6)

            In a deployment which use a mix of SSL and plain clients one may use 60k clients connected to the system (30k of each type), but deployments which only use PLAIN connections may only use 30k connections (trying to use 30 001 would fail, even if we don’t use a single SSL connection).
            Proposed solution:
            The configuration file provided to memcached contains two new configuration parameters:

            max_connections
            A number containing the maximum number of connections allowed to memcached (65k if we use the values in the example above)
            system_connections
            A number reserved for users authenticated as system users. (5k if we use the values in the example above).

            Memcached needs to be able to allow a few extra connections “temporarily” from they connect until they authenticate. An unauthenticated client trying to perform any operation except from: HELO, SASL MECH or SASL_AUTH/STEP will be disconnected once we reach the limit for unauthenticated connections.
            h1. Problem


             Memcached currently have a “per port” setting of the maximum number of client which may be connected to the port. The motivation behind that was that we wanted to keep a pool of connections available to make sure that ns_server could connect to the system (via the 11209 port). Later on when we added support for SSL we didn’t have time to look at the overall model, we just copied the “per port” setting into the new SSL connection. This leads into the following “problem” with the current configuration:

            I can have 5000 connections to 11209 (plain, ipv4/6)
             I can have 30 000 connections to 11210 (plain, ipv4/6)
             I can have 30 000 connections to 11207 (SSL, ipv4/6)

            In a deployment which use a mix of SSL and plain clients one may use 60k clients connected to the system (30k of each type), but deployments which only use PLAIN connections may only use 30k connections (trying to use 30 001 would fail, even if we don’t use a single SSL connection).
            h1. Solution


             The configuration file provided to memcached contains two new toplevel keys in the configuration:
            ||Key||Description||
            |max_connections|A number containing the maximum number of connections allowed to memcached (65k if we use the values in the example above)|
            |system_connections|A number reserved for users authenticated as system users. (5k if we use the values in the example above)|

            memcached will accept all sockets until we reach max_connections, at that time it'll close the connection immediately. As part of the command validation phase memcached determines if the connection should be disconnected by using the following logic:

             
             # If the connection is represent a system-internal user, leave the connection alone and continue to execute the command.
             # If we've exceeded the number of "user" connections (60k in the example above) and used more than half of the system connections the connection is disconnected if it is authenticated or the commad being executed isn't one of Hello, SaslListMech, SaslAuth or SaslStep.

             
            trond Trond Norbye made changes -
            Labels releasenote
            trond Trond Norbye made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            trond Trond Norbye made changes -
            Actual End 2019-01-24 06:53 (issue has been resolved)
            trond Trond Norbye made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            Build couchbase-server-6.5.0-2135 contains kv_engine commit 9850da3 with commit message:
            MB-32704: Remove per-port setting of max connections

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-2135 contains kv_engine commit 9850da3 with commit message: MB-32704 : Remove per-port setting of max connections
            drigby Dave Rigby made changes -
            Link This issue relates to MB-32791 [ MB-32791 ]

            People

              trond Trond Norbye
              trond Trond Norbye
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty