Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31651 Improve experience dealing with port conflicts
  3. MB-31105

Query service should fail to start if it cannot listen on all required ports

    XMLWordPrintable

    Details

      Description

      Summary

      KV-Engine can fail to bind to IPv4 port 11210 (meaning clients cannot connect to it) but still be considered successfully running by ns_server if it managed to bind to IPv6 port 11210.

      Details

      When initialising its listening sockets KV-Engine currently[1] will consider an interface successfully setup if it can open a listening socket on at least one address for each requested protocol (IPv4 / IPv6) - see server_socket().

      Only if none of the requested protocols could be opened does initialisation fail and the memcached process fail to start.

      Because ns_server doesn't specify which of IPv4 / IPv6 an interface should listen on in the memcached.json config; memcached defaults to a value of true for both and will consider success if either IPv4 or IPv6 binds.

      Given that most Linux distributions enable the IPv6 stack by default; if IPv4 port 11210 is not available (for example if a connection is already established on it) the the result is that while memcached fails to bind on IPv4 11210 it will nethertheless overall report success as long as it could bind the IPv6 11210 port.

      [1]: In all versions prior to 5.5.0; and 5.5.1 onwards - see also MB-30610).

      Workaround

      By disabling the IPv6 stack at the OS level, memcached will not be able to bind any IPv6 ports; and as such will only report success from server_socket() if the IPv4 port could be bound.

      Note there are two high-level methods to disable IPv6 on Linux:

      1. Disable the entire IPv6 protocol stack - typically by adding a boot-time grub parameter: ipv6.disable=1
      2. Disable assignment of IPv6 addresses to interfaces - either by setting the boot-time grub parameter: ipv6.disable_ipv6=1, or dynamically by setting the appropriate net.ipv6.conf sysctl properties.

      Only method 1 avoids the problem - method two stops IPv6 traffic from bring routed; but does not prevent IPv6 listening sockets from being created.

      Proposed Solution

      1. ns_server should explicitly set in memcached.json config which if IPv4 / IPv6 it should listen on (by adding ipv4 / ipv6 keys to each configured interface.
        • If we only need to support single-stack deployments (IPv4 XOR IPv6) then that change will be sufficient - memcached will only be told to listen for a single protocol; and if that protocol fails then the whole setup will fail and it will terminate with a fatal error.
        • If we need to support dual-stack deployments - (IPv4 OR IPv6) then additional changes are needed in the memcached <-> ns_server interface, as memcached needs to be told what it should do in the event of one of the protocols not binding.

      In the dual-stack case I would propose changing the ipv4 / ipv6 fields from booleans to a tri-state enum:

      "ipv4" / "ipv6": "never", "optional", "always" 
      

      KV-Engine would only attempt to bind on the given protocol if optional or always is specified; if it failed it would only be fatal if always had been specified. This moves the policy up to ns_server.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            till Till Westmann added a comment -

            Moving back to 6.6.0 for resolution.

            Show
            till Till Westmann added a comment - Moving back to 6.6.0 for resolution.
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.0-7853 contains query commit f6246ed with commit message:
            MB-31105 - Disable IPv4/IPv6 and http2 fixes for mad-hatter/windows

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7853 contains query commit f6246ed with commit message: MB-31105 - Disable IPv4/IPv6 and http2 fixes for mad-hatter/windows
            Hide
            mihir.kamdar Mihir Kamdar added a comment -

            Isha Kandaswamy Kamini Jagtiani are we good to close this ticket now as the windows issue is tracked separately. QE validation is completed.

            Show
            mihir.kamdar Mihir Kamdar added a comment - Isha Kandaswamy Kamini Jagtiani are we good to close this ticket now as the windows issue is tracked separately. QE validation is completed.
            Hide
            kamini.jagtiani Kamini Jagtiani added a comment -

            Mihir Kamdar yes, we can close it.

            Show
            kamini.jagtiani Kamini Jagtiani added a comment - Mihir Kamdar  yes, we can close it.
            Hide
            mihir.kamdar Mihir Kamdar added a comment -

            Closed as verified

            Show
            mihir.kamdar Mihir Kamdar added a comment - Closed as verified

              People

              Assignee:
              isha Isha Kandaswamy
              Reporter:
              perry Perry Krug
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  PagerDuty