Query service should fail to start if it cannot listen on all required ports

Summary

KV-Engine can fail to bind to IPv4 port 11210 (meaning clients cannot connect to it) but still be considered successfully running by ns_server if it managed to bind to IPv6 port 11210.

Details

When initialising its listening sockets KV-Engine currently[1] will consider an interface successfully setup if it can open a listening socket on at least one address for each requested protocol (IPv4 / IPv6) - see server_socket().

Only if none of the requested protocols could be opened does initialisation fail and the memcached process fail to start.

Because ns_server doesn't specify which of IPv4 / IPv6 an interface should listen on in the memcached.json config; memcached defaults to a value of true for both and will consider success if either IPv4 or IPv6 binds.

Given that most Linux distributions enable the IPv6 stack by default; if IPv4 port 11210 is not available (for example if a connection is already established on it) the the result is that while memcached fails to bind on IPv4 11210 it will nethertheless overall report success as long as it could bind the IPv6 11210 port.

[1]: In all versions prior to 5.5.0; and 5.5.1 onwards - see also MB-30610).

Workaround

By disabling the IPv6 stack at the OS level, memcached will not be able to bind any IPv6 ports; and as such will only report success from server_socket() if the IPv4 port could be bound.

Note there are two high-level methods to disable IPv6 on Linux:

1. Disable the entire IPv6 protocol stack - typically by adding a boot-time grub parameter: ipv6.disable=1
2. Disable assignment of IPv6 addresses to interfaces - either by setting the boot-time grub parameter: ipv6.disable_ipv6=1, or dynamically by setting the appropriate net.ipv6.conf sysctl properties.

Only method 1 avoids the problem - method two stops IPv6 traffic from bring routed; but does not prevent IPv6 listening sockets from being created.

Proposed Solution

1. ns_server should explicitly set in memcached.json config which if IPv4 / IPv6 it should listen on (by adding ipv4 / ipv6 keys to each configured interface.
   • If we only need to support single-stack deployments (IPv4 XOR IPv6) then that change will be sufficient - memcached will only be told to listen for a single protocol; and if that protocol fails then the whole setup will fail and it will terminate with a fatal error.
   • If we need to support dual-stack deployments - (IPv4 OR IPv6) then additional changes are needed in the memcached <-> ns_server interface, as memcached needs to be told what it should do in the event of one of the protocols not binding.

In the dual-stack case I would propose changing the ipv4 / ipv6 fields from booleans to a tri-state enum:

"ipv4" / "ipv6": "never", "optional", "always" 

KV-Engine would only attempt to bind on the given protocol if optional or always is specified; if it failed it would only be fatal if always had been specified. This moves the policy up to ns_server.