Critical Description
Summary
Memcached is unable to start in 5.5.0 if the IPv6 protocol stack has not been loaded on the machine:
2018-07-24T14:57:52.970425Z INFO Enable management port(s)
|
2018-07-24T14:57:52.970426Z INFO Enable user port(s)
|
2018-07-24T14:57:52.971117Z INFO 35 Listen on 0.0.0.0:11210
|
2018-07-24T14:57:52.971638Z CRITICAL Failed to create IPv6 port for "*:11210"
|
2018-07-24T14:57:52.972086Z INFO 36 Listen on 0.0.0.0:11209
|
2018-07-24T14:57:52.972554Z CRITICAL Failed to create IPv6 port for "*:11209"
|
2018-07-24T14:57:52.973039Z INFO 37 Listen on 0.0.0.0:11207
|
2018-07-24T14:57:52.973491Z CRITICAL Failed to create IPv6 port for "*:11207"
|
2018-07-24T14:57:52.973494Z CRITICAL Failed to create listening socket(s)
|
This is a regression from 5.1.x where memcached would start up fine.
Looking at the code, we check the values of the interfaces passed in memcached.json and see if each interface is ipv4 or ipv6.
Based on these values we then try to setup ipv4 and ipv6 sockets, if either of these fail then we terminate memcached.
However, I can't actually see anywhere that ns_server ever populates the ipv4 and ipv6 fields of the config, so memcached always defaults these to true, even if ipv6 is disabled at the kernel-level.
Reproduction
- Start up a centos6 machine (with Couchbase Server 5.5 installed)
- Create the file /etc/modprobe.d/ipv6.conf
- Add the line 'options ipv6 disable=1' to this file (seems to be a recommend way to disable ipv6 https://www.thegeekdiary.com/how-to-enable-ipv6-on-centos-rhel-6/)
- Reboot the machine
- Observe that memcached can no longer startup properly
Workaround
There are two methods to disable IPv6 on recent Linux distributions:
- Disable the entire IPv6 protocol stack - typically by adding a boot-time grub parameter: ipv6.disable=1
- Disable assignment of IPv6 addresses to interfaces - either by setting the boot-time grub parameter: ipv6.disable_ipv6=1, or dynamically by setting the appropriate net.ipv6.conf sysctl properties.
Only method 1 triggers this issue. If method 2 is used, then the problem does not occur.
Logs
https://s3.amazonaws.com/cb-engineering/MB-30610/ipv6.zip
Contents of memcached.json:
{
|
"admin": "@ns_server",
|
"audit_file": "/opt/couchbase/var/lib/couchbase/config/audit.json",
|
"breakpad": {
|
"enabled": true,
|
"minidump_dir": "/opt/couchbase/var/lib/couchbase/crash"
|
},
|
"client_cert_auth": {
|
"state": "disable",
|
"prefixes": []
|
},
|
"connection_idle_time": 0,
|
"datatype_snappy": true,
|
"dedupe_nmvb_maps": false,
|
"interfaces": [
|
{
|
"host": "*",
|
"port": 11210,
|
"maxconn": 30000
|
},
|
{
|
"host": "*",
|
"port": 11209,
|
"maxconn": 5000
|
},
|
{
|
"host": "*",
|
"port": 11207,
|
"maxconn": 30000,
|
"ssl": {
|
"key": "/opt/couchbase/var/lib/couchbase/config/memcached-key.pem",
|
"cert": "/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem"
|
}
|
}
|
],
|
"logger": {
|
"filename": "/opt/couchbase/var/lib/couchbase/logs/memcached.log",
|
"cyclesize": 10485760,
|
"sleeptime": 19
|
},
|
"privilege_debug": false,
|
"rbac_file": "/opt/couchbase/var/lib/couchbase/config/memcached.rbac",
|
"root": "/opt/couchbase",
|
"ssl_cipher_list": "HIGH",
|
"ssl_minimum_protocol": "tlsv1",
|
"tracing_enabled": true,
|
"verbosity": 0,
|
"xattr_enabled": true
|
}
|
