Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30610

Memcached unable to start if IPv6 protocol stack is not loaded

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 5.5.0
    • 5.5.1
    • memcached
    • None
    • Untriaged
    • Yes

    Description

      Summary

      Memcached is unable to start in 5.5.0 if the IPv6 protocol stack has not been loaded on the machine:

      2018-07-24T14:57:52.970425Z INFO Enable management port(s)
      2018-07-24T14:57:52.970426Z INFO Enable user port(s)
      2018-07-24T14:57:52.971117Z INFO 35 Listen on 0.0.0.0:11210
      2018-07-24T14:57:52.971638Z CRITICAL Failed to create IPv6 port for "*:11210"
      2018-07-24T14:57:52.972086Z INFO 36 Listen on 0.0.0.0:11209
      2018-07-24T14:57:52.972554Z CRITICAL Failed to create IPv6 port for "*:11209"
      2018-07-24T14:57:52.973039Z INFO 37 Listen on 0.0.0.0:11207
      2018-07-24T14:57:52.973491Z CRITICAL Failed to create IPv6 port for "*:11207"
      2018-07-24T14:57:52.973494Z CRITICAL Failed to create listening socket(s)
      

      This is a regression from 5.1.x where memcached would start up fine.

      Looking at the code, we check the values of the interfaces passed in memcached.json and see if each interface is ipv4 or ipv6.
      Based on these values we then try to setup ipv4 and ipv6 sockets, if either of these fail then we terminate memcached.
      However, I can't actually see anywhere that ns_server ever populates the ipv4 and ipv6 fields of the config, so memcached always defaults these to true, even if ipv6 is disabled at the kernel-level.

      Reproduction

      1. Start up a centos6 machine (with Couchbase Server 5.5 installed)
      2. Create the file /etc/modprobe.d/ipv6.conf
      3. Add the line 'options ipv6 disable=1' to this file (seems to be a recommend way to disable ipv6 https://www.thegeekdiary.com/how-to-enable-ipv6-on-centos-rhel-6/)
      4. Reboot the machine
      5. Observe that memcached can no longer startup properly

      Workaround

      There are two methods to disable IPv6 on recent Linux distributions:

      1. Disable the entire IPv6 protocol stack - typically by adding a boot-time grub parameter: ipv6.disable=1
      2. Disable assignment of IPv6 addresses to interfaces - either by setting the boot-time grub parameter: ipv6.disable_ipv6=1, or dynamically by setting the appropriate net.ipv6.conf sysctl properties.

      Only method 1 triggers this issue. If method 2 is used, then the problem does not occur.

      Logs

      https://s3.amazonaws.com/cb-engineering/MB-30610/ipv6.zip

      Contents of memcached.json:

      {
        "admin": "@ns_server",
        "audit_file": "/opt/couchbase/var/lib/couchbase/config/audit.json",
        "breakpad": {
          "enabled": true,
          "minidump_dir": "/opt/couchbase/var/lib/couchbase/crash"
        },
        "client_cert_auth": {
          "state": "disable",
          "prefixes": []
        },
        "connection_idle_time": 0,
        "datatype_snappy": true,
        "dedupe_nmvb_maps": false,
        "interfaces": [
          {
            "host": "*",
            "port": 11210,
            "maxconn": 30000
          },
          {
            "host": "*",
            "port": 11209,
            "maxconn": 5000
          },
          {
            "host": "*",
            "port": 11207,
            "maxconn": 30000,
            "ssl": {
              "key": "/opt/couchbase/var/lib/couchbase/config/memcached-key.pem",
              "cert": "/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem"
            }
          }
        ],
        "logger": {
          "filename": "/opt/couchbase/var/lib/couchbase/logs/memcached.log",
          "cyclesize": 10485760,
          "sleeptime": 19
        },
        "privilege_debug": false,
        "rbac_file": "/opt/couchbase/var/lib/couchbase/config/memcached.rbac",
        "root": "/opt/couchbase",
        "ssl_cipher_list": "HIGH",
        "ssl_minimum_protocol": "tlsv1",
        "tracing_enabled": true,
        "verbosity": 0,
        "xattr_enabled": true
      }
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-30610
          # Subject Branch Project Status CR V

          Activity

            Build couchbase-server-5.5.2-3701 contains kv_engine commit 610e6d3 with commit message:
            MB-30610: Don't fail server_socket() if at least one protocol successful

            build-team Couchbase Build Team added a comment - Build couchbase-server-5.5.2-3701 contains kv_engine commit 610e6d3 with commit message: MB-30610 : Don't fail server_socket() if at least one protocol successful

            Verified on build 5.5.1-3507. 

            Steps to verify.

            1. Install 5.5.1-3507 on a centos 7.5 server
            2. Verify UI is up and able to create a bucket0
            3. Add a line options ipv6 disable=1 to /etc/modprobe.d/ipv6.conf
            4. Reboot centos 7 server
            5. Verify memcached starts up nomally and able to create new bucket1

            I will verify second method  of disable ipv6 via grub

            thuan Thuan Nguyen added a comment - Verified on build 5.5.1-3507.  Steps to verify. Install 5.5.1-3507 on a centos 7.5 server Verify UI is up and able to create a bucket0 Add a line options ipv6 disable=1 to /etc/modprobe.d/ipv6.conf Reboot centos 7 server Verify memcached starts up nomally and able to create new bucket1 I will verify second method  of disable ipv6 via grub
            thuan Thuan Nguyen added a comment -

            Verified on build 5.5.1-3507. 

            Steps to verify.

            1. Install 5.5.1-3507 on a centos 7.5 server
            2. Verify UI is up and able to create a bucket0
            3. Add ipv6.disable=1 to line  GRUB_CMDLINE_LINUX in /etc/default/grub
            4. Run 'grub2-mkconfig -o /boot/grub2/grub.cfg'
            5. Reboot centos 7 server
            6. Verify no ipv6 enable via command ifconfig -a | grep inet6
            7. Verify memcached starts up nomally and able to create new bucket1

             

            thuan Thuan Nguyen added a comment - Verified on build 5.5.1-3507.  Steps to verify. Install 5.5.1-3507 on a centos 7.5 server Verify UI is up and able to create a bucket0 Add ipv6.disable=1 to line  GRUB_CMDLINE_LINUX in /etc/default/grub Run 'grub2-mkconfig -o /boot/grub2/grub.cfg' Reboot centos 7 server Verify no ipv6 enable via command ifconfig -a | grep inet6 Verify memcached starts up nomally and able to create new bucket1  

            Build couchbase-server-6.5.0-1186 contains kv_engine commit 40f020d with commit message:
            Merge branch 'vulcan'

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1186 contains kv_engine commit 40f020d with commit message: Merge branch 'vulcan'

            Build couchbase-server-6.5.0-1186 contains kv_engine commit 610e6d3 with commit message:
            MB-30610: Don't fail server_socket() if at least one protocol successful

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1186 contains kv_engine commit 610e6d3 with commit message: MB-30610 : Don't fail server_socket() if at least one protocol successful

            People

              matt.carabine Matt Carabine
              matt.carabine Matt Carabine
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty