Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4896

moxi mem leak when using haproxy roundrobin load-balancing

    Details

      Description

      Can't find the exact bug for this (other bugs also mention moxi memory leak, and might be the same, but they don't mention haproxy).

      Reproduced this customer reported issue, and there's a quick config workaround that can slow the leak.

      More info:

      After spinning up a 20 node cluster, with haproxy, valgrind, and a special debug build of moxi, using a configuration similar to XXX's, I was able to reproduce a significant memory leak in moxi. It occurs during topology changes, or, when moxi thinks there's a cluster topology change. Other customers probably never noticed, since topology changes are usually infrequent.

      Additionally, XXX's use of haproxy, in roundrobin load-balancing configuration significantly exacerbated the bug/leak in moxi. (I recall Tim had another report of a moxi mem leak from another customer. Perhaps they're also using haproxy?)

      Here's XXX's haproxy configuration...

      -----------------
      global
      log 127.0.0.1 local2
      chroot /var/lib/haproxy
      pidfile /var/run/haproxy.pid
      maxconn 4000
      user haproxy
      group haproxy
      daemon

      defaults
      mode http
      log global
      option dontlognull
      option httpclose
      option httplog
      option forwardfor
      timeout connect 10000
      timeout client 300000
      timeout server 300000
      maxconn 60000
      retries 3
      stats enable
      stats uri /haproxy-status
      stats refresh 5s

      frontend moxi *:8092
      default_backend moxi

      backend moxi
      balance roundrobin
      server node1 10.80.68.152:8091 check
      server node2 10.80.68.178:8091 check
      server node3 10.80.68.146:8091 check
      server node4 10.80.68.166:8091 check
      server node5 10.80.68.154:8091 check
      server node6 10.80.68.158:8091 check
      server node7 10.80.68.156:8091 check
      server node8 10.80.68.160:8091 check
      server node9 10.80.68.162:8091 check
      server node10 10.80.68.144:8091 check
      server node11 10.80.68.170:8091 check
      server node12 10.80.68.174:8091 check
      server node13 10.80.68.164:8091 check
      server node14 10.80.68.168:8091 check
      server node15 10.80.68.150:8091 check
      server node16 10.80.68.148:8091 check
      server node17 10.80.68.176:8091 check
      server node18 10.80.68.172:8091 check
      -----------------

      The workaround to reduce the leak includes...

      = change from haproxy's 'balance roundrobin' to some other load balancing choice.

      For example, when I instead used 'balance source' instead of 'balance roundrobin' in my haproxy configuration, the leak went away. (Caveat: it went away until I did an actual real topology change.)

      The underlying issue is moxi's doing a simple string comparison to decide whether the topology has changed. And, every node in a cluster gives a slightly different answer as to the topology. When moxi thinks the topology has changed, moxi will tear-down its data structures and dynamically reconfigure, and there's a leak there somewhere.

      Normally, moxi, expects its HTTP/REST connection to be very long lived. However, when haproxy's in the middle, the haproxy might decide to timeout a HTTP connection that's still open but hasn't been doing anything. (e.g, the HTTP/REST connection hasn't been doing anything because there's no topology change). This leads to the second haproxy config workaround suggestion...

      = increase haproxy's timeouts

      XXX's currently using 5 minute timeouts (in millisecs)...

      timeout client 300000
      timeout server 300000

      So, every 5 minutes, haproxy times out the connection and closes it. moxi sees the closed HTTP/REST connection and tries again. haproxy will choose the next server node on its list (since haproxy is in 'balance roundrobin' configuration). That next server node will return a slightly different topology answer. Then moxi (because it's doing simple string comparison) will inadvertently think the topology configuration has changed (when it actually hasn't), exposing the leak.

      This was with haproxy 1.4.20.

        Activity

        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        We need to update our 1.7 and 1.8 release notes with this information

        Show
        farshid Farshid Ghods (Inactive) added a comment - We need to update our 1.7 and 1.8 release notes with this information
        Hide
        steve Steve Yen added a comment -

        Also, the valgrind memleak report was...

        ==23125== 280,896 bytes in 4 blocks are definitely lost in loss record 172 of 173
        ==23125== at 0x4C25A28: calloc (vg_replace_malloc.c:467)
        ==23125== by 0x424855: cproxy_copy_behaviors (cproxy_config.c:780)
        ==23125== by 0x41C326: cproxy_create_downstream (cproxy.c:1270)
        ==23125== by 0x41B22A: cproxy_add_downstream (cproxy.c:882)
        ==23125== by 0x41B304: cproxy_reserve_downstream (cproxy.c:908)
        ==23125== by 0x41DB61: cproxy_assign_downstream (cproxy.c:1784)
        ==23125== by 0x41EDBF: cproxy_pause_upstream_for_downstream (cproxy.c:2222)
        ==23125== by 0x43098A: cproxy_process_upstream_binary_nread (cproxy_protocol_b.c:188)
        ==23125== by 0x40911C: complete_nread (memcached.c:1990)
        ==23125== by 0x40F1E2: drive_machine (memcached.c:3576)
        ==23125== by 0x40FD4C: event_handler (memcached.c:3799)
        ==23125== by 0x4E3DDF8: event_base_loop (in /opt/couchbase/lib/libevent-2.0.so.5.1.0)

        Show
        steve Steve Yen added a comment - Also, the valgrind memleak report was... ==23125== 280,896 bytes in 4 blocks are definitely lost in loss record 172 of 173 ==23125== at 0x4C25A28: calloc (vg_replace_malloc.c:467) ==23125== by 0x424855: cproxy_copy_behaviors (cproxy_config.c:780) ==23125== by 0x41C326: cproxy_create_downstream (cproxy.c:1270) ==23125== by 0x41B22A: cproxy_add_downstream (cproxy.c:882) ==23125== by 0x41B304: cproxy_reserve_downstream (cproxy.c:908) ==23125== by 0x41DB61: cproxy_assign_downstream (cproxy.c:1784) ==23125== by 0x41EDBF: cproxy_pause_upstream_for_downstream (cproxy.c:2222) ==23125== by 0x43098A: cproxy_process_upstream_binary_nread (cproxy_protocol_b.c:188) ==23125== by 0x40911C: complete_nread (memcached.c:1990) ==23125== by 0x40F1E2: drive_machine (memcached.c:3576) ==23125== by 0x40FD4C: event_handler (memcached.c:3799) ==23125== by 0x4E3DDF8: event_base_loop (in /opt/couchbase/lib/libevent-2.0.so.5.1.0)
        Show
        steve Steve Yen added a comment - http://review.couchbase.org/#change,13939

          People

          • Assignee:
            steve Steve Yen
            Reporter:
            steve Steve Yen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes