Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8724

Moxi 1.8.1 leaks memory and crashes via the linux OOM killer

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.5.0
    • 1.8.1
    • moxi
    • Security Level: Public
    • CentOS 6.4 x86_64
      2GB Ram
      2 CPU
      50G Disk
    • Centos 64-bit
    • Hide
      lowmem_reserve[]: 0 0 0 0
      Node 0 DMA: 36*4kB 35*8kB 14*16kB 15*32kB 23*64kB 8*128kB 3*256kB 1*512kB 1*1024kB 2*2048kB 0*4096kB = 10024kB
      Node 0 DMA32: 465*4kB 416*8kB 314*16kB 218*32kB 135*64kB 155*128kB 88*256kB 43*512kB 20*1024kB 11*2048kB 31*4096kB = 260196kB
      1619 total pagecache pages
      498 pages in swap cache
      Swap cache stats: add 527164, delete 526666, find 110287/110624
      Free swap = 0kB
      Total swap = 2097144kB
      523263 pages RAM
      43644 pages reserved
      744 pages shared
      408139 pages non-shared
      [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
      [ 444] 0 444 2760 1 1 -17 -1000 udevd
      [ 1025] 0 1025 1538 2 0 0 0 portreserve
      [ 1032] 0 1032 247656 859 1 0 0 rsyslogd
      [ 1057] 0 1057 2704 24 0 0 0 irqbalance
      [ 1071] 32 1071 4743 16 0 0 0 rpcbind
      [ 1103] 29 1103 5835 2 0 0 0 rpc.statd
      [ 1122] 0 1122 2924 131 1 0 0 xe-daemon
      [ 1209] 81 1209 5896 38 0 0 0 dbus-daemon
      [ 1224] 0 1224 39584 11821 0 0 0 ruby
      [ 1239] 0 1239 1019 1 1 0 0 acpid
      [ 1285] 0 1285 21478 126 0 0 0 pele_admin
      [ 1294] 0 1294 50209 1325 0 0 0 snmpd
      [ 1306] 0 1306 16029 24 1 -17 -1000 sshd
      [ 1314] 0 1314 6532 43 0 0 0 ntpd
      [ 1394] 0 1394 12909 23 0 0 0 master
      [ 1396] 89 1396 13467 20 0 0 0 qmgr
      [ 1425] 0 1425 27544 2 1 0 0 abrtd
      [ 1449] 0 1449 27018 24 0 0 0 abrt-dump-oops
      [ 1492] 0 1492 29302 33 0 0 0 crond
      [ 1503] 0 1503 5363 5 0 0 0 atd
      [ 1525] 0 1525 5757 1 0 0 0 oddjobd
      [ 1567] 249 1567 9724 8 1 0 0 nrpe
      [ 1584] 0 1584 1014 2 1 0 0 mingetty
      [ 1586] 0 1586 1014 2 1 0 0 mingetty
      [ 1588] 0 1588 1014 2 1 0 0 mingetty
      [ 1590] 0 1590 2759 1 1 -17 -1000 udevd
      [ 1591] 0 1591 2759 1 1 -17 -1000 udevd
      [ 1592] 0 1592 1014 2 1 0 0 mingetty
      [ 1594] 0 1594 1014 2 1 0 0 mingetty
      [ 1596] 0 1596 1014 2 1 0 0 mingetty
      [ 1627] 0 1627 6909 31 1 -17 -1000 auditd
      [ 2039] 999 2039 950197 334195 0 0 0 moxi
      [ 5081] 0 5081 168943 397 1 0 0 peled
      [28593] 0 28593 54483 51 1 0 0 sssd
      [28594] 0 28594 62922 1803 0 0 0 sssd_be
      [28595] 0 28595 55260 134 0 0 0 sssd_nss
      [28596] 0 28596 52240 44 1 0 0 sssd_pam
      [11234] 89 11234 13449 141 0 0 0 pickup
      [15869] 0 15869 1017 18 1 0 0 sleep
      [15870] 0 15870 38220 103 1 0 0 crond
      [15871] 0 15871 2307 37 0 0 0 sh
      [15872] 0 15872 52895 19361 0 0 0 puppet
      [16213] 0 16213 96226 21403 1 0 0 python
      Out of memory: Kill process 2039 (moxi) score 853 or sacrifice child
      Killed process 2039, UID 999, (moxi) total-vm:3800788kB, anon-rss:1336424kB, file-rss:356kB
      netlink: 12 bytes leftover after parsing attributes.
      Show
      lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 36*4kB 35*8kB 14*16kB 15*32kB 23*64kB 8*128kB 3*256kB 1*512kB 1*1024kB 2*2048kB 0*4096kB = 10024kB Node 0 DMA32: 465*4kB 416*8kB 314*16kB 218*32kB 135*64kB 155*128kB 88*256kB 43*512kB 20*1024kB 11*2048kB 31*4096kB = 260196kB 1619 total pagecache pages 498 pages in swap cache Swap cache stats: add 527164, delete 526666, find 110287/110624 Free swap = 0kB Total swap = 2097144kB 523263 pages RAM 43644 pages reserved 744 pages shared 408139 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 444] 0 444 2760 1 1 -17 -1000 udevd [ 1025] 0 1025 1538 2 0 0 0 portreserve [ 1032] 0 1032 247656 859 1 0 0 rsyslogd [ 1057] 0 1057 2704 24 0 0 0 irqbalance [ 1071] 32 1071 4743 16 0 0 0 rpcbind [ 1103] 29 1103 5835 2 0 0 0 rpc.statd [ 1122] 0 1122 2924 131 1 0 0 xe-daemon [ 1209] 81 1209 5896 38 0 0 0 dbus-daemon [ 1224] 0 1224 39584 11821 0 0 0 ruby [ 1239] 0 1239 1019 1 1 0 0 acpid [ 1285] 0 1285 21478 126 0 0 0 pele_admin [ 1294] 0 1294 50209 1325 0 0 0 snmpd [ 1306] 0 1306 16029 24 1 -17 -1000 sshd [ 1314] 0 1314 6532 43 0 0 0 ntpd [ 1394] 0 1394 12909 23 0 0 0 master [ 1396] 89 1396 13467 20 0 0 0 qmgr [ 1425] 0 1425 27544 2 1 0 0 abrtd [ 1449] 0 1449 27018 24 0 0 0 abrt-dump-oops [ 1492] 0 1492 29302 33 0 0 0 crond [ 1503] 0 1503 5363 5 0 0 0 atd [ 1525] 0 1525 5757 1 0 0 0 oddjobd [ 1567] 249 1567 9724 8 1 0 0 nrpe [ 1584] 0 1584 1014 2 1 0 0 mingetty [ 1586] 0 1586 1014 2 1 0 0 mingetty [ 1588] 0 1588 1014 2 1 0 0 mingetty [ 1590] 0 1590 2759 1 1 -17 -1000 udevd [ 1591] 0 1591 2759 1 1 -17 -1000 udevd [ 1592] 0 1592 1014 2 1 0 0 mingetty [ 1594] 0 1594 1014 2 1 0 0 mingetty [ 1596] 0 1596 1014 2 1 0 0 mingetty [ 1627] 0 1627 6909 31 1 -17 -1000 auditd [ 2039] 999 2039 950197 334195 0 0 0 moxi [ 5081] 0 5081 168943 397 1 0 0 peled [28593] 0 28593 54483 51 1 0 0 sssd [28594] 0 28594 62922 1803 0 0 0 sssd_be [28595] 0 28595 55260 134 0 0 0 sssd_nss [28596] 0 28596 52240 44 1 0 0 sssd_pam [11234] 89 11234 13449 141 0 0 0 pickup [15869] 0 15869 1017 18 1 0 0 sleep [15870] 0 15870 38220 103 1 0 0 crond [15871] 0 15871 2307 37 0 0 0 sh [15872] 0 15872 52895 19361 0 0 0 puppet [16213] 0 16213 96226 21403 1 0 0 python Out of memory: Kill process 2039 (moxi) score 853 or sacrifice child Killed process 2039, UID 999, (moxi) total-vm:3800788kB, anon-rss:1336424kB, file-rss:356kB netlink: 12 bytes leftover after parsing attributes.

    Description

      Moxi 1.8.1 leaks memory and crashes via the linux OOM killer. Using it to proxy classic memcache connections on each host (1:1 mapping, app to local moxi instance -> couchbase). I find that over a few days it gradually uses more ram until the OOM killer stops it.

      lowmem_reserve[]: 0 0 0 0
      Node 0 DMA: 36*4kB 35*8kB 14*16kB 15*32kB 23*64kB 8*128kB 3*256kB 1*512kB 1*1024kB 2*2048kB 0*4096kB = 10024kB
      Node 0 DMA32: 465*4kB 416*8kB 314*16kB 218*32kB 135*64kB 155*128kB 88*256kB 43*512kB 20*1024kB 11*2048kB 31*4096kB = 260196kB
      1619 total pagecache pages
      498 pages in swap cache
      Swap cache stats: add 527164, delete 526666, find 110287/110624
      Free swap = 0kB
      Total swap = 2097144kB
      523263 pages RAM
      43644 pages reserved
      744 pages shared
      408139 pages non-shared
      [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
      [ 444] 0 444 2760 1 1 -17 -1000 udevd
      [ 1025] 0 1025 1538 2 0 0 0 portreserve
      [ 1032] 0 1032 247656 859 1 0 0 rsyslogd
      [ 1057] 0 1057 2704 24 0 0 0 irqbalance
      [ 1071] 32 1071 4743 16 0 0 0 rpcbind
      [ 1103] 29 1103 5835 2 0 0 0 rpc.statd
      [ 1122] 0 1122 2924 131 1 0 0 xe-daemon
      [ 1209] 81 1209 5896 38 0 0 0 dbus-daemon
      [ 1224] 0 1224 39584 11821 0 0 0 ruby
      [ 1239] 0 1239 1019 1 1 0 0 acpid
      [ 1285] 0 1285 21478 126 0 0 0 pele_admin
      [ 1294] 0 1294 50209 1325 0 0 0 snmpd
      [ 1306] 0 1306 16029 24 1 -17 -1000 sshd
      [ 1314] 0 1314 6532 43 0 0 0 ntpd
      [ 1394] 0 1394 12909 23 0 0 0 master
      [ 1396] 89 1396 13467 20 0 0 0 qmgr
      [ 1425] 0 1425 27544 2 1 0 0 abrtd
      [ 1449] 0 1449 27018 24 0 0 0 abrt-dump-oops
      [ 1492] 0 1492 29302 33 0 0 0 crond
      [ 1503] 0 1503 5363 5 0 0 0 atd
      [ 1525] 0 1525 5757 1 0 0 0 oddjobd
      [ 1567] 249 1567 9724 8 1 0 0 nrpe
      [ 1584] 0 1584 1014 2 1 0 0 mingetty
      [ 1586] 0 1586 1014 2 1 0 0 mingetty
      [ 1588] 0 1588 1014 2 1 0 0 mingetty
      [ 1590] 0 1590 2759 1 1 -17 -1000 udevd
      [ 1591] 0 1591 2759 1 1 -17 -1000 udevd
      [ 1592] 0 1592 1014 2 1 0 0 mingetty
      [ 1594] 0 1594 1014 2 1 0 0 mingetty
      [ 1596] 0 1596 1014 2 1 0 0 mingetty
      [ 1627] 0 1627 6909 31 1 -17 -1000 auditd
      [ 2039] 999 2039 950197 334195 0 0 0 moxi
      [ 5081] 0 5081 168943 397 1 0 0 peled
      [28593] 0 28593 54483 51 1 0 0 sssd
      [28594] 0 28594 62922 1803 0 0 0 sssd_be
      [28595] 0 28595 55260 134 0 0 0 sssd_nss
      [28596] 0 28596 52240 44 1 0 0 sssd_pam
      [11234] 89 11234 13449 141 0 0 0 pickup
      [15869] 0 15869 1017 18 1 0 0 sleep
      [15870] 0 15870 38220 103 1 0 0 crond
      [15871] 0 15871 2307 37 0 0 0 sh
      [15872] 0 15872 52895 19361 0 0 0 puppet
      [16213] 0 16213 96226 21403 1 0 0 python
      Out of memory: Kill process 2039 (moxi) score 853 or sacrifice child
      Killed process 2039, UID 999, (moxi) total-vm:3800788kB, anon-rss:1336424kB, file-rss:356kB
      netlink: 12 bytes leftover after parsing attributes.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            thuan Thuan Nguyen
            bryan_seitz Bryan Seitz
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty