Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8724

Moxi 1.8.1 leaks memory and crashes via the linux OOM killer

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.8.1
    • Fix Version/s: 2.5.0
    • Component/s: moxi
    • Security Level: Public
    • Labels:
    • Environment:
      CentOS 6.4 x86_64
      2GB Ram
      2 CPU
      50G Disk
    • Operating System:
      Centos 64-bit
    • Link to Log File, atop/blg, CBCollectInfo, Core dump:
      Hide
      lowmem_reserve[]: 0 0 0 0
      Node 0 DMA: 36*4kB 35*8kB 14*16kB 15*32kB 23*64kB 8*128kB 3*256kB 1*512kB 1*1024kB 2*2048kB 0*4096kB = 10024kB
      Node 0 DMA32: 465*4kB 416*8kB 314*16kB 218*32kB 135*64kB 155*128kB 88*256kB 43*512kB 20*1024kB 11*2048kB 31*4096kB = 260196kB
      1619 total pagecache pages
      498 pages in swap cache
      Swap cache stats: add 527164, delete 526666, find 110287/110624
      Free swap = 0kB
      Total swap = 2097144kB
      523263 pages RAM
      43644 pages reserved
      744 pages shared
      408139 pages non-shared
      [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
      [ 444] 0 444 2760 1 1 -17 -1000 udevd
      [ 1025] 0 1025 1538 2 0 0 0 portreserve
      [ 1032] 0 1032 247656 859 1 0 0 rsyslogd
      [ 1057] 0 1057 2704 24 0 0 0 irqbalance
      [ 1071] 32 1071 4743 16 0 0 0 rpcbind
      [ 1103] 29 1103 5835 2 0 0 0 rpc.statd
      [ 1122] 0 1122 2924 131 1 0 0 xe-daemon
      [ 1209] 81 1209 5896 38 0 0 0 dbus-daemon
      [ 1224] 0 1224 39584 11821 0 0 0 ruby
      [ 1239] 0 1239 1019 1 1 0 0 acpid
      [ 1285] 0 1285 21478 126 0 0 0 pele_admin
      [ 1294] 0 1294 50209 1325 0 0 0 snmpd
      [ 1306] 0 1306 16029 24 1 -17 -1000 sshd
      [ 1314] 0 1314 6532 43 0 0 0 ntpd
      [ 1394] 0 1394 12909 23 0 0 0 master
      [ 1396] 89 1396 13467 20 0 0 0 qmgr
      [ 1425] 0 1425 27544 2 1 0 0 abrtd
      [ 1449] 0 1449 27018 24 0 0 0 abrt-dump-oops
      [ 1492] 0 1492 29302 33 0 0 0 crond
      [ 1503] 0 1503 5363 5 0 0 0 atd
      [ 1525] 0 1525 5757 1 0 0 0 oddjobd
      [ 1567] 249 1567 9724 8 1 0 0 nrpe
      [ 1584] 0 1584 1014 2 1 0 0 mingetty
      [ 1586] 0 1586 1014 2 1 0 0 mingetty
      [ 1588] 0 1588 1014 2 1 0 0 mingetty
      [ 1590] 0 1590 2759 1 1 -17 -1000 udevd
      [ 1591] 0 1591 2759 1 1 -17 -1000 udevd
      [ 1592] 0 1592 1014 2 1 0 0 mingetty
      [ 1594] 0 1594 1014 2 1 0 0 mingetty
      [ 1596] 0 1596 1014 2 1 0 0 mingetty
      [ 1627] 0 1627 6909 31 1 -17 -1000 auditd
      [ 2039] 999 2039 950197 334195 0 0 0 moxi
      [ 5081] 0 5081 168943 397 1 0 0 peled
      [28593] 0 28593 54483 51 1 0 0 sssd
      [28594] 0 28594 62922 1803 0 0 0 sssd_be
      [28595] 0 28595 55260 134 0 0 0 sssd_nss
      [28596] 0 28596 52240 44 1 0 0 sssd_pam
      [11234] 89 11234 13449 141 0 0 0 pickup
      [15869] 0 15869 1017 18 1 0 0 sleep
      [15870] 0 15870 38220 103 1 0 0 crond
      [15871] 0 15871 2307 37 0 0 0 sh
      [15872] 0 15872 52895 19361 0 0 0 puppet
      [16213] 0 16213 96226 21403 1 0 0 python
      Out of memory: Kill process 2039 (moxi) score 853 or sacrifice child
      Killed process 2039, UID 999, (moxi) total-vm:3800788kB, anon-rss:1336424kB, file-rss:356kB
      netlink: 12 bytes leftover after parsing attributes.
      Show
      lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 36*4kB 35*8kB 14*16kB 15*32kB 23*64kB 8*128kB 3*256kB 1*512kB 1*1024kB 2*2048kB 0*4096kB = 10024kB Node 0 DMA32: 465*4kB 416*8kB 314*16kB 218*32kB 135*64kB 155*128kB 88*256kB 43*512kB 20*1024kB 11*2048kB 31*4096kB = 260196kB 1619 total pagecache pages 498 pages in swap cache Swap cache stats: add 527164, delete 526666, find 110287/110624 Free swap = 0kB Total swap = 2097144kB 523263 pages RAM 43644 pages reserved 744 pages shared 408139 pages non-shared [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 444] 0 444 2760 1 1 -17 -1000 udevd [ 1025] 0 1025 1538 2 0 0 0 portreserve [ 1032] 0 1032 247656 859 1 0 0 rsyslogd [ 1057] 0 1057 2704 24 0 0 0 irqbalance [ 1071] 32 1071 4743 16 0 0 0 rpcbind [ 1103] 29 1103 5835 2 0 0 0 rpc.statd [ 1122] 0 1122 2924 131 1 0 0 xe-daemon [ 1209] 81 1209 5896 38 0 0 0 dbus-daemon [ 1224] 0 1224 39584 11821 0 0 0 ruby [ 1239] 0 1239 1019 1 1 0 0 acpid [ 1285] 0 1285 21478 126 0 0 0 pele_admin [ 1294] 0 1294 50209 1325 0 0 0 snmpd [ 1306] 0 1306 16029 24 1 -17 -1000 sshd [ 1314] 0 1314 6532 43 0 0 0 ntpd [ 1394] 0 1394 12909 23 0 0 0 master [ 1396] 89 1396 13467 20 0 0 0 qmgr [ 1425] 0 1425 27544 2 1 0 0 abrtd [ 1449] 0 1449 27018 24 0 0 0 abrt-dump-oops [ 1492] 0 1492 29302 33 0 0 0 crond [ 1503] 0 1503 5363 5 0 0 0 atd [ 1525] 0 1525 5757 1 0 0 0 oddjobd [ 1567] 249 1567 9724 8 1 0 0 nrpe [ 1584] 0 1584 1014 2 1 0 0 mingetty [ 1586] 0 1586 1014 2 1 0 0 mingetty [ 1588] 0 1588 1014 2 1 0 0 mingetty [ 1590] 0 1590 2759 1 1 -17 -1000 udevd [ 1591] 0 1591 2759 1 1 -17 -1000 udevd [ 1592] 0 1592 1014 2 1 0 0 mingetty [ 1594] 0 1594 1014 2 1 0 0 mingetty [ 1596] 0 1596 1014 2 1 0 0 mingetty [ 1627] 0 1627 6909 31 1 -17 -1000 auditd [ 2039] 999 2039 950197 334195 0 0 0 moxi [ 5081] 0 5081 168943 397 1 0 0 peled [28593] 0 28593 54483 51 1 0 0 sssd [28594] 0 28594 62922 1803 0 0 0 sssd_be [28595] 0 28595 55260 134 0 0 0 sssd_nss [28596] 0 28596 52240 44 1 0 0 sssd_pam [11234] 89 11234 13449 141 0 0 0 pickup [15869] 0 15869 1017 18 1 0 0 sleep [15870] 0 15870 38220 103 1 0 0 crond [15871] 0 15871 2307 37 0 0 0 sh [15872] 0 15872 52895 19361 0 0 0 puppet [16213] 0 16213 96226 21403 1 0 0 python Out of memory: Kill process 2039 (moxi) score 853 or sacrifice child Killed process 2039, UID 999, (moxi) total-vm:3800788kB, anon-rss:1336424kB, file-rss:356kB netlink: 12 bytes leftover after parsing attributes.

      Description

      Moxi 1.8.1 leaks memory and crashes via the linux OOM killer. Using it to proxy classic memcache connections on each host (1:1 mapping, app to local moxi instance -> couchbase). I find that over a few days it gradually uses more ram until the OOM killer stops it.

      lowmem_reserve[]: 0 0 0 0
      Node 0 DMA: 36*4kB 35*8kB 14*16kB 15*32kB 23*64kB 8*128kB 3*256kB 1*512kB 1*1024kB 2*2048kB 0*4096kB = 10024kB
      Node 0 DMA32: 465*4kB 416*8kB 314*16kB 218*32kB 135*64kB 155*128kB 88*256kB 43*512kB 20*1024kB 11*2048kB 31*4096kB = 260196kB
      1619 total pagecache pages
      498 pages in swap cache
      Swap cache stats: add 527164, delete 526666, find 110287/110624
      Free swap = 0kB
      Total swap = 2097144kB
      523263 pages RAM
      43644 pages reserved
      744 pages shared
      408139 pages non-shared
      [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
      [ 444] 0 444 2760 1 1 -17 -1000 udevd
      [ 1025] 0 1025 1538 2 0 0 0 portreserve
      [ 1032] 0 1032 247656 859 1 0 0 rsyslogd
      [ 1057] 0 1057 2704 24 0 0 0 irqbalance
      [ 1071] 32 1071 4743 16 0 0 0 rpcbind
      [ 1103] 29 1103 5835 2 0 0 0 rpc.statd
      [ 1122] 0 1122 2924 131 1 0 0 xe-daemon
      [ 1209] 81 1209 5896 38 0 0 0 dbus-daemon
      [ 1224] 0 1224 39584 11821 0 0 0 ruby
      [ 1239] 0 1239 1019 1 1 0 0 acpid
      [ 1285] 0 1285 21478 126 0 0 0 pele_admin
      [ 1294] 0 1294 50209 1325 0 0 0 snmpd
      [ 1306] 0 1306 16029 24 1 -17 -1000 sshd
      [ 1314] 0 1314 6532 43 0 0 0 ntpd
      [ 1394] 0 1394 12909 23 0 0 0 master
      [ 1396] 89 1396 13467 20 0 0 0 qmgr
      [ 1425] 0 1425 27544 2 1 0 0 abrtd
      [ 1449] 0 1449 27018 24 0 0 0 abrt-dump-oops
      [ 1492] 0 1492 29302 33 0 0 0 crond
      [ 1503] 0 1503 5363 5 0 0 0 atd
      [ 1525] 0 1525 5757 1 0 0 0 oddjobd
      [ 1567] 249 1567 9724 8 1 0 0 nrpe
      [ 1584] 0 1584 1014 2 1 0 0 mingetty
      [ 1586] 0 1586 1014 2 1 0 0 mingetty
      [ 1588] 0 1588 1014 2 1 0 0 mingetty
      [ 1590] 0 1590 2759 1 1 -17 -1000 udevd
      [ 1591] 0 1591 2759 1 1 -17 -1000 udevd
      [ 1592] 0 1592 1014 2 1 0 0 mingetty
      [ 1594] 0 1594 1014 2 1 0 0 mingetty
      [ 1596] 0 1596 1014 2 1 0 0 mingetty
      [ 1627] 0 1627 6909 31 1 -17 -1000 auditd
      [ 2039] 999 2039 950197 334195 0 0 0 moxi
      [ 5081] 0 5081 168943 397 1 0 0 peled
      [28593] 0 28593 54483 51 1 0 0 sssd
      [28594] 0 28594 62922 1803 0 0 0 sssd_be
      [28595] 0 28595 55260 134 0 0 0 sssd_nss
      [28596] 0 28596 52240 44 1 0 0 sssd_pam
      [11234] 89 11234 13449 141 0 0 0 pickup
      [15869] 0 15869 1017 18 1 0 0 sleep
      [15870] 0 15870 38220 103 1 0 0 crond
      [15871] 0 15871 2307 37 0 0 0 sh
      [15872] 0 15872 52895 19361 0 0 0 puppet
      [16213] 0 16213 96226 21403 1 0 0 python
      Out of memory: Kill process 2039 (moxi) score 853 or sacrifice child
      Killed process 2039, UID 999, (moxi) total-vm:3800788kB, anon-rss:1336424kB, file-rss:356kB
      netlink: 12 bytes leftover after parsing attributes.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        steve Steve Yen added a comment -

        There was a memory leak in the libvbucket library which was fixed, so recent builds of moxi have that memory leak fix.

        Show
        steve Steve Yen added a comment - There was a memory leak in the libvbucket library which was fixed, so recent builds of moxi have that memory leak fix.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        Pavel,

        do you have performance tests that can verify this? we need your help to verify this.
        if you have instructions for this, pls send it to us. Thanks.

        Show
        maria Maria McDuff (Inactive) added a comment - Pavel, do you have performance tests that can verify this? we need your help to verify this. if you have instructions for this, pls send it to us. Thanks.
        Hide
        pavelpaulau Pavel Paulau added a comment -

        Sorry, I have no tests for moxi.

        Show
        pavelpaulau Pavel Paulau added a comment - Sorry, I have no tests for moxi.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        Tony, can you do a load using moxi 2.5?

        Show
        maria Maria McDuff (Inactive) added a comment - Tony, can you do a load using moxi 2.5?
        Hide
        thuan Thuan Nguyen added a comment -

        Maria, I need a vm in few day to verify this fix.

        Show
        thuan Thuan Nguyen added a comment - Maria, I need a vm in few day to verify this fix.

          People

          • Assignee:
            thuan Thuan Nguyen
            Reporter:
            bryan_seitz Bryan Seitz
          • Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes