Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34971

Large memory consumption by memcached and erl (Windows)

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 6.5.0
    • 6.5.0
    • memcached
    • Windows 10, CB Enterprise Edition 6.5.0 build 3516
    • Untriaged
    • Unknown
    • KV-Engine MH 2nd Beta

    Description

      Summary: I'm running an internal build (3516) of EE, and I'm seeing a large amount of memory being taken up by memcached and erl. I'm running a similar setup to releases of 6.x, where I never saw this kind of spike. 

      I have not yet had time to try and attach a debugger, but if I'm able to and when I get to it, I will update this ticket.

      I've run cbcollect as requested when I brought this up in #engineering on Slack (Dave Rigby). However, the file is too big to attach (100mb limit), so I will try to figure out some other way to send this file (and update this ticket). UPDATE: Here's a WeTransfer link: https://we.tl/t-8dMptSTZU9 - this is only available for 7 days. I will keep the file on my local machine. If you need it after 7 days let me know and I will WeTransfer it again.

      Attachments

        1. cb settings.png
          cb settings.png
          29 kB
        2. cb task manager.png
          cb task manager.png
          27 kB
        3. image-2019-08-20-10-50-51-899.png
          image-2019-08-20-10-50-51-899.png
          163 kB
        4. image-2019-08-21-10-19-25-036.png
          image-2019-08-21-10-19-25-036.png
          106 kB
        5. screenshot-1.png
          screenshot-1.png
          232 kB
        6. screenshot-2.png
          screenshot-2.png
          126 kB
        7. screenshot-of-build-6.5.0-4130.png
          screenshot-of-build-6.5.0-4130.png
          1.15 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          matthew.groves Matthew Groves created issue -
          drigby Dave Rigby made changes -
          Field Original Value New Value
          Assignee Trond Norbye [ trond ] Daniel Owen [ owend ]
          matthew.groves Matthew Groves made changes -
          Description *Summary*: I'm running an internal build (3516) of EE, and I'm seeing a large amount of memory being taken up by memcached and erl. I'm running a similar setup to releases of 6.x, where I never saw this kind of spike. 

          I have not yet had time to try and attach a debugger, but if I'm able to and when I get to it, I will update this ticket.

          I've run cbcollect as requested when I brought this up in #engineering on Slack (Dave Rigby). However, the file is too big to attach (100mb limit), so I will try to figure out some other way to send this file (and update this ticket)
          *Summary*: I'm running an internal build (3516) of EE, and I'm seeing a large amount of memory being taken up by memcached and erl. I'm running a similar setup to releases of 6.x, where I never saw this kind of spike. 

          I have not yet had time to try and attach a debugger, but if I'm able to and when I get to it, I will update this ticket.

          I've run cbcollect as requested when I brought this up in #engineering on Slack (Dave Rigby). However, the file is too big to attach (100mb limit), so I will try to figure out some other way to send this file (and update this ticket). *UPDATE*: Here's a WeTransfer link: https://we.tl/t-8dMptSTZU9
          matthew.groves Matthew Groves made changes -
          Description *Summary*: I'm running an internal build (3516) of EE, and I'm seeing a large amount of memory being taken up by memcached and erl. I'm running a similar setup to releases of 6.x, where I never saw this kind of spike. 

          I have not yet had time to try and attach a debugger, but if I'm able to and when I get to it, I will update this ticket.

          I've run cbcollect as requested when I brought this up in #engineering on Slack (Dave Rigby). However, the file is too big to attach (100mb limit), so I will try to figure out some other way to send this file (and update this ticket). *UPDATE*: Here's a WeTransfer link: https://we.tl/t-8dMptSTZU9
          *Summary*: I'm running an internal build (3516) of EE, and I'm seeing a large amount of memory being taken up by memcached and erl. I'm running a similar setup to releases of 6.x, where I never saw this kind of spike. 

          I have not yet had time to try and attach a debugger, but if I'm able to and when I get to it, I will update this ticket.

          I've run cbcollect as requested when I brought this up in #engineering on Slack (Dave Rigby). However, the file is too big to attach (100mb limit), so I will try to figure out some other way to send this file (and update this ticket). *UPDATE*: Here's a WeTransfer link: [https://we.tl/t-8dMptSTZU9] - this is only available for 7 days. I will keep the file on my local machine. If you need it after 7 days let me know and I will WeTransfer it again.

          Thanks Dave Rigby I will check out a newer release when I get a chance. I'm not sure that ticket applies to my situation, but it's worth checking out anyway.

          I've got the collection running right now via the UI. (Assuming it finishes successfully, it will not match the cbcollect zip file that I sent via WeTransfer).

          Also note that this is not customer data; it's just my own CB server that I run for dogfooding purposes.

          matthew.groves Matthew Groves added a comment - Thanks Dave Rigby I will check out a newer release when I get a chance. I'm not sure that ticket applies to my situation, but it's worth checking out anyway. I've got the collection running right now via the UI. (Assuming it finishes successfully, it will not match the cbcollect zip file that I sent via WeTransfer). Also note that this is not customer data; it's just my own CB server that I run for dogfooding purposes.
          dfinlay Dave Finlay made changes -
          Fix Version/s Mad-Hatter [ 15037 ]

          Dave Rigby I am using build 3758 now and still seeing large increases in RAM usage by memcached, erl (and cbq-engine from time to time). I will try collecting/uploading logs again.

          matthew.groves Matthew Groves added a comment - Dave Rigby I am using build 3758 now and still seeing large increases in RAM usage by memcached, erl (and cbq-engine from time to time). I will try collecting/uploading logs again.

          I'm not sure if doing it via the web-based UI worked or not, but I think I was able to upload via the command line: https://s3.amazonaws.com/cb-engineering/Couchbase/34971/output_file.zip

          matthew.groves Matthew Groves added a comment - I'm not sure if doing it via the web-based UI worked or not, but I think I was able to upload via the command line:  https://s3.amazonaws.com/cb-engineering/Couchbase/34971/output_file.zip

          Hello, just checking in on this issue to see if there's anything else I can do to help. I'm on build 3758 now and memory usage still seems to be ballooning up over time.

          matthew.groves Matthew Groves added a comment - Hello, just checking in on this issue to see if there's anything else I can do to help. I'm on build 3758 now and memory usage still seems to be ballooning up over time.

          Dave Rigby environment could absolutely be the case–all the software I'm running is evergreen these days, so it would be hard for me to nail that down, but I have everything in Program Files excluded from virus scanning. I will try it on my new laptop and see if I'm seeing the same increase. If you strongly suspect this is an environmental issue, then absolutely treat it as low priority.

          My concern is that this is something specific to Windows, and I'm afraid (but maybe it's just my bias/perception) that Windows doesn't get the same level of attention as mac/linux. If I'm incorrect and there are others running these builds on Windows and not seeing the same issue, then there's no reason to worry

          matthew.groves Matthew Groves added a comment - Dave Rigby environment could absolutely be the case–all the software I'm running is evergreen these days, so it would be hard for me to nail that down, but I have everything in Program Files excluded from virus scanning. I will try it on my new laptop and see if I'm seeing the same increase. If you strongly suspect this is an environmental issue, then absolutely treat it as low priority. My concern is that this is something specific to Windows, and I'm afraid (but maybe it's just my bias/perception) that Windows doesn't get the same level of attention as mac/linux. If I'm incorrect and there are others running these builds on Windows and not seeing the same issue, then there's no reason to worry
          drigby Dave Rigby made changes -
          Sprint KV-Engine MH Beta Refresh [ 872 ]
          drigby Dave Rigby made changes -
          Rank Ranked lower
          drigby Dave Rigby made changes -
          Assignee Daniel Owen [ owend ]
          drigby Dave Rigby made changes -
          Assignee Richard deMellow [ richard.demellow ]
          richard.demellow Richard deMellow made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          richard.demellow Richard deMellow made changes -
          Attachment screenshot-1.png [ 72441 ]
          richard.demellow Richard deMellow added a comment - - edited

          With 6.5.0-3987 I'm also observing a steady memory growth over time that gets very large. In the image below you can see that memcached.exe is sitting at 10GB of allocated memory. The cluster is only has one node and one bucket of 100MB allocated too it.

          richard.demellow Richard deMellow added a comment - - edited With 6.5.0-3987 I'm also observing a steady memory growth over time that gets very large. In the image below you can see that memcached.exe is sitting at 10GB of allocated memory. The cluster is only has one node and one bucket of 100MB allocated too it.
          richard.demellow Richard deMellow made changes -
          Attachment image-2019-08-20-10-50-51-899.png [ 72442 ]
          richard.demellow Richard deMellow made changes -
          Attachment screenshot-2.png [ 72516 ]
          richard.demellow Richard deMellow added a comment - - edited

          Testing build 6.5.0-4050 I also see the same memory leak, again just using a bucket with only 100MB allocated to it (see bellow):

          I then created a toy build of had-hatter and disabled our use of jemalloc so that the build would use the system memory library. However, no memory leak was observed when giving memcached.exe the same workload on the same sized bucket.

          Having checked the jemalloc release notes they state that there has been a "critical virtual memory leak" on all versions of Windows since jemalloc version 5.0.0, we're currently using 5.1.0 in our master branch. Thus, I expect jemalloc is responsible for this leek and that to fix it we need to move to version 5.2.1.

          richard.demellow Richard deMellow added a comment - - edited Testing build 6.5.0-4050 I also see the same memory leak, again just using a bucket with only 100MB allocated to it (see bellow): I then created a toy build of had-hatter and disabled our use of jemalloc so that the build would use the system memory library. However, no memory leak was observed when giving memcached.exe the same workload on the same sized bucket. Having checked the jemalloc release notes they state that there has been a "critical virtual memory leak" on all versions of Windows since jemalloc version 5.0.0, we're currently using 5.1.0 in our master branch. Thus, I expect jemalloc is responsible for this leek and that to fix it we need to move to version 5.2.1.
          richard.demellow Richard deMellow made changes -
          Attachment image-2019-08-21-10-19-25-036.png [ 72517 ]

          Build couchbase-lite-core-2.6.0-2201 contains tlm commit cd04b23 with commit message:
          MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.6.0-2201 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          Build couchbase-lite-log-2.5.0-155 contains tlm commit cd04b23 with commit message:
          MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.5.0-155 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          Build couchbase-lite-log-2.6.0-162 contains tlm commit cd04b23 with commit message:
          MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.6.0-162 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          Build couchbase-lite-core-2.7.0-38 contains tlm commit cd04b23 with commit message:
          MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.7.0-38 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          Build libcouchbase-2.8.5-504 contains tlm commit cd04b23 with commit message:
          MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          build-team Couchbase Build Team added a comment - Build libcouchbase-2.8.5-504 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          Build couchbase-server-6.5.0-4080 contains tlm commit cd04b23 with commit message:
          MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4080 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          Build couchbase-lite-cblite-2.6.0-130 contains tlm commit cd04b23 with commit message:
          MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          build-team Couchbase Build Team added a comment - Build couchbase-lite-cblite-2.6.0-130 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          Build couchbase-lite-core-2.1.5-84 contains tlm commit cd04b23 with commit message:
          MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.1.5-84 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          Build couchbase-lite-core-2.5.3-15 contains tlm commit cd04b23 with commit message:
          MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.5.3-15 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

          Build couchbase-lite-log-2.5.0-156 contains tlm commit 7431e13 with commit message:
          MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.5.0-156 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          Build couchbase-lite-log-2.6.0-167 contains tlm commit 7431e13 with commit message:
          MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.6.0-167 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          Build couchbase-lite-core-2.6.0-2203 contains tlm commit 7431e13 with commit message:
          MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.6.0-2203 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          Build couchbase-server-6.5.0-4106 contains tlm commit 7431e13 with commit message:
          MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4106 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          Build couchbase-lite-core-2.7.0-43 contains tlm commit 7431e13 with commit message:
          MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.7.0-43 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          Build libcouchbase-2.8.5-506 contains tlm commit 7431e13 with commit message:
          MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          build-team Couchbase Build Team added a comment - Build libcouchbase-2.8.5-506 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          Build couchbase-lite-cblite-2.6.0-134 contains tlm commit 7431e13 with commit message:
          MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          build-team Couchbase Build Team added a comment - Build couchbase-lite-cblite-2.6.0-134 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          Build couchbase-lite-core-2.1.5-85 contains tlm commit 7431e13 with commit message:
          MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.1.5-85 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          Build couchbase-lite-core-2.5.3-16 contains tlm commit 7431e13 with commit message:
          MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.5.3-16 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

          Build couchbase-server-6.5.0-4130 contains tlm commit 2ef168a with commit message:
          MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4130 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          Marking resolved as jemalloc changes are now in master

          richard.demellow Richard deMellow added a comment - Marking resolved as jemalloc changes are now in master
          richard.demellow Richard deMellow made changes -
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]

          Build couchbase-lite-core-2.6.0-2205 contains tlm commit 2ef168a with commit message:
          MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.6.0-2205 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          Build libcouchbase-2.8.5-510 contains tlm commit 2ef168a with commit message:
          MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          build-team Couchbase Build Team added a comment - Build libcouchbase-2.8.5-510 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          Build couchbase-lite-core-2.7.0-46 contains tlm commit 2ef168a with commit message:
          MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.7.0-46 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1
          richard.demellow Richard deMellow made changes -
          Attachment screenshot-of-build-6.5.0-4130.png [ 72807 ]

          Marking closed as I've performed the steps of reproduction as stated before and as can be seen in the screenshot below there is no memory growth as seen build 6.5.0-4050. Insead we're sitting between 150MB to 165MB being used for the hole of memcached.exe and 35MB being used by the bucket.

          richard.demellow Richard deMellow added a comment - Marking closed as I've performed the steps of reproduction as stated before and as can be seen in the screenshot below there is no memory growth as seen build 6.5.0-4050. Insead we're sitting between 150MB to 165MB being used for the hole of memcached.exe and 35MB being used by the bucket.
          richard.demellow Richard deMellow made changes -
          VERIFICATION STEPS 1. install Couchbase Server EE on Windows
          2. Set up a one node cluster
          3. Load beer-sample bucket
          4. Use Process Explorer in admin mode to monitor memory usage of memcached.exe
          5. perform a pillow-fight against the beer-sample bucket

          Memory used during the pillow-fight shouldn't grow constantly and instead should stabilise.
          Status Resolved [ 5 ] Closed [ 6 ]

          Build couchbase-lite-log-2.5.0-158 contains tlm commit 2ef168a with commit message:
          MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.5.0-158 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          Build couchbase-lite-log-2.6.0-170 contains tlm commit 2ef168a with commit message:
          MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          build-team Couchbase Build Team added a comment - Build couchbase-lite-log-2.6.0-170 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          Build couchbase-lite-cblite-2.6.0-136 contains tlm commit 2ef168a with commit message:
          MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          build-team Couchbase Build Team added a comment - Build couchbase-lite-cblite-2.6.0-136 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          Build couchbase-lite-core-2.1.5-86 contains tlm commit 2ef168a with commit message:
          MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.1.5-86 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          Build couchbase-lite-core-2.5.3-17 contains tlm commit 2ef168a with commit message:
          MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          build-team Couchbase Build Team added a comment - Build couchbase-lite-core-2.5.3-17 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

          People

            richard.demellow Richard deMellow
            matthew.groves Matthew Groves
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty