[MB-34971] Large memory consumption by memcached and erl (Windows) - Couchbase database

Matthew Groves created issue - 10/Jul/19 7:28 AM

Dave Rigby made changes - 10/Jul/19 7:29 AM

Field	Original Value	New Value
Assignee	Trond Norbye [ trond ]	Daniel Owen [ owend ]

Matthew Groves made changes - 10/Jul/19 7:31 AM

Description

*Summary*: I'm running an internal build (3516) of EE, and I'm seeing a large amount of memory being taken up by memcached and erl. I'm running a similar setup to releases of 6.x, where I never saw this kind of spike.

I have not yet had time to try and attach a debugger, but if I'm able to and when I get to it, I will update this ticket.

I've run cbcollect as requested when I brought this up in #engineering on Slack (Dave Rigby). However, the file is too big to attach (100mb limit), so I will try to figure out some other way to send this file (and update this ticket)

*Summary*: I'm running an internal build (3516) of EE, and I'm seeing a large amount of memory being taken up by memcached and erl. I'm running a similar setup to releases of 6.x, where I never saw this kind of spike.

I have not yet had time to try and attach a debugger, but if I'm able to and when I get to it, I will update this ticket.

I've run cbcollect as requested when I brought this up in #engineering on Slack (Dave Rigby). However, the file is too big to attach (100mb limit), so I will try to figure out some other way to send this file (and update this ticket). *UPDATE*: Here's a WeTransfer link: https://we.tl/t-8dMptSTZU9

Matthew Groves made changes - 10/Jul/19 7:33 AM

Description

*Summary*: I'm running an internal build (3516) of EE, and I'm seeing a large amount of memory being taken up by memcached and erl. I'm running a similar setup to releases of 6.x, where I never saw this kind of spike.

I have not yet had time to try and attach a debugger, but if I'm able to and when I get to it, I will update this ticket.

I've run cbcollect as requested when I brought this up in #engineering on Slack (Dave Rigby). However, the file is too big to attach (100mb limit), so I will try to figure out some other way to send this file (and update this ticket). *UPDATE*: Here's a WeTransfer link: https://we.tl/t-8dMptSTZU9

*Summary*: I'm running an internal build (3516) of EE, and I'm seeing a large amount of memory being taken up by memcached and erl. I'm running a similar setup to releases of 6.x, where I never saw this kind of spike.

I have not yet had time to try and attach a debugger, but if I'm able to and when I get to it, I will update this ticket.

I've run cbcollect as requested when I brought this up in #engineering on Slack (Dave Rigby). However, the file is too big to attach (100mb limit), so I will try to figure out some other way to send this file (and update this ticket). *UPDATE*: Here's a WeTransfer link: [https://we.tl/t-8dMptSTZU9] - this is only available for 7 days. I will keep the file on my local machine. If you need it after 7 days let me know and I will WeTransfer it again.

Matthew Groves added a comment - 10/Jul/19 7:42 AM

Thanks Dave Rigby I will check out a newer release when I get a chance. I'm not sure that ticket applies to my situation, but it's worth checking out anyway.

I've got the collection running right now via the UI. (Assuming it finishes successfully, it will not match the cbcollect zip file that I sent via WeTransfer).

Also note that this is not customer data; it's just my own CB server that I run for dogfooding purposes.

Matthew Groves added a comment - 10/Jul/19 7:42 AM Thanks Dave Rigby I will check out a newer release when I get a chance. I'm not sure that ticket applies to my situation, but it's worth checking out anyway. I've got the collection running right now via the UI. (Assuming it finishes successfully, it will not match the cbcollect zip file that I sent via WeTransfer). Also note that this is not customer data; it's just my own CB server that I run for dogfooding purposes.

Dave Finlay made changes - 10/Jul/19 8:21 AM

Fix Version/s

Mad-Hatter [ 15037 ]

Matthew Groves added a comment - 15/Jul/19 6:20 AM

Dave Rigby I am using build 3758 now and still seeing large increases in RAM usage by memcached, erl (and cbq-engine from time to time). I will try collecting/uploading logs again.

Matthew Groves added a comment - 15/Jul/19 6:20 AM Dave Rigby I am using build 3758 now and still seeing large increases in RAM usage by memcached, erl (and cbq-engine from time to time). I will try collecting/uploading logs again.

Matthew Groves added a comment - 15/Jul/19 6:43 AM

I'm not sure if doing it via the web-based UI worked or not, but I think I was able to upload via the command line: https://s3.amazonaws.com/cb-engineering/Couchbase/34971/output_file.zip

Matthew Groves added a comment - 15/Jul/19 6:43 AM I'm not sure if doing it via the web-based UI worked or not, but I think I was able to upload via the command line: https://s3.amazonaws.com/cb-engineering/Couchbase/34971/output_file.zip

Matthew Groves added a comment - 30/Jul/19 10:53 AM

Hello, just checking in on this issue to see if there's anything else I can do to help. I'm on build 3758 now and memory usage still seems to be ballooning up over time.

Matthew Groves added a comment - 30/Jul/19 10:53 AM Hello, just checking in on this issue to see if there's anything else I can do to help. I'm on build 3758 now and memory usage still seems to be ballooning up over time.

Matthew Groves added a comment - 30/Jul/19 1:15 PM

Dave Rigby environment could absolutely be the case–all the software I'm running is evergreen these days, so it would be hard for me to nail that down, but I have everything in Program Files excluded from virus scanning. I will try it on my new laptop and see if I'm seeing the same increase. If you strongly suspect this is an environmental issue, then absolutely treat it as low priority.

My concern is that this is something specific to Windows, and I'm afraid (but maybe it's just my bias/perception) that Windows doesn't get the same level of attention as mac/linux. If I'm incorrect and there are others running these builds on Windows and not seeing the same issue, then there's no reason to worry

Matthew Groves added a comment - 30/Jul/19 1:15 PM Dave Rigby environment could absolutely be the case–all the software I'm running is evergreen these days, so it would be hard for me to nail that down, but I have everything in Program Files excluded from virus scanning. I will try it on my new laptop and see if I'm seeing the same increase. If you strongly suspect this is an environmental issue, then absolutely treat it as low priority. My concern is that this is something specific to Windows, and I'm afraid (but maybe it's just my bias/perception) that Windows doesn't get the same level of attention as mac/linux. If I'm incorrect and there are others running these builds on Windows and not seeing the same issue, then there's no reason to worry

Dave Rigby made changes - 12/Aug/19 6:48 AM

Sprint

KV-Engine MH Beta Refresh [ 872 ]

Dave Rigby made changes - 12/Aug/19 6:48 AM

Rank

Ranked lower

Dave Rigby made changes - 12/Aug/19 6:48 AM

Assignee

Daniel Owen [ owend ]

Dave Rigby made changes - 13/Aug/19 2:50 AM

Assignee

Richard deMellow [ richard.demellow ]

Richard deMellow made changes - 19/Aug/19 8:19 AM

Status

Open [ 1 ]

In Progress [ 3 ]

Richard deMellow made changes - 20/Aug/19 2:48 AM

Attachment

screenshot-1.png [ 72441 ]

Richard deMellow added a comment - 20/Aug/19 2:50 AM - edited

With 6.5.0-3987 I'm also observing a steady memory growth over time that gets very large. In the image below you can see that memcached.exe is sitting at 10GB of allocated memory. The cluster is only has one node and one bucket of 100MB allocated too it.

Richard deMellow added a comment - 20/Aug/19 2:50 AM - edited With 6.5.0-3987 I'm also observing a steady memory growth over time that gets very large. In the image below you can see that memcached.exe is sitting at 10GB of allocated memory. The cluster is only has one node and one bucket of 100MB allocated too it.

Richard deMellow made changes - 20/Aug/19 2:50 AM

Attachment

image-2019-08-20-10-50-51-899.png [ 72442 ]

Richard deMellow made changes - 21/Aug/19 1:59 AM

Attachment

screenshot-2.png [ 72516 ]

Richard deMellow added a comment - 21/Aug/19 2:19 AM - edited

Testing build 6.5.0-4050 I also see the same memory leak, again just using a bucket with only 100MB allocated to it (see bellow):

I then created a toy build of had-hatter and disabled our use of jemalloc so that the build would use the system memory library. However, no memory leak was observed when giving memcached.exe the same workload on the same sized bucket.

Having checked the jemalloc release notes they state that there has been a "critical virtual memory leak" on all versions of Windows since jemalloc version 5.0.0, we're currently using 5.1.0 in our master branch. Thus, I expect jemalloc is responsible for this leek and that to fix it we need to move to version 5.2.1.

Richard deMellow added a comment - 21/Aug/19 2:19 AM - edited Testing build 6.5.0-4050 I also see the same memory leak, again just using a bucket with only 100MB allocated to it (see bellow): I then created a toy build of had-hatter and disabled our use of jemalloc so that the build would use the system memory library. However, no memory leak was observed when giving memcached.exe the same workload on the same sized bucket. Having checked the jemalloc release notes they state that there has been a "critical virtual memory leak" on all versions of Windows since jemalloc version 5.0.0, we're currently using 5.1.0 in our master branch. Thus, I expect jemalloc is responsible for this leek and that to fix it we need to move to version 5.2.1.

Richard deMellow made changes - 21/Aug/19 2:19 AM

Attachment

image-2019-08-21-10-19-25-036.png [ 72517 ]

Couchbase Build Team added a comment - 21/Aug/19 7:08 AM

Build couchbase-lite-core-2.6.0-2201 contains tlm commit cd04b23 with commit message:
~~MB-34971~~ Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:08 AM Build couchbase-lite-core-2.6.0-2201 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:17 AM

Build couchbase-lite-log-2.5.0-155 contains tlm commit cd04b23 with commit message:
~~MB-34971~~ Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:17 AM Build couchbase-lite-log-2.5.0-155 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:27 AM

Build couchbase-lite-log-2.6.0-162 contains tlm commit cd04b23 with commit message:
~~MB-34971~~ Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:27 AM Build couchbase-lite-log-2.6.0-162 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:28 AM

Build couchbase-lite-core-2.7.0-38 contains tlm commit cd04b23 with commit message:
~~MB-34971~~ Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:28 AM Build couchbase-lite-core-2.7.0-38 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:38 AM

Build libcouchbase-2.8.5-504 contains tlm commit cd04b23 with commit message:
~~MB-34971~~ Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:38 AM Build libcouchbase-2.8.5-504 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:43 AM

Build couchbase-server-6.5.0-4080 contains tlm commit cd04b23 with commit message:
~~MB-34971~~ Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:43 AM Build couchbase-server-6.5.0-4080 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:48 AM

Build couchbase-lite-cblite-2.6.0-130 contains tlm commit cd04b23 with commit message:
~~MB-34971~~ Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 7:48 AM Build couchbase-lite-cblite-2.6.0-130 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 11:23 AM

Build couchbase-lite-core-2.1.5-84 contains tlm commit cd04b23 with commit message:
~~MB-34971~~ Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 11:23 AM Build couchbase-lite-core-2.1.5-84 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 11:33 AM

Build couchbase-lite-core-2.5.3-15 contains tlm commit cd04b23 with commit message:
~~MB-34971~~ Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 21/Aug/19 11:33 AM Build couchbase-lite-core-2.5.3-15 contains tlm commit cd04b23 with commit message: MB-34971 Update jemalloc package to 5.2.1 to fix windows memory leak

Couchbase Build Team added a comment - 23/Aug/19 7:30 AM

Build couchbase-lite-log-2.5.0-156 contains tlm commit 7431e13 with commit message:
~~MB-34971~~ Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 7:30 AM Build couchbase-lite-log-2.5.0-156 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 7:50 AM

Build couchbase-lite-log-2.6.0-167 contains tlm commit 7431e13 with commit message:
~~MB-34971~~ Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 7:50 AM Build couchbase-lite-log-2.6.0-167 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 7:56 AM

Build couchbase-lite-core-2.6.0-2203 contains tlm commit 7431e13 with commit message:
~~MB-34971~~ Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 7:56 AM Build couchbase-lite-core-2.6.0-2203 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 8:06 AM

Build couchbase-server-6.5.0-4106 contains tlm commit 7431e13 with commit message:
~~MB-34971~~ Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 8:06 AM Build couchbase-server-6.5.0-4106 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 8:10 AM

Build couchbase-lite-core-2.7.0-43 contains tlm commit 7431e13 with commit message:
~~MB-34971~~ Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 8:10 AM Build couchbase-lite-core-2.7.0-43 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 8:26 AM

Build libcouchbase-2.8.5-506 contains tlm commit 7431e13 with commit message:
~~MB-34971~~ Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 8:26 AM Build libcouchbase-2.8.5-506 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 8:28 AM

Build couchbase-lite-cblite-2.6.0-134 contains tlm commit 7431e13 with commit message:
~~MB-34971~~ Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 8:28 AM Build couchbase-lite-cblite-2.6.0-134 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 11:27 AM

Build couchbase-lite-core-2.1.5-85 contains tlm commit 7431e13 with commit message:
~~MB-34971~~ Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 11:27 AM Build couchbase-lite-core-2.1.5-85 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 11:38 AM

Build couchbase-lite-core-2.5.3-16 contains tlm commit 7431e13 with commit message:
~~MB-34971~~ Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 23/Aug/19 11:38 AM Build couchbase-lite-core-2.5.3-16 contains tlm commit 7431e13 with commit message: MB-34971 Update jemalloc_windows.bat to use msbuild.exe

Couchbase Build Team added a comment - 27/Aug/19 1:16 AM

Build couchbase-server-6.5.0-4130 contains tlm commit 2ef168a with commit message:
~~MB-34971~~ Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 1:16 AM Build couchbase-server-6.5.0-4130 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

Richard deMellow added a comment - 27/Aug/19 1:43 AM

Marking resolved as jemalloc changes are now in master

Richard deMellow added a comment - 27/Aug/19 1:43 AM Marking resolved as jemalloc changes are now in master

Richard deMellow made changes - 27/Aug/19 1:43 AM

Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Resolved [ 5 ]

Couchbase Build Team added a comment - 27/Aug/19 1:54 AM

Build couchbase-lite-core-2.6.0-2205 contains tlm commit 2ef168a with commit message:
~~MB-34971~~ Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 1:54 AM Build couchbase-lite-core-2.6.0-2205 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 1:55 AM

Build libcouchbase-2.8.5-510 contains tlm commit 2ef168a with commit message:
~~MB-34971~~ Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 1:55 AM Build libcouchbase-2.8.5-510 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 2:02 AM

Build couchbase-lite-core-2.7.0-46 contains tlm commit 2ef168a with commit message:
~~MB-34971~~ Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 2:02 AM Build couchbase-lite-core-2.7.0-46 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

Richard deMellow made changes - 27/Aug/19 2:25 AM

Attachment

screenshot-of-build-6.5.0-4130.png [ 72807 ]

Richard deMellow added a comment - 27/Aug/19 2:28 AM

Marking closed as I've performed the steps of reproduction as stated before and as can be seen in the screenshot below there is no memory growth as seen build 6.5.0-4050. Insead we're sitting between 150MB to 165MB being used for the hole of memcached.exe and 35MB being used by the bucket.

Richard deMellow added a comment - 27/Aug/19 2:28 AM Marking closed as I've performed the steps of reproduction as stated before and as can be seen in the screenshot below there is no memory growth as seen build 6.5.0-4050. Insead we're sitting between 150MB to 165MB being used for the hole of memcached.exe and 35MB being used by the bucket.

Richard deMellow made changes - 27/Aug/19 2:42 AM

VERIFICATION STEPS		1. install Couchbase Server EE on Windows 2. Set up a one node cluster 3. Load beer-sample bucket 4. Use Process Explorer in admin mode to monitor memory usage of memcached.exe 5. perform a pillow-fight against the beer-sample bucket Memory used during the pillow-fight shouldn't grow constantly and instead should stabilise.
Status	Resolved [ 5 ]	Closed [ 6 ]

Couchbase Build Team added a comment - 27/Aug/19 3:53 AM

Build couchbase-lite-log-2.5.0-158 contains tlm commit 2ef168a with commit message:
~~MB-34971~~ Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 3:53 AM Build couchbase-lite-log-2.5.0-158 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 4:35 AM

Build couchbase-lite-log-2.6.0-170 contains tlm commit 2ef168a with commit message:
~~MB-34971~~ Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 4:35 AM Build couchbase-lite-log-2.6.0-170 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 5:04 AM

Build couchbase-lite-cblite-2.6.0-136 contains tlm commit 2ef168a with commit message:
~~MB-34971~~ Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 5:04 AM Build couchbase-lite-cblite-2.6.0-136 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 11:30 AM

Build couchbase-lite-core-2.1.5-86 contains tlm commit 2ef168a with commit message:
~~MB-34971~~ Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 11:30 AM Build couchbase-lite-core-2.1.5-86 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 11:45 AM

Build couchbase-lite-core-2.5.3-17 contains tlm commit 2ef168a with commit message:
~~MB-34971~~ Build Couchbase-Server with jemalloc 5.2.1

Couchbase Build Team added a comment - 27/Aug/19 11:45 AM Build couchbase-lite-core-2.5.3-17 contains tlm commit 2ef168a with commit message: MB-34971 Build Couchbase-Server with jemalloc 5.2.1

Couchbase Server

Large memory consumption by memcached and erl (Windows)

Details

Description

Attachments

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty