Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4738

compile ep-engine on windows with tcmalloc ( memory accoutning issues in windows when node is actively ejecting items when it reaches low watermark ( delta between mem_used and OS reported allocated memory up to 4 GB ))

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.6.5.4, 1.7.1, 1.7.1.1, 1.7.2, 1.8.0
    • Fix Version/s: 1.8.1, 2.0-beta
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Environment:
      windows 2008 64-bit
      15 GB ram
      8 B ram

      Description

      this happens because memcached in windows is not using tcmalloc implementation and memcached process is not releasing the memory which was allocated before.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        this happens when node has to eject items when it reaches the low and high water mark

        the mem_used reports 6 GB but OS reports that memcached.exe process is using 10GB .

        we dont have enough evidence that this happens during rebalancing but ejection and disk fetch increase the swap usage and memcached.exe memory usage.

        Show
        farshid Farshid Ghods (Inactive) added a comment - this happens when node has to eject items when it reaches the low and high water mark the mem_used reports 6 GB but OS reports that memcached.exe process is using 10GB . we dont have enough evidence that this happens during rebalancing but ejection and disk fetch increase the swap usage and memcached.exe memory usage.
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Thuan has put this report together by running mcsoda load against 3 node cluster where each node has 15 GB ram and resident ratio drops from 70->60%

        Show
        farshid Farshid Ghods (Inactive) added a comment - Thuan has put this report together by running mcsoda load against 3 node cluster where each node has 15 GB ram and resident ratio drops from 70->60%
        Hide
        chiyoung Chiyoung Seo added a comment -

        Our memory accounting stat "mem_used" is incremented or decremented within the constructor / deconstructor of Blob value class that are automatically invoked when memory is allocated or deallocated. Therefore, if "mem_used" stat shows 6GB while memcached resident memory is 10GB, this means that even if we explicitly release memory to the windows memory allocator, the allocator doesn't use these freed memory areas. This is mostly caused by memory fragmentation issues on the windows default allocator.

        Show
        chiyoung Chiyoung Seo added a comment - Our memory accounting stat "mem_used" is incremented or decremented within the constructor / deconstructor of Blob value class that are automatically invoked when memory is allocated or deallocated. Therefore, if "mem_used" stat shows 6GB while memcached resident memory is 10GB, this means that even if we explicitly release memory to the windows memory allocator, the allocator doesn't use these freed memory areas. This is mostly caused by memory fragmentation issues on the windows default allocator.
        Hide
        bcui Bin Cui added a comment -

        Integrate tcmalloc with memcached and ep-engine on cygwin/mingw build environment. It will dramatically reduce the memory fragementation issue found on windows.

        Show
        bcui Bin Cui added a comment - Integrate tcmalloc with memcached and ep-engine on cygwin/mingw build environment. It will dramatically reduce the memory fragementation issue found on windows.
        Hide
        steve Steve Yen added a comment -

        Adding this here, because I know I'll never be able to find this again in my webmail....

        > Trond,
        >
        > I didn't hear about this possibility before
        > Regardless , Is that something I can do during the runtime or we have
        > to recompile epengine with that option ?
        > If you have the binary compiled with that option I can run some tests
        > on it today to confirm standard malloc itself is sufficient
        >

        I believe all you need to do is to export two variables at runtime:
        ./ns_server/couchbase-server.sh.in:GLIBCPP_FORCE_NEW=1
        ./ns_server/couchbase-server.sh.in:export GLIBCPP_FORCE_NEW
        ./ns_server/couchbase-server.sh.in:GLIBCXX_FORCE_NEW=1
        ./ns_server/couchbase-server.sh.in:export GLIBCXX_FORCE_NEW

        Trond

        > Farshid
        >
        > On Feb 17, 2012, at 11:32 PM, Trond Norbye <trond.norbye@couchbase.com> wrote:
        >
        >> Did we ever try running the test with the standard malloc and using the FORCE_NEW flag? I've asked this question a number of times now, but I haven't gotten an answer yet. Replacing the standard memory allocator with tcmalloc didn't solve the problem itself, so it seemed that it root of the problem was the memory allocator inside the C++ layer.
        >>
        >> From the experiments we've had during the integration it didn't feel that the windows support was that mature and well tested. It feels a bit "risky business" to me to give a version to a customer where we change such a central component on platform with so little testing. I would personally sleep way better at night using Microsofts memory allocator if it solves the problem.
        >>
        >> just my 0.5 nok
        >>
        >> trond

        Show
        steve Steve Yen added a comment - Adding this here, because I know I'll never be able to find this again in my webmail.... > Trond, > > I didn't hear about this possibility before > Regardless , Is that something I can do during the runtime or we have > to recompile epengine with that option ? > If you have the binary compiled with that option I can run some tests > on it today to confirm standard malloc itself is sufficient > I believe all you need to do is to export two variables at runtime: ./ns_server/couchbase-server.sh.in:GLIBCPP_FORCE_NEW=1 ./ns_server/couchbase-server.sh.in:export GLIBCPP_FORCE_NEW ./ns_server/couchbase-server.sh.in:GLIBCXX_FORCE_NEW=1 ./ns_server/couchbase-server.sh.in:export GLIBCXX_FORCE_NEW Trond > Farshid > > On Feb 17, 2012, at 11:32 PM, Trond Norbye <trond.norbye@couchbase.com> wrote: > >> Did we ever try running the test with the standard malloc and using the FORCE_NEW flag? I've asked this question a number of times now, but I haven't gotten an answer yet. Replacing the standard memory allocator with tcmalloc didn't solve the problem itself, so it seemed that it root of the problem was the memory allocator inside the C++ layer. >> >> From the experiments we've had during the integration it didn't feel that the windows support was that mature and well tested. It feels a bit "risky business" to me to give a version to a customer where we change such a central component on platform with so little testing. I would personally sleep way better at night using Microsofts memory allocator if it solves the problem. >> >> just my 0.5 nok >> >> trond
        Hide
        dipti Dipti Borkar added a comment -

        Memory accounting is required on 1.8.1

        Show
        dipti Dipti Borkar added a comment - Memory accounting is required on 1.8.1
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #254 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/254/)
        Revert "MB-4738 Have tcmalloc support for windows" (Revision bb54125fc68628f29dc4ced156cdf3d8729ef2ec)

        Result = SUCCESS
        Bin Cui :
        Files :

        • win32/Makefile.mingw
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #254 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/254/ ) Revert " MB-4738 Have tcmalloc support for windows" (Revision bb54125fc68628f29dc4ced156cdf3d8729ef2ec) Result = SUCCESS Bin Cui : Files : win32/Makefile.mingw

          People

          • Assignee:
            bcui Bin Cui
            Reporter:
            farshid Farshid Ghods (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews