Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32061

Increased memory fragmentation and overhead

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • Yes

    Description

      Spdlog migration mistake caused us to allocate a buffer far bigger than we need (2GB).

      Original Description
      ====================================================================

      We have few memory-oriented tests like memory fragmentation, max .memcached RSS, memory overhead. Most of them show degradation  in Mad-Hatter:  http://showfast.sc.couchbase.com/#/timeline/Linux/kv/memory/all

      I wasn't able to identify the root cause or the code change of this issue yet.

      The issue was introduced somewhere between 6.5.0-1398 (good build) and 6.5.0-1405. Can't find exact build as we didn't have any centos enterprise builds in between. And this issue doesn't reproduce neither in CE not in Windows for some reason.

       

      Logs from "memory fragmentation" test on 1405.  

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-8930/172.23.133.11.zip

       

      Also comparison between 1398 and 1405:

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-1405_load_and_append_a3f4&label=6.5.0-1405&snapshot=ares_650-1398_load_and_append_b045&label=6.5.0-1398

       

       

       

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Hi Alex, my change to lower the spdlog buffer size is present in the latest Mad-Hatter build. Could you please rerun the test?

            I'd expect to see a lower memcached RSS.

            ben.huddleston Ben Huddleston added a comment - Hi Alex, my change to lower the spdlog buffer size is present in the latest Mad-Hatter build. Could you please rerun the test? I'd expect to see a lower memcached RSS.
            oleksandr.gyryk Alex Gyryk (Inactive) added a comment - - edited

            Tests are still running but I see that memory is looking good now:

            I'll leave ticket open until all tests are done

            oleksandr.gyryk Alex Gyryk (Inactive) added a comment - - edited Tests are still running but I see that memory is looking good now: I'll leave ticket open until all tests are done

            I'm leaving this ticket open and assigning it back to Ben Huddleston as looks like this change causes about 3-4% regression in KV throughput
            See https://issues.couchbase.com/browse/MB-32388 

            and https://issues.couchbase.com/browse/MB-32389

             

            oleksandr.gyryk Alex Gyryk (Inactive) added a comment - I'm leaving this ticket open and assigning it back to Ben Huddleston as looks like this change causes about 3-4% regression in KV throughput See https://issues.couchbase.com/browse/MB-32388   and https://issues.couchbase.com/browse/MB-32389  
            drigby Dave Rigby added a comment -

            Resolving this one as the memory issue has been addressed, and performance issues are being tracked via MB-32388 & MB-32389.

            drigby Dave Rigby added a comment - Resolving this one as the memory issue has been addressed, and performance issues are being tracked via MB-32388 & MB-32389 .
            wayne Wayne Siu added a comment -

            The issue has been addressed (see attached).

            wayne Wayne Siu added a comment - The issue has been addressed (see attached).

            People

              wayne Wayne Siu
              oleksandr.gyryk Alex Gyryk (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty