Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32061

Increased memory fragmentation and overhead

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • Yes

    Description

      Spdlog migration mistake caused us to allocate a buffer far bigger than we need (2GB).

      Original Description
      ====================================================================

      We have few memory-oriented tests like memory fragmentation, max .memcached RSS, memory overhead. Most of them show degradation  in Mad-Hatter:  http://showfast.sc.couchbase.com/#/timeline/Linux/kv/memory/all

      I wasn't able to identify the root cause or the code change of this issue yet.

      The issue was introduced somewhere between 6.5.0-1398 (good build) and 6.5.0-1405. Can't find exact build as we didn't have any centos enterprise builds in between. And this issue doesn't reproduce neither in CE not in Windows for some reason.

       

      Logs from "memory fragmentation" test on 1405.  

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-8930/172.23.133.11.zip

       

      Also comparison between 1398 and 1405:

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-1405_load_and_append_a3f4&label=6.5.0-1405&snapshot=ares_650-1398_load_and_append_b045&label=6.5.0-1398

       

       

       

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            oleksandr.gyryk Alex Gyryk (Inactive) created issue -
            drigby Dave Rigby made changes -
            Field Original Value New Value
            Assignee Trond Norbye [ trond ] Ben Huddleston [ ben.huddleston ]
            ben.huddleston Ben Huddleston made changes -
            ben.huddleston Ben Huddleston made changes -
            Description We have few memory-oriented tests like memory fragmentation, max .memcached RSS, memory overhead. Most of them show degradation  in Mad-Hatter:  [http://showfast.sc.couchbase.com/#/timeline/Linux/kv/memory/all]

            I wasn't able to identify the root cause or the code change of this issue yet.

            The issue was introduced somewhere between 6.5.0-1398 (good build) and 6.5.0-1405. Can't find exact build as we didn't have any centos enterprise builds in between. And this issue doesn't reproduce neither in CE not in Windows for some reason.

             

            Logs from "memory fragmentation" test on 1405.  

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-8930/172.23.133.11.zip]

             

            Also comparison between 1398 and 1405:

            [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-1405_load_and_append_a3f4&label=6.5.0-1405&snapshot=ares_650-1398_load_and_append_b045&label=6.5.0-1398]

            !image-2018-11-16-13-31-05-208.png!

             

             

             

             


            Original Description
            ====================================================================

            We have few memory-oriented tests like memory fragmentation, max .memcached RSS, memory overhead. Most of them show degradation  in Mad-Hatter:  [http://showfast.sc.couchbase.com/#/timeline/Linux/kv/memory/all]

            I wasn't able to identify the root cause or the code change of this issue yet.

            The issue was introduced somewhere between 6.5.0-1398 (good build) and 6.5.0-1405. Can't find exact build as we didn't have any centos enterprise builds in between. And this issue doesn't reproduce neither in CE not in Windows for some reason.

             

            Logs from "memory fragmentation" test on 1405.  

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-8930/172.23.133.11.zip]

             

            Also comparison between 1398 and 1405:

            [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-1405_load_and_append_a3f4&label=6.5.0-1405&snapshot=ares_650-1398_load_and_append_b045&label=6.5.0-1398]

            !image-2018-11-16-13-31-05-208.png!

             

             

             

             
            ben.huddleston Ben Huddleston made changes -
            Description

            Original Description
            ====================================================================

            We have few memory-oriented tests like memory fragmentation, max .memcached RSS, memory overhead. Most of them show degradation  in Mad-Hatter:  [http://showfast.sc.couchbase.com/#/timeline/Linux/kv/memory/all]

            I wasn't able to identify the root cause or the code change of this issue yet.

            The issue was introduced somewhere between 6.5.0-1398 (good build) and 6.5.0-1405. Can't find exact build as we didn't have any centos enterprise builds in between. And this issue doesn't reproduce neither in CE not in Windows for some reason.

             

            Logs from "memory fragmentation" test on 1405.  

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-8930/172.23.133.11.zip]

             

            Also comparison between 1398 and 1405:

            [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-1405_load_and_append_a3f4&label=6.5.0-1405&snapshot=ares_650-1398_load_and_append_b045&label=6.5.0-1398]

            !image-2018-11-16-13-31-05-208.png!

             

             

             

             
            Spdlog migration mistake caused us to allocate a buffer far bigger than we need (2GB).

            Original Description
            ====================================================================

            We have few memory-oriented tests like memory fragmentation, max .memcached RSS, memory overhead. Most of them show degradation  in Mad-Hatter:  [http://showfast.sc.couchbase.com/#/timeline/Linux/kv/memory/all]

            I wasn't able to identify the root cause or the code change of this issue yet.

            The issue was introduced somewhere between 6.5.0-1398 (good build) and 6.5.0-1405. Can't find exact build as we didn't have any centos enterprise builds in between. And this issue doesn't reproduce neither in CE not in Windows for some reason.

             

            Logs from "memory fragmentation" test on 1405.  

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-8930/172.23.133.11.zip]

             

            Also comparison between 1398 and 1405:

            [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_650-1405_load_and_append_a3f4&label=6.5.0-1405&snapshot=ares_650-1398_load_and_append_b045&label=6.5.0-1398]

            !image-2018-11-16-13-31-05-208.png!

             

             

             

             
            ben.huddleston Ben Huddleston made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Alex Gyryk [ oleksandr.gyryk ]
            oleksandr.gyryk Alex Gyryk (Inactive) made changes -
            oleksandr.gyryk Alex Gyryk (Inactive) made changes -
            Link This issue causes MB-32388 [ MB-32388 ]
            oleksandr.gyryk Alex Gyryk (Inactive) made changes -
            Link This issue relates to MB-32389 [ MB-32389 ]
            oleksandr.gyryk Alex Gyryk (Inactive) made changes -
            Assignee Alex Gyryk [ oleksandr.gyryk ] Ben Huddleston [ ben.huddleston ]
            drigby Dave Rigby made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Alex Gyryk [ oleksandr.gyryk ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            drigby Dave Rigby made changes -
            Actual End 2019-02-07 10:12 (issue has been resolved)
            ritam.sharma Ritam Sharma made changes -
            Assignee Alex Gyryk [ oleksandr.gyryk ] Wayne Siu [ wayne ]
            wayne Wayne Siu made changes -
            Attachment MB-32061.png [ 77394 ]
            wayne Wayne Siu made changes -
            Attachment MB-32061.png [ 77394 ]
            wayne Wayne Siu made changes -
            Attachment MB-32061.png [ 77395 ]
            wayne Wayne Siu made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              wayne Wayne Siu
              oleksandr.gyryk Alex Gyryk (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty