Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38012

1% DGM Test: Write ops/s dropped from 230k/s to 400/s due to swapping

    XMLWordPrintable

Details

    • Untriaged
    • Unknown
    • Magma: Jan 20 - Feb 2

    Description

      A 2 node cluster running 1% DGM test abruptly dropped ops rate from 240k writes/sec to 400/s when the active resident is at 27%. The test was in the initial loading phase of 12B docs (loaded 460M). Memory used by magma is around 3G per node.

       Logs:
      https://uploads.couchbase.com/magma/collectinfo-2020-02-21T092448-ns_1%40172.23.97.25.zip
      https://uploads.couchbase.com/magma/collectinfo-2020-02-21T092448-ns_1%40172.23.97.26.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            sarath Sarath Lakshman created issue -
            sarath Sarath Lakshman made changes -
            Field Original Value New Value
            Description A 2 node cluster running 1% DGM test abruptly dropped ops rate from 240k writes/sec to 400/s when the active resident is at 27%. The test was in the initial loading phase of 12B docs (loaded 460M). Memory used by magma is around 3G per node.

             

             
            A 2 node cluster running 1% DGM test abruptly dropped ops rate from 240k writes/sec to 400/s when the active resident is at 27%. The test was in the initial loading phase of 12B docs (loaded 460M). Memory used by magma is around 3G per node.

             Logs:
            [https://uploads.couchbase.com/magma/collectinfo-2020-02-21T092448-ns_1%40172.23.97.25.zip]
            [https://uploads.couchbase.com/magma/collectinfo-2020-02-21T092448-ns_1%40172.23.97.26.zip]
            ben.huddleston Ben Huddleston made changes -
            Attachment gdb.txt [ 84936 ]
            owend Daniel Owen made changes -
            Epic Link MB-30659 [ 88207 ]
            owend Daniel Owen made changes -
            Sprint KV Sprint 2020-February [ 979 ]
            ben.huddleston Ben Huddleston made changes -
            ben.huddleston Ben Huddleston made changes -
            ben.huddleston Ben Huddleston made changes -
            ben.huddleston Ben Huddleston made changes -
            ben.huddleston Ben Huddleston made changes -
            ben.huddleston Ben Huddleston made changes -
            drigby Dave Rigby made changes -
            Attachment image.png [ 85115 ]
            drigby Dave Rigby made changes -
            Attachment image.png [ 85115 ]
            ben.huddleston Ben Huddleston made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Sarath Lakshman [ sarath ]
            sarath Sarath Lakshman made changes -
            Link This issue relates to MB-38079 [ MB-38079 ]
            sarath Sarath Lakshman made changes -
            Component/s couchbase-bucket [ 10173 ]
            wayne Wayne Siu made changes -
            Labels magma magma-dp affects-cc-testing magma magma-dp
            wayne Wayne Siu made changes -
            Labels affects-cc-testing magma magma-dp magma magma-dp
            sarath Sarath Lakshman made changes -
            Epic Link MB-30659 [ 88207 ] CBSS-305 [ 112304 ]
            sarath Sarath Lakshman made changes -
            Sprint KV Sprint 2020-February [ 979 ] Magma: Jan 20 - Feb 2 [ 968 ]
            sarath Sarath Lakshman made changes -
            Rank Ranked higher
            sarath Sarath Lakshman made changes -
            Summary Write ops/s dropped from 230k/s to 400/s with tmp_oom errors even when there is sufficient memory 1% DGM Test: Write ops/s dropped from 230k/s to 400/s due to swapping
            srinath.duvuru Srinath Duvuru made changes -
            Sprint Magma: Jan 20 - Feb 2 [ 968 ] Magma: Jan 20 - Feb 2, ForestDB: Oct 26 - Nov 14 [ 968, 999 ]
            srinath.duvuru Srinath Duvuru made changes -
            Sprint Magma: Jan 20 - Feb 2, ForestDB: Oct 26 - Nov 14 [ 968, 999 ] Magma: Jan 20 - Feb 2 [ 968 ]
            sarath Sarath Lakshman made changes -
            Sprint Magma: Jan 20 - Feb 2 [ 968 ] Magma: Jan 20 - Feb 2, Magma: Jan 20 - Feb 3 [ 968, 1000 ]
            sarath Sarath Lakshman made changes -
            Assignee Sarath Lakshman [ sarath ] Ben Huddleston [ ben.huddleston ]
            sarath Sarath Lakshman made changes -
            Component/s couchbase-bucket [ 10173 ]
            sarath Sarath Lakshman made changes -
            Link This issue relates to MB-36370 [ MB-36370 ]
            ben.huddleston Ben Huddleston made changes -
            ben.huddleston Ben Huddleston made changes -
            ben.huddleston Ben Huddleston made changes -
            ben.huddleston Ben Huddleston made changes -
            ben.huddleston Ben Huddleston made changes -
            Link This issue relates to MB-35889 [ MB-35889 ]
            ben.huddleston Ben Huddleston made changes -
            Attachment jemalloc.svg [ 88270 ]
            ben.huddleston Ben Huddleston made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Sarath Lakshman [ sarath ]
            srinath.duvuru Srinath Duvuru made changes -
            Sprint Magma: Jan 20 - Feb 2, Magma: Apr2-Apr16 [ 968, 1000 ] Magma: Jan 20 - Feb 2 [ 968 ]
            srinath.duvuru Srinath Duvuru made changes -
            Rank Ranked higher
            sarath Sarath Lakshman made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            sarath Sarath Lakshman made changes -
            Assignee Sarath Lakshman [ sarath ] Bo-Chun Wang [ bo-chun.wang ]
            sarath Sarath Lakshman made changes -
            Epic Link CBSS-305 [ 112304 ] CBSS-376 [ 121213 ]
            sarath Sarath Lakshman made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.0 [ 17233 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s Cheshire-Cat [ 15915 ]

            People

              bo-chun.wang Bo-Chun Wang
              sarath Sarath Lakshman
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty