Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38012

1% DGM Test: Write ops/s dropped from 230k/s to 400/s due to swapping

    XMLWordPrintable

Details

    • Untriaged
    • Unknown
    • Magma: Jan 20 - Feb 2

    Description

      A 2 node cluster running 1% DGM test abruptly dropped ops rate from 240k writes/sec to 400/s when the active resident is at 27%. The test was in the initial loading phase of 12B docs (loaded 460M). Memory used by magma is around 3G per node.

       Logs:
      https://uploads.couchbase.com/magma/collectinfo-2020-02-21T092448-ns_1%40172.23.97.25.zip
      https://uploads.couchbase.com/magma/collectinfo-2020-02-21T092448-ns_1%40172.23.97.26.zip

      Attachments

        1. gdb.txt
          571 kB
          Ben Huddleston
        2. jemalloc.svg
          105 kB
          Ben Huddleston
        3. Screenshot 2020-02-21 at 13.36.59.png
          391 kB
          Ben Huddleston
        4. Screenshot 2020-02-21 at 13.40.31.png
          443 kB
          Ben Huddleston
        5. Screenshot 2020-02-21 at 13.43.29.png
          365 kB
          Ben Huddleston
        6. Screenshot 2020-02-21 at 13.46.28.png
          362 kB
          Ben Huddleston
        7. Screenshot 2020-02-21 at 13.48.41.png
          299 kB
          Ben Huddleston
        8. Screenshot 2020-02-21 at 13.49.30.png
          369 kB
          Ben Huddleston
        9. Screenshot 2020-03-04 at 11.31.22.png
          209 kB
          Ben Huddleston
        10. Screenshot 2020-03-04 at 12.48.38.png
          202 kB
          Ben Huddleston
        11. Screenshot 2020-03-04 at 12.51.03.png
          365 kB
          Ben Huddleston
        12. Screenshot 2020-03-04 at 15.25.25.png
          383 kB
          Ben Huddleston

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-35889 Replication can get stuck when checkpoint memory overhead is very high

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. MB-36370 HiDD: Scalable DCP and disk backfill performance

          • Major - Major loss of function.
          • Reopened

          Improvement - An improvement or enhancement to an existing feature or task. MB-38079 Disk usage estimation with godu does not scale with large number of files

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              bo-chun.wang Bo-Chun Wang
              sarath Sarath Lakshman
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty