Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31834

Injection rate (initial load) degradation in DGM

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 6.5.0
    • 6.5.0
    • couchbase-bucket
    • 4 nodes
      Cluster: hera
      OS: CentOS 7
      CPU: E5-2630 v3 (32 vCPU)
      Memory: 64GB
      Disk: Samsung PM863
    • Untriaged
    • Centos 64-bit
    • Yes

    Description

      ...since build 6.5.0-1218

      Seems like this particular change causing the degradation:
      https://github.com/couchbase/kv_engine/commit/992ea93c17b636ce92fc34c1629790416bbe44fd

      Other changes in 1218:

      kv_engine changed from ccb8dbcbdd0c1387adf19468517a764809fdc8e9 to master
      [+] 0169935 Allow for setting AltClientResponse magic in Response class
      [+] 992ea93 MB-30916: Exploit the logic lifetime of CM::checkpointList
      [+] d09a656 MB-30302: Fix brace escaping and hex output in log messages

       

      Behavior:
      Injection rate goes down once server gets into non-100% memory residence during initial load.

      Test:
      4 nodes, "Full Eviction" mode, loading 1B docs with ~ 200K doc/sec rate.

       

      6.5.0-1217 vs 6.5.0-1225:
      full comparison: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hera_650-1225_load_33be&label=6.5.0-1225&snapshot=hera_650-1217_load_7aa7&label=6.5.0-1217

       

       

       

      logs 6.5.0-1225:

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2093/172.23.96.117.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2093/172.23.96.118.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2093/172.23.96.119.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2093/172.23.96.120.zip

       

      Attachments

        1. cpu.png
          cpu.png
          95 kB
        2. ops.png
          ops.png
          73 kB
        For Gerrit Dashboard: MB-31834
        # Subject Branch Project Status CR V

        Activity

          People

            oleksandr.gyryk Alex Gyryk (Inactive)
            oleksandr.gyryk Alex Gyryk (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty