Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49674

Reduce seqindex delta writeamp (was: Investigate high writeamp in showfast tests)

    XMLWordPrintable

Details

    • Task
    • Resolution: Unresolved
    • Major
    • Morpheus
    • None
    • storage-engine

    Description

      In 1B read heavy test, writeamp seen is around 9.6. In theory we expect it to be around 4. 

      Looking  at a single shard stats (file attached), it looks like lot of file count compactions are happening.
      "NCompacts": 416,
      "NFileCountCompacts": 300,

      Further 73% of writes done are due to compactions:
      "WriteAmp": 9.632818222045898
      "BytesIncoming": 125205009,
      "NWriteBytes": 1206077081,
      "NWriteBytesCompact": 888291328,

       

      Also it is odd that seqIndex L3 has high NWriteBytes as compared to total BytesIncoming seen by seqIndex.

      "BytesIncoming": 126275385,
      L3: "NWriteBytes": 206098432
       


      YCSB A latency test also has very high write amp (7.5.0-3014)

      90th Percentile READ Latency(ms), Workload A, 3 nodes, 1 bucket x 1.5B x 1KB, Uniform distribution, 30K ops/sec, 1% Resident Ratio, Magma

      Again the reason seems to be NFileCountCompactions. Note from a pure write IOPS perspective, this is making us do more of them in comparison to Couchstore (65k iops vs 40k iops per node).

       

      Note: We should check if this is seen in any other test as well.


      10B write heavy test:
      http://perf.jenkins.couchbase.com/job/rhea-5node2-10B/183/
      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=rhea_760-1186_access_a805

      Attachments

        For Gerrit Dashboard: MB-49674
        # Subject Branch Project Status CR V

        Activity

          People

            sarath Sarath Lakshman
            rohan.suri Rohan Suri
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There is 1 open Gerrit change

                PagerDuty