Details
-
Task
-
Resolution: Unresolved
-
Major
-
None
Description
In 1B read heavy test, writeamp seen is around 9.6. In theory we expect it to be around 4.
Looking at a single shard stats (file attached), it looks like lot of file count compactions are happening.
"NCompacts": 416,
"NFileCountCompacts": 300,
Further 73% of writes done are due to compactions:
"WriteAmp": 9.632818222045898
"BytesIncoming": 125205009,
"NWriteBytes": 1206077081,
"NWriteBytesCompact": 888291328,
Also it is odd that seqIndex L3 has high NWriteBytes as compared to total BytesIncoming seen by seqIndex.
"BytesIncoming": 126275385,
L3: "NWriteBytes": 206098432
YCSB A latency test also has very high write amp (7.5.0-3014)
90th Percentile READ Latency(ms), Workload A, 3 nodes, 1 bucket x 1.5B x 1KB, Uniform distribution, 30K ops/sec, 1% Resident Ratio, Magma
Again the reason seems to be NFileCountCompactions. Note from a pure write IOPS perspective, this is making us do more of them in comparison to Couchstore (65k iops vs 40k iops per node).
Note: We should check if this is seen in any other test as well.
10B write heavy test:
http://perf.jenkins.couchbase.com/job/rhea-5node2-10B/183/
http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=rhea_760-1186_access_a805
Attachments
Gerrit Reviews
For Gerrit Dashboard: MB-49674 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
193974,4 | MB-49674 toy: fix dynamic sizing | master | magma | Status: NEW | 0 | -1 |