Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.5.0
-
4 nodes
Cluster: hera
OS: CentOS 7
CPU: E5-2630 v3 (32 vCPU)
Memory: 64GB
Disk: Samsung PM863
-
Untriaged
-
Centos 64-bit
-
Yes
Description
...since build 6.5.0-1218
Seems like this particular change causing the degradation:
https://github.com/couchbase/kv_engine/commit/992ea93c17b636ce92fc34c1629790416bbe44fd
Other changes in 1218:
kv_engine changed from ccb8dbcbdd0c1387adf19468517a764809fdc8e9 to master
[+] 0169935 Allow for setting AltClientResponse magic in Response class
[+] 992ea93 MB-30916: Exploit the logic lifetime of CM::checkpointList
[+] d09a656 MB-30302: Fix brace escaping and hex output in log messages
Behavior:
Injection rate goes down once server gets into non-100% memory residence during initial load.
Test:
4 nodes, "Full Eviction" mode, loading 1B docs with ~ 200K doc/sec rate.
6.5.0-1217 vs 6.5.0-1225:
full comparison: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hera_650-1225_load_33be&label=6.5.0-1225&snapshot=hera_650-1217_load_7aa7&label=6.5.0-1217
logs 6.5.0-1225:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2093/172.23.96.117.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2093/172.23.96.118.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2093/172.23.96.119.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2093/172.23.96.120.zip
Attachments
For Gerrit Dashboard: MB-31834 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
98714,11 | MB-30916: Don't free closed-checkpoint-memory under lock (2) | master | kv_engine | Status: MERGED | +2 | +1 |
101373,14 | MB-30916: Fix the QueueDirtyWithManyClosedUnrefCheckpoints benchmark | master | kv_engine | Status: MERGED | +2 | +1 |