Details
-
Improvement
-
Resolution: Fixed
-
Major
-
6.5.0
Description
A quick Q1 test on my little perfrunner cluster shows that if the accounting store is set to stub: rather than gometrics:, a 50 client run has a 20k reqs/sec and an average end to end latency of 2.5 ms, vs 17k reqs/sec and an end to end latency of 2.9ms for gometrics:
In short +10% throughput -15% latency.
This is due to the fact that accounting each requests requires incrementing 14 counters, and accessing each counter is done exclusively locking the metrics store:
With 192 servicers each running a request, there will be a pile up of 2.7k lock requests before each servicer can move on the next request.
We need to have our own fork of gometrics were counter access is done using read locks.
Ideally, counters should be slices rather than maps, so that we don't have to hash the same strings over and over again to increment an atomic integer!
The impact of this further improvement has yet to be assessed, though.
Attachments
For Gerrit Dashboard: MB-31381 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
99923,4 | MB-31381 improve performance of metrics recording | master | query | Status: MERGED | +2 | +1 |
100382,3 | MB-31381 improve performance of metrics recording | master | query | Status: MERGED | +2 | +1 |
100383,2 | MB-31381 improve performance of metrics recording | master | query | Status: MERGED | +2 | +1 |
101591,4 | MB-31381 Accruing Vitals is a bottleneck | master | query | Status: MERGED | +2 | +1 |
101601,5 | MB-31381 bottleneck in Vitals | master | query | Status: MERGED | +2 | +1 |
101614,4 | MB-31381 bottleneck in Vitals | master | query | Status: MERGED | +2 | +1 |
101876,4 | MB-31381 shard sample reservoir to avoid contention | master | query | Status: MERGED | +2 | +1 |