query engine accounting information is generating unique keys causing ns_server to use & leak memory.
Description
Components
Labels
None
Environment
None
Link to Log File, atop/blg, CBCollectInfo, Core dump
None
Release Notes Description
None
blocks
Activity
Show:
Patrick Varley October 28, 2016 at 10:17 AM
Reopen it quickly to update the affected versions.
Patrick Varley October 27, 2016 at 2:15 PM
I'm I correct in believing 4.1.0 and 4.1.2 are also affected by this bug?
CB robot September 29, 2016 at 9:58 PM
Build 4.6.0-3318 contains query commit 7b304ddcca42e6b848e9ee26693d522f9ebd5142 with commit message:
https://couchbasecloud.atlassian.net/browse/MB-20521#icft=MB-20521 format statistic names avoiding fixed buffers
https://github.com/couchbase/query/commit/7b304ddcca42e6b848e9ee26693d522f9ebd5142
Matt Carabine August 26, 2016 at 4:10 PM
Verified that the memory bloat observed as a result of this issue is resolved after running build 4.5.1-2817 for a week.
Marco Greco August 15, 2016 at 4:16 PM
fixed with changes 66756 and 66765
Fixed
Pinned fields
Click on the next to a field label to start pinning.
Details
Details
Assignee
Marco Greco
Marco Greco(Deactivated)Reporter
Keshav Murthy
Keshav MurthyIs this a Regression?
No
Triage
Untriaged
Priority
CriticalInstabug
Open Instabug
PagerDuty
PagerDuty Incident
PagerDuty
PagerDuty Incident
Sentry
Linked Issues
Sentry
Linked Issues
Zendesk Support
Linked Tickets
Zendesk Support
Linked Tickets
Created August 13, 2016 at 12:19 AM
Updated October 3, 2017 at 10:51 AM
Resolved October 28, 2016 at 10:17 AM
Instabug
The leak happens in query stats collector. We expect a stable and finite set of stats from the cbq-engine. We cache some internal information on a per stat basis, so our memory usage is proportional to the number of stats returned by cbq-engine. In this particular case (and I was able to reproduce it easily), cbq-engine returns bunch of uniquely named stats, which results in ever growing memory consumption.
Short excerpt of the kind of stats returned by the cbq-engine:
<<"request_timer.stddevrequest_timer.99%request_timer.medianrequest_timer.75%request_timer.95%request_timer.1m.raterequest_timer.5m.raterequest_timer.countrequest_timer.minrequest_timer.mean.raterequest_timer.maxrequest_timer.meanrequest_timer.99.9%request_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.raterequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.raterequest_timer.mean.raterequest_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.raterequest_timer.mean.raterequest_timer.95%request_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.raterequest_timer.mean.raterequest_timer.95%request_timer.75%request_timer.1m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.raterequest_timer.mean.raterequest_timer.95%request_timer.75%request_timer.1m.raterequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.raterequest_timer.mean.raterequest_timer.95%request_timer.75%request_timer.1m.raterequest_timer.countrequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.raterequest_timer.mean.raterequest_timer.95%request_timer.75%request_timer.1m.raterequest_timer.countrequest_timer.maxrequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.raterequest_timer.mean.raterequest_timer.95%request_timer.75%request_timer.1m.raterequest_timer.countrequest_timer.maxrequest_timer.meanrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.raterequest_timer.mean.raterequest_timer.95%request_timer.75%request_timer.1m.raterequest_timer.countrequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.raterequest_timer.mean.raterequest_timer.95%request_timer.75%request_timer.1m.raterequest_timer.countrequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.99%request_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.15m.raterequest_timer.mean.raterequest_timer.95%request_timer.75%request_timer.1m.raterequest_timer.countrequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.99%request_timer.5m.raterequest_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.5m.raterequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.5m.raterequest_timer.maxrequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.5m.raterequest_timer.maxrequest_timer.75%request_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.5m.raterequest_timer.maxrequest_timer.75%request_timer.99%request_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.5m.raterequest_timer.maxrequest_timer.75%request_timer.99%request_timer.15m.raterequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.5m.raterequest_timer.maxrequest_timer.75%request_timer.99%request_timer.15m.raterequest_timer.mean.raterequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.5m.raterequest_timer.maxrequest_timer.75%request_timer.99%request_timer.15m.raterequest_timer.mean.raterequest_timer.countrequest_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.5m.raterequest_timer.maxrequest_timer.75%request_timer.99%request_timer.15m.raterequest_timer.mean.raterequest_timer.countrequest_timer.95%request_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.5m.raterequest_timer.maxrequest_timer.75%request_timer.99%request_timer.15m.raterequest_timer.mean.raterequest_timer.countrequest_timer.95%request_timer.medianrequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.5m.raterequest_timer.maxrequest_timer.75%request_timer.99%request_timer.15m.raterequest_timer.mean.raterequest_timer.countrequest_timer.95%request_timer.medianrequest_timer.meanrequest_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.countrequest_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.countrequest_timer.99%request_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.countrequest_timer.99%request_timer.5m.raterequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.countrequest_timer.99%request_timer.5m.raterequest_timer.maxrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.countrequest_timer.99%request_timer.5m.raterequest_timer.maxrequest_timer.medianrequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.countrequest_timer.99%request_timer.5m.raterequest_timer.maxrequest_timer.medianrequest_timer.75%request_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.countrequest_timer.99%request_timer.5m.raterequest_timer.maxrequest_timer.medianrequest_timer.75%request_timer.95%request_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.countrequest_timer.99%request_timer.5m.raterequest_timer.maxrequest_timer.medianrequest_timer.75%request_timer.95%request_timer.minrequest_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.countrequest_timer.99%request_timer.5m.raterequest_timer.maxrequest_timer.medianrequest_timer.75%request_timer.95%request_timer.minrequest_timer.15m.raterequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.countrequest_timer.99%request_timer.5m.raterequest_timer.maxrequest_timer.medianrequest_timer.75%request_timer.95%request_timer.minrequest_timer.15m.raterequest_timer.mean.raterequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean.raterequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean.raterequest_timer.meanrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean.raterequest_timer.meanrequest_timer.medianrequest_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean.raterequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean.raterequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean.raterequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean.raterequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.15m.raterequest_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean.raterequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.15m.raterequest_timer.minrequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean.raterequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.15m.raterequest_timer.minrequest_timer.countrequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.mean.raterequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.15m.raterequest_timer.minrequest_timer.countrequest_timer.maxrequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.meanrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.meanrequest_timer.medianrequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.meanrequest_timer.medianrequest_timer.mean.raterequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.meanrequest_timer.medianrequest_timer.mean.raterequest_timer.maxrequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.meanrequest_timer.medianrequest_timer.mean.raterequest_timer.maxrequest_timer.75%request_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.meanrequest_timer.medianrequest_timer.mean.raterequest_timer.maxrequest_timer.75%request_timer.95%request_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.meanrequest_timer.medianrequest_timer.mean.raterequest_timer.maxrequest_timer.75%request_timer.95%request_timer.99%request_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.meanrequest_timer.medianrequest_timer.mean.raterequest_timer.maxrequest_timer.75%request_timer.95%request_timer.99%request_timer.countrequest_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.meanrequest_timer.medianrequest_timer.mean.raterequest_timer.maxrequest_timer.75%request_timer.95%request_timer.99%request_timer.countrequest_timer.minrequest_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.meanrequest_timer.medianrequest_timer.mean.raterequest_timer.maxrequest_timer.75%request_timer.95%request_timer.99%request_timer.countrequest_timer.minrequest_timer.5m.raterequest_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.medianrequest_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.medianrequest_timer.95%request_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.meanrequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.meanrequest_timer.75%request_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.meanrequest_timer.75%request_timer.countrequest_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.meanrequest_timer.75%request_timer.countrequest_timer.minrequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.meanrequest_timer.75%request_timer.countrequest_timer.minrequest_timer.maxrequest_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.medianrequest_timer.95%request_timer.99%request_timer.5m.raterequest_timer.meanrequest_timer.75%request_timer.countrequest_timer.minrequest_timer.maxrequest_timer.15m.raterequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.minrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.minrequest_timer.medianrequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.minrequest_timer.medianrequest_timer.75%request_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.minrequest_timer.medianrequest_timer.75%request_timer.5m.raterequest_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.minrequest_timer.medianrequest_timer.75%request_timer.5m.raterequest_timer.15m.raterequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.minrequest_timer.medianrequest_timer.75%request_timer.5m.raterequest_timer.15m.raterequest_timer.countrequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.minrequest_timer.medianrequest_timer.75%request_timer.5m.raterequest_timer.15m.raterequest_timer.countrequest_timer.meanrequest_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.minrequest_timer.medianrequest_timer.75%request_timer.5m.raterequest_timer.15m.raterequest_timer.countrequest_timer.meanrequest_timer.95%request_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.minrequest_timer.medianrequest_timer.75%request_timer.5m.raterequest_timer.15m.raterequest_timer.countrequest_timer.meanrequest_timer.95%request_timer.99%request_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.1m.raterequest_timer.minrequest_timer.medianrequest_timer.75%request_timer.5m.raterequest_timer.15m.raterequest_timer.countrequest_timer.meanrequest_timer.95%request_timer.99%request_timer.mean.raterequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.maxrequest_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.maxrequest_timer.99%request_timer.1m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.maxrequest_timer.99%request_timer.1m.raterequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.maxrequest_timer.99%request_timer.1m.raterequest_timer.countrequest_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.maxrequest_timer.99%request_timer.1m.raterequest_timer.countrequest_timer.minrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.maxrequest_timer.99%request_timer.1m.raterequest_timer.countrequest_timer.minrequest_timer.medianrequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.maxrequest_timer.99%request_timer.1m.raterequest_timer.countrequest_timer.minrequest_timer.medianrequest_timer.meanrequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.maxrequest_timer.99%request_timer.1m.raterequest_timer.countrequest_timer.minrequest_timer.medianrequest_timer.meanrequest_timer.75%request_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.maxrequest_timer.99%request_timer.1m.raterequest_timer.countrequest_timer.minrequest_timer.medianrequest_timer.meanrequest_timer.75%request_timer.95%request_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.maxrequest_timer.99%request_timer.1m.raterequest_timer.countrequest_timer.minrequest_timer.medianrequest_timer.meanrequest_timer.75%request_timer.95%request_timer.15m.raterequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%request_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%request_timer.15m.raterequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%request_timer.15m.raterequest_timer.maxrequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%request_timer.15m.raterequest_timer.maxrequest_timer.meanrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%request_timer.15m.raterequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%request_timer.15m.raterequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%request_timer.15m.raterequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.minrequest_timer.1m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%request_timer.15m.raterequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.minrequest_timer.1m.raterequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.minrequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.minrequest_timer.75%request_timer.1m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.minrequest_timer.75%request_timer.1m.raterequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.minrequest_timer.75%request_timer.1m.raterequest_timer.medianrequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.minrequest_timer.75%request_timer.1m.raterequest_timer.medianrequest_timer.countrequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.minrequest_timer.75%request_timer.1m.raterequest_timer.medianrequest_timer.countrequest_timer.meanrequest_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.minrequest_timer.75%request_timer.1m.raterequest_timer.medianrequest_timer.countrequest_timer.meanrequest_timer.95%request_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.minrequest_timer.75%request_timer.1m.raterequest_timer.medianrequest_timer.countrequest_timer.meanrequest_timer.95%request_timer.99%request_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.5m.raterequest_timer.mean.raterequest_timer.minrequest_timer.75%request_timer.1m.raterequest_timer.medianrequest_timer.countrequest_timer.meanrequest_timer.95%request_timer.99%request_timer.15m.raterequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%request_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%request_timer.99%request_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%request_timer.99%request_timer.minrequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%request_timer.99%request_timer.minrequest_timer.75%request_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%request_timer.99%request_timer.minrequest_timer.75%request_timer.medianrequest_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%request_timer.99%request_timer.minrequest_timer.75%request_timer.medianrequest_timer.5m.raterequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%request_timer.99%request_timer.minrequest_timer.75%request_timer.medianrequest_timer.5m.raterequest_timer.maxrequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%request_timer.99%request_timer.minrequest_timer.75%request_timer.medianrequest_timer.5m.raterequest_timer.maxrequest_timer.meanrequest_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%request_timer.99%request_timer.minrequest_timer.75%request_timer.medianrequest_timer.5m.raterequest_timer.maxrequest_timer.meanrequest_timer.15m.raterequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%request_timer.99%request_timer.minrequest_timer.75%request_timer.medianrequest_timer.5m.raterequest_timer.maxrequest_timer.meanrequest_timer.15m.raterequest_timer.mean.raterequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.95%request_timer.99%request_timer.minrequest_timer.75%request_timer.medianrequest_timer.5m.raterequest_timer.maxrequest_timer.meanrequest_timer.15m.raterequest_timer.mean.raterequest_timer.countrequest_timer.1m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.countrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.countrequest_timer.medianrequest_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.countrequest_timer.medianrequest_timer.15m.raterequest_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.countrequest_timer.medianrequest_timer.15m.raterequest_timer.minrequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.countrequest_timer.medianrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.countrequest_timer.medianrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.1m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.countrequest_timer.medianrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.1m.raterequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.countrequest_timer.medianrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.1m.raterequest_timer.mean.raterequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.countrequest_timer.medianrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.1m.raterequest_timer.mean.raterequest_timer.75%request_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.countrequest_timer.medianrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.1m.raterequest_timer.mean.raterequest_timer.75%request_timer.95%request_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.countrequest_timer.medianrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.1m.raterequest_timer.mean.raterequest_timer.75%request_timer.95%request_timer.99%request_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.raterequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.raterequest_timer.countrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.raterequest_timer.countrequest_timer.medianrequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.raterequest_timer.countrequest_timer.medianrequest_timer.75%request_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.raterequest_timer.countrequest_timer.medianrequest_timer.75%request_timer.95%request_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.raterequest_timer.countrequest_timer.medianrequest_timer.75%request_timer.95%request_timer.99%request_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.raterequest_timer.countrequest_timer.medianrequest_timer.75%request_timer.95%request_timer.99%request_timer.5m.raterequest_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.raterequest_timer.countrequest_timer.medianrequest_timer.75%request_timer.95%request_timer.99%request_timer.5m.raterequest_timer.minrequest_timer.1m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.raterequest_timer.countrequest_timer.medianrequest_timer.75%request_timer.95%request_timer.99%request_timer.5m.raterequest_timer.minrequest_timer.1m.raterequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.raterequest_timer.countrequest_timer.medianrequest_timer.75%request_timer.95%request_timer.99%request_timer.5m.raterequest_timer.minrequest_timer.1m.raterequest_timer.maxrequest_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.mean.raterequest_timer.countrequest_timer.medianrequest_timer.75%request_timer.95%request_timer.99%request_timer.5m.raterequest_timer.minrequest_timer.1m.raterequest_timer.maxrequest_timer.15m.raterequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.min">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.99%request_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.99%request_timer.meanrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.99%request_timer.meanrequest_timer.medianrequest_timer.1m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.99%request_timer.meanrequest_timer.medianrequest_timer.1m.raterequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.99%request_timer.meanrequest_timer.medianrequest_timer.1m.raterequest_timer.countrequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.99%request_timer.meanrequest_timer.medianrequest_timer.1m.raterequest_timer.countrequest_timer.75%request_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.99%request_timer.meanrequest_timer.medianrequest_timer.1m.raterequest_timer.countrequest_timer.75%request_timer.95%request_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.99%request_timer.meanrequest_timer.medianrequest_timer.1m.raterequest_timer.countrequest_timer.75%request_timer.95%request_timer.5m.raterequest_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.99%request_timer.meanrequest_timer.medianrequest_timer.1m.raterequest_timer.countrequest_timer.75%request_timer.95%request_timer.5m.raterequest_timer.15m.raterequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.99%request_timer.meanrequest_timer.medianrequest_timer.1m.raterequest_timer.countrequest_timer.75%request_timer.95%request_timer.5m.raterequest_timer.15m.raterequest_timer.mean.raterequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.max">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.maxrequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.1m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.1m.raterequest_timer.5m.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.1m.raterequest_timer.5m.raterequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.1m.raterequest_timer.5m.raterequest_timer.mean.raterequest_timer.count">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.1m.raterequest_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.1m.raterequest_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%request_timer.75%">>,
<<"request_timer.stddevrequest_timer.99.9%request_timer.minrequest_timer.maxrequest_timer.meanrequest_timer.medianrequest_timer.95%request_timer.1m.raterequest_timer.5m.raterequest_timer.mean.raterequest_timer.countrequest_timer.99%request_timer.75%request_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.count">>,
<<"request_timer.stddevrequest_timer.countrequest_timer.mean">>,
<<"request_timer.stddevrequest_timer.countrequest_timer.meanrequest_timer.15m.rate">>,
<<"request_timer.stddevrequest_timer.countrequest_timer.meanrequest_timer.15m.raterequest_timer.min">>,
<<"request_timer.stddevrequest_timer.countrequest_timer.meanrequest_timer.15m.raterequest_timer.minrequest_timer.max">>,
<<"request_timer.stddevrequest_timer.countrequest_timer.meanrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.median">>,
<<"request_timer.stddevrequest_timer.countrequest_timer.meanrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.medianrequest_timer.mean.rate">>,
<<"request_timer.stddevrequest_timer.countrequest_timer.meanrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.medianrequest_timer.mean.raterequest_timer.75%">>,
<<"request_timer.stddevrequest_timer.countrequest_timer.meanrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.medianrequest_timer.mean.raterequest_timer.75%request_timer.95%">>,
<<"request_timer.stddevrequest_timer.countrequest_timer.meanrequest_timer.15m.raterequest_timer.minrequest_timer.maxrequest_timer.medianrequest_timer.mean.raterequest_timer.75%request_timer.95%request_timer.99%">>,