Duplicating attributes from dispatch_to_server to improve debugging
Description
Environment
None
Gerrit Reviews
None
Release Notes Description
None
Activity
Show:
Fixed
Details
Details
Assignee

Reporter

Story Points
1
Fix versions
Priority
Instabug
Open Instabug
PagerDuty
PagerDuty Incident
PagerDuty

PagerDuty Incident
Sentry
Linked Issues
Sentry
Linked Issues
Zendesk Support
Linked Tickets
Zendesk Support

Linked Tickets
Created July 13, 2021 at 12:11 PM
Updated July 13, 2021 at 12:20 PM
Resolved July 13, 2021 at 12:20 PM
Most, perhaps all, current tracing systems don't allow performing queries across spans. E.g. there is no way of finding dispatch_to_server spans where <some attribute in a parent span> == <some value>. One useful query along those lines would be checking the server_duration times against different buckets.
After discussions with one of the tracing providers, we understand there are two solutions to this: duplicate some attributes between spans, or use OpenTelemetry 'baggage'. In practice, baggage does not seem useful: it is unclear/undocumented how to attach it through the API and, many (perhaps all) tracing systems do not appear to expose it.
So, instead we will duplicate some useful attributes between the dispatch_to_server and top-level KV span. A full justification for which attributes is provided here: https://docs.google.com/document/d/1BAPS8bPMv8-4FPIdysgpxEsKrUgd595EAGOU-_nXHRY/edit?disco=AAAALpOPEZU