OpenTelemetry request tracing doesn't properly handle sampling

Description

When tracing, a very common trace configuration is to pair two concepts for sampling:

  • If there is a parent span, follow and use it's sampling decision

  • If there is no parent span, make a ratio-based sampling decision

An example result of this approach is typically that 5% of all incoming HTTP requests are traced from beginning to end. In OpenTelemetry, this could be setup as follows:

Currently, if the parent span (i.e. the incoming HTTP request handler) is not sampled then this is not respected by the Couchbase "request_encoding", "dispatch_to_server", and similar spans (i.e. compression).

This is because the parent span link is lost when the request is queued on the connection pool when the parent span is not sampled. In this case, when the inner span is created a new sampling decision is made based on the ratio, and the parent span's decision to not be sampled is ignored.

The end result, for the example above, is that 5% of all HTTP requests are fully traced with all Couchbase internal operations traced. However, of the other 95% of operations, 5% of the Couchbase internal operations are randomly sampled as spans without a linked parent.

Environment

None

Gerrit Reviews

None

Release Notes Description

None

Activity

Show:
Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Story Points

Components

Fix versions

Affects versions

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created May 10, 2023 at 5:58 PM
Updated May 12, 2023 at 9:07 PM
Resolved May 12, 2023 at 9:07 PM
Instabug