otel trace sampling

[n1ru4l]

Customers want control over sampling the graphql requests reported by the gateway to hive console. E.g. they want to ALWAYS report all spans whose parent span has errors, but have a custom sampling rate for every other "successful" graphql request.

It should be possible to customize this behaviour via simple and intuitive configuration.

[EmrysMyrddin]

We can explore this, but the official way in OTEL to do this would be to use a Tail Processor. It would need to setup an OTEL Collector that would concentrate all the traces before sending them to Hive Console.

OTEL Collector can be configured to transform and filter traces with a set of rules. It is possible to apply those rules after the trace is considered complete (the root span has ended). For this to work, all spans have to be reported to the same collector so that it can have the whole picture of trace and know if any error happened in any of its spans.

If you think it makes sense, I can explore doing something similar at the gateway level, but it would imply that it will work only on errors known by the gateway itself. And it would imply manually retaining spans until they actually end so that they can be sent or not based on its final state.

[EmrysMyrddin]

After exploring what can be done, I'm not sure it is doable at all the gateway level:

• We will have to enable tracing for all requests (with the associated cost), in case the trace turns out to have an error state. So even for successful requests, we will just not actually send them to the tracing backend, but we will pay the runtime cost of recording the spans.
• We have no ways to inform the subgraph not to send spans to the tracing backend that are part of a trace that shouldn't be sampled. This means you will end up with orphan spans with trace ID pointing to non-existing trace.
• We will have to implement this as a SpanProcessor, which is a bit misleading, and not that straight forward to do without error (we will have to keep a hierarchy of spans, with the edge case of linked spans that we should filter if they finally are not recorded).