Define a default tracing policy

[krisztianfekete]

Expected Behavior

Users of substrate should be able to use tracing to identify problems in their systems by default.

See also #580, #581

Actual Behavior

Today, users cannot really use tracing to investigate failures because

• every components hardcodes its sampler at startup
• router is set to NeverSample
• ateapi requires --trace via the CLI to be traces
• we never read OTEL_TRACES_SAMPLER

Proposal

To follow OpenTelemetry recommendations, sampling should happen in the pipeline, not in the observed application itself.

So I am proposing the following:

• Control-plane sampled by default:
  • ateapi, atelet, ateom set ParentBased(AlwaysSample)
  • Control plane request volume is small compared to the data plane, and these are the traces we need when suspend/resume/restore needs troubleshooting.
  • We can extend this via exemplars on the upcoming lifecycle duration histograms.
• Data plane parent-based with a low default ratio at the root: The router sets ParentBased(TraceIDRatioBased(small)). Clients sampled already stay sampled; the ratio defines how many unsampled requests we pull in.
• Leverage OTEL_TRACES_SAMPLER: This is the OTel-native way of exposing this to end users

Non-goals

• Shipping or mandating a specific collector setup. We can have guidelines around this as per Need more guidance on the configurability and scalability for OTel Collector setup #563
• Any per-workload or per-tenant sampling configs

Open questions

1. Is a ratio at the router root acceptable for everyone as the data plane default, and what is a sane starting value? I would start conservative, e.g. 1%, it's configurable anyway.
2. Do we want always_on for the lifecycle plane at target scale (1B actors means a lot of lifecycle ops), or should ateapi also default to a higher (than 1%) ratio?