Repository navigation
Define a default tracing policy #584
Description
Expected Behavior
Users of substrate should be able to use tracing to identify problems in their systems by default.
Actual Behavior
Today, users cannot really use tracing to investigate failures because
- every components hardcodes its sampler at startup
- router is set to
NeverSample - ateapi requires
--tracevia the CLI to be traces - we never read
OTEL_TRACES_SAMPLER
Proposal
To follow OpenTelemetry recommendations, sampling should happen in the pipeline, not in the observed application itself.
So I am proposing the following:
- Control-plane sampled by default:
- ateapi, atelet, ateom set
ParentBased(AlwaysSample) - Control plane request volume is small compared to the data plane, and these are the traces we need when suspend/resume/restore needs troubleshooting.
- We can extend this via exemplars on the upcoming lifecycle duration histograms.
- ateapi, atelet, ateom set
- Data plane parent-based with a low default ratio at the root: The router sets
ParentBased(TraceIDRatioBased(small)). Clients sampled already stay sampled; the ratio defines how many unsampled requests we pull in. - Leverage
OTEL_TRACES_SAMPLER: This is the OTel-native way of exposing this to end users
Non-goals
- Shipping or mandating a specific collector setup. We can have guidelines around this as per Need more guidance on the configurability and scalability for OTel Collector setup #563
- Any per-workload or per-tenant sampling configs
Open questions
- Is a ratio at the router root acceptable for everyone as the data plane default, and what is a sane starting value? I would start conservative, e.g. 1%, it's configurable anyway.
- Do we want
always_onfor the lifecycle plane at target scale (1B actors means a lot of lifecycle ops), or shouldateapialso default to a higher (than 1%) ratio?
Activity
zlammerts-svg commentedon Jul 30, 2026
Da Huang (@git286) will review and provide feedback directly. Thanks!
git286 commentedon Jul 30, 2026
Thanks for this proposal and agreed with the direction! Sampling policy being unowned and unconfigurable is a real problem.
On the open questions: (1) yeah, let's be more conservative here (1% or less). I have some concern of the scalability of the centralized otel collector and we need to do more stress testing and benchmarking (cc Max Smythe (@maxsmythe) Jeff Luo (@JeffLuoo)) to understand the limit. (2) Can we split into operator-initiated (e.g., human run cmd, always on) vs. machine-driven lifecycle (ratio)?
krisztianfekete commentedon Jul 31, 2026
Thanks Da Huang (@git286)!
Re. (1): agreed, let's start at 1% or lower, I am fine with either, since it's configurable via the env var. Worth noting the low ratio we set is basically head sampling in the SDK, so whatever we drop at the root never reaches the collector at all. Happy to do the benchmarking work too to have actual numbers.
Re. (2): we can do the split via
ParentBasedI think:--trace, and every parent-based downstream keeps the whole request sampledOne thing to keep in mind with the operator path is that honoring a client-set sampled flag means any caller can force sampling for its own requests. I think that's acceptable as long as the collector owns the real volume limits (as it should), but if it ever becomes a concern, we can think about gating
--trace.If all sounds right I'll put up the PR with these defaults.
git286 commentedon Jul 31, 2026
yeah, the collector should limit the volume to the backend. Sounds good! Thanks, Krisztian F (@krisztianfekete) !
krisztianfekete commentedon Aug 3, 2026
Thanks Da Huang (@git286), the PR is open here: #711
Define a production-ready default tracing policy (#711)