Define a production-ready default tracing policy by krisztianfekete · Pull Request #711 · agent-substrate/substrate

Krisztian F (krisztianfekete) · 2026-08-03T09:58:10Z

Today everything outside ateapi hardcodes ParentBased(NeverSample), and setting OTEL_TRACES_SAMPLER does nothing because an explicit sampler silences the SDK's env handling. This makes troubleshooting production issues impossible via traces.

This PR gives every component a sane default and makes the standard OTel env vars work:

Control plane (ateapi, atelet, ateom) defaults to parentbased_traceidratio 0.1, the router (data plane root) to 0.01, per the discussion on Define a default tracing policy #584.
OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG override any of this without a rebuild. Invalid values keep the component default and log a warning instead of inheriting the SDK's fall-open-to-100% behavior.
Envoy's RandomSampling is derived from the router's resolved policy, so the two root decisions cannot drift.
kubectl-ate without --trace no longer installs a tracer provider at all: the old NeverSample provider injected sampled=0, which pinned every parent based sampler downstream and would have defeated the server side ratios. --trace still forces a full end to end trace.
ate-controller propagates the two env vars to the ateom worker pods it creates, same as the metric export vars.
kind pins ateapi to parentbased_always_on, so the local Jaeger flow keeps showing every API call.
The agentgateway integration follows what we have above. its randomSampling changes from true to 0.01 to match the data plane default, though as static config it does not follow OTEL_TRACES_SAMPLER overrides, so the two need adjusting together.

Verified on a kind cluster e2e manually. It resolved samplers logged at startup, a --trace resume produced one trace across ateapi, atelet, and ateom, an unsampled CLI call got picked up server side, Envoy continued a sampled traceparent while sampling 0 of 30 parentless requests at the 1% default, and an invalid env value fell back to the component default.

Gating who may use --trace stays a separate follow-up (and a discussion), and the more sophisticaed tracing policies belongs in a collector, not in substrate.

Fixes #584
cc. Da Huang (@git286)

Tests pass
Appropriate changes to documentation are included in the PR

Max Smythe (maxsmythe) · 2026-08-04T06:19:57Z

Da Huang (@git286) for observability

Also, there are merge conflicts that need resolving

Krisztian F (krisztianfekete) · 2026-08-04T06:28:13Z

Thanks Max Smythe (@maxsmythe), one usually has to resolve conflicts 5+ times before reviews, will do it shortly.
Also, not sure what we can do about auto-adding folks for reviews, or just adding the ability for non-org members to request reviews as currently I cannot do that. Maybe it could improve the process as well.

Max Smythe (maxsmythe) · 2026-08-04T06:32:52Z

Yeah, as far as adding reviewers... we are looking to at least get some manual coverage for that in the short term.

No worries if you want to wait for reviewer activity for rebase.

…AMPLER

Jeff Luo (JeffLuoo) · 2026-08-04T13:57:50Z

+		return def, nil
+	}
+
+	ratio := 1.0


My understanding is that if the arg is unset, we will set it to 100% here. It may conflict with we intent to not set it to 100% in this PR. Should we set the default value here to a lower value?

The current behavior follows the OTel spec (missing OTEL_TRACES_SAMPLER_ARG means ratio 1.0) with a warning. But you have point that this is the one remaining fail open to 100% path, and the PR's approach is that invalid missing configs keeps the component default, so I can make this consistent with the rest, thank you!

Jeff Luo (JeffLuoo) · 2026-08-04T13:58:57Z

+		slog.WarnContext(ctx, "Invalid trace sampler environment, keeping the component default",
+			slog.String("sampler", name),
+			slog.String("arg", arg),
+			slog.String("default", def.sampler.Description()),


def.sampler may be unset, can you handle it here and and a unit test to cover unset def.sampler?

Yep, will do this shortly!

…n warn log

The measurement method was a set of manual steps: four kubectl set env calls, a curl to the locust web UI, kubectl top, and results typed into markdown by hand. Move each step onto the machinery the suite already has, so a run repeats without a runbook. - hack/install-ate.sh gains --otlp-endpoint. Every control plane component reads OTEL_EXPORTER_OTLP_ENDPOINT from the ate-otel-config ConfigMap through envFrom, and ate-controller copies it on to the ateom worker pods, so one patch replaces the four set env calls. A ConfigMap edit starts no rollout on its own; restart the consumers that already exist. - The glutton ActorTemplate declared no env, so its exporter fell back to localhost:4317 and every actor span was discarded silently. Set the endpoint and service.name, and route it through a new --otlp-endpoint on workloads/deploy.sh. ActorTemplate.spec is immutable, so deploy now deletes the templates before applying. - Move telemetry-meter.yaml to telemetry/meter.yaml and add telemetry/README.md with the method: deploy, measure, read, clean up. The meter itself is unchanged. - Split observability.md by shelf life. The volume model and the Deployment/DaemonSet guidance move to docs/dev/best-practices/otel-collector.md, next to the sizing section that already stated a shorter version of the same model, now corrected for the ParentBased defaults from agent-substrate#711. Results stay, marked as a snapshot measured 2026-08-04 at 15eecd0. - Point the scenario ladder at headless runs. The web UI warning described two problems that only exist there: --trace-probability never reaching the worker, and the boomer-glutton sidecar generating load regardless of the selected user class. runner.py passes the flag at process start and records it in the run config header. The tests.yaml ladder and the per-run artifact writer follow separately.

The measurement method was a sequence of manual steps: four kubectl set env commands, a curl to the locust web UI, kubectl top, and results typed into markdown by hand. Move each step to the machinery that the suite already has, thus an operator can repeat a run without a runbook. - hack/install-ate.sh gets --otlp-endpoint. Each control plane component reads OTEL_EXPORTER_OTLP_ENDPOINT from the ate-otel-config ConfigMap through envFrom, and ate-controller copies the value to the ateom worker pods. Thus one patch replaces the four set env commands. A change to a ConfigMap starts no rollout, thus restart the consumers that are already present. - The glutton ActorTemplate had no env. Thus its exporter used localhost:4317, and substrate discarded each actor span with no error message. Set the endpoint and service.name, and add --otlp-endpoint to workloads/deploy.sh. ActorTemplate.spec is immutable, thus the deploy function now deletes the templates before it applies them. - Move telemetry-meter.yaml to telemetry/meter.yaml and add telemetry/README.md with the method: install, measure, read, remove. The meter itself does not change. - Divide observability.md by how long the content stays correct. The volume model and the guidance for Deployment or DaemonSet move to docs/dev/best-practices/otel-collector.md, adjacent to the size section that gave a shorter form of the same model. That section is now correct for the ParentBased defaults from agent-substrate#711. The results stay in observability.md as a snapshot, measured 2026-08-04 at 15eecd0. - Send the scenario ladder to headless runs. The warning about the web UI described two problems that occur only there: --trace-probability does not come to the worker, and the boomer-glutton sidecar makes load for each selected user class. runner.py sends the flag at process start and writes it in the run config header. - Write all prose in Simplified Technical English (ASD-STE100). The tests.yaml ladder and the writer for the artifacts of each run come in a separate PR.

The measurement method was a sequence of manual steps: four kubectl set env commands, a curl to the locust web UI, kubectl top, and results typed into markdown by hand. Move each step to the machinery that the suite already has, thus an operator can repeat a run without a runbook. - hack/install-ate.sh gets --otlp-endpoint. Each control plane component reads OTEL_EXPORTER_OTLP_ENDPOINT from the ate-otel-config ConfigMap through envFrom, and ate-controller copies the value to the ateom worker pods. Thus one patch replaces the four set env commands. A change to a ConfigMap starts no rollout, thus restart the consumers that are already present. - The glutton ActorTemplate had no env. Thus its exporter used localhost:4317, and substrate discarded each actor span with no error message. Set the endpoint and service.name, and add --otlp-endpoint to workloads/deploy.sh. ActorTemplate.spec is immutable, thus the deploy function now deletes the templates before it applies them. - Move telemetry-meter.yaml to telemetry/meter.yaml and add telemetry/README.md with the method: install, measure, read, remove. The meter itself does not change. - Divide observability.md by how long the content stays correct. The volume model and the guidance for Deployment or DaemonSet move to docs/dev/best-practices/otel-collector.md, adjacent to the size section that gave a shorter form of the same model. That section is now correct for the ParentBased defaults from agent-substrate#711, and it records the date and the commit of the measurement that gave its values. - Keep no results in the repository. observability.md holds the prerequisites and the scenario ladder only. A result in the repository becomes incorrect after a change to the sample rates or to the instruments, and no reader can then see which copy is correct. The automation writes the values of each run to GCS. - Send the scenario ladder to headless runs. The warning about the web UI described two problems that occur only there: --trace-probability does not come to the worker, and the boomer-glutton sidecar makes load for each selected user class. runner.py sends the flag at process start and writes it in the run config header. - Write all prose in Simplified Technical English (ASD-STE100). The tests.yaml ladder and the writer for the artifacts of each run come in a separate PR.

Part of agent-substrate#563. The suite measured how fast substrate is. Nothing measured how much telemetry it produces, or what that telemetry costs the OTel collector. This change adds the measurement, and builds it from the parts that the suite already has. The managed collector sends its self-metrics to Cloud Monitoring only, and gives no data for each source. Thus benchmarking/telemetry/meter.yaml adds a second collector that counts the spans and the datapoints by service.name. The meter is a tee: it counts the telemetry and then sends it on to the managed collector, thus one run gives the volume for each service and the cost of the load together. Contents: - telemetry/meter.yaml and telemetry/README.md: the instrument and the method. Install, measure, read, remove. - hack/install-ate.sh --otlp-endpoint: sends all control plane telemetry to the meter. Each component reads the endpoint from the ate-otel-config ConfigMap through envFrom, and ate-controller copies the value to the ateom worker pods, thus one patch is sufficient for all of them. The override runs after the bundle apply, because the bundle has its own copy of the ConfigMap and would replace an earlier patch with no error message. deploy_locust.sh forwards the flag. - workloads/deploy.sh --otlp-endpoint: the same for actor containers, which substrate configures separately. - monitoring.yaml: scrapes the meter and cAdvisor. - observability.md: the prerequisites and the scenario ladder. - docs/dev/best-practices/otel-collector.md: the volume model and the size guidance for Deployment or DaemonSet. This file already had a shorter form of the same model, which was incorrect after the ParentBased defaults from agent-substrate#711. Correction: the glutton ActorTemplate had no env. Substrate puts no OTLP configuration in an actor container, thus the exporter used the SDK default localhost:4317, no process listens on that address in the sandbox, and substrate discarded each actor span with no error message. The template now sets OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_RESOURCE_ATTRIBUTES. ActorTemplate.spec is immutable, thus the deploy function deletes the templates before it applies them. Findings: - With ParentBased, the caller makes the root decision. Locust always sends a traceparent, thus its --trace-probability is the root rate for benchmark traffic and the default in ateapi does not apply. - The metrics increase with the number of components, not with the load. The spans increase with the rate of the operations. - An idle worker pod sends zero datapoints. Lifecycle traffic starts the telemetry of ateom, not the number of actors. - atelet and ateom-gvisor send no spans of their own. They inherit the decision that ateapi makes. - Actor throughput becomes constant at the limit of the snapshot suspend and restore operations, not at a limit of the telemetry. At that load the collector used a small part of its memory limit but stayed at maxReplicas, thus throughput is the wrong axis for the size calculation. The number of pods in the cluster sets the limit. - A suspend or resume operation fails on a node pool with more than one CPU vendor. gVisor restores a saved kernel image, thus the host that restores it must have the CPU functions of the host that made the checkpoint. observability.md documents the nodeAffinity correction. The guidance is qualitative. It gives the shape of each curve and tells the reader to measure the values for their own cluster. No result tables go in the repository, because a result becomes incorrect after a change to the sample rates or to the instruments. All prose is in Simplified Technical English (ASD-STE100).

Part of agent-substrate#563. The suite measured how fast substrate is. Nothing measured how much telemetry it produces, or what that telemetry costs the OTel collector. This change adds the measurement, and builds it from the parts that the suite already has. The managed collector sends its self-metrics to Cloud Monitoring only, and gives no data for each source. Thus benchmarking/telemetry/meter.yaml adds a second collector that counts the spans and the datapoints by service.name. The meter is a tee: it counts the telemetry and then sends it on to the managed collector, thus one run gives the volume for each service and the cost of the load together. Contents: - telemetry/meter.yaml and telemetry/README.md: the instrument and the method. Install, measure, read, remove. - hack/install-ate.sh --otlp-endpoint: sends all control plane telemetry to the meter. Each component reads the endpoint from the ate-otel-config ConfigMap through envFrom, and ate-controller copies the value to the ateom worker pods, thus one patch is sufficient for all of them. The override runs after the bundle apply, because the bundle has its own copy of the ConfigMap and would replace an earlier patch with no error message. deploy_locust.sh forwards the flag. - workloads/deploy.sh --otlp-endpoint: the same for actor containers, which substrate configures separately. - monitoring.yaml: scrapes the meter and cAdvisor. - observability.md: the prerequisites and the scenario ladder. - docs/dev/best-practices/otel-collector.md: the volume model and the size guidance for Deployment or DaemonSet. This file already had a shorter form of the same model, which was incorrect after the ParentBased defaults from agent-substrate#711. Correction: the glutton ActorTemplate had no env. Substrate puts no OTLP configuration in an actor container, thus the exporter used the SDK default localhost:4317, no process listens on that address in the sandbox, and substrate discarded each actor span with no error message. The template now sets OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_RESOURCE_ATTRIBUTES. ActorTemplate.spec is immutable, thus the deploy function deletes the templates before it applies them. Findings: - With ParentBased, the caller makes the root decision. Locust always sends a traceparent, thus its --trace-probability is the root rate for benchmark traffic and the default in ateapi does not apply. - The metrics increase with the number of components, not with the load. The spans increase with the rate of the operations. - An idle worker pod sends zero datapoints. Lifecycle traffic starts the telemetry of ateom, not the number of actors. - atelet and ateom-gvisor send no spans of their own. They inherit the decision that ateapi makes. - Actor throughput becomes constant at the limit of the snapshot suspend and restore operations, not at a limit of the telemetry. At that load the collector used a small part of its memory limit but stayed at maxReplicas, thus throughput is the wrong axis for the size calculation. The number of pods in the cluster sets the limit. The guidance is qualitative. It gives the shape of each curve and tells the reader to measure the values for their own cluster. No result tables go in the repository, because a result becomes incorrect after a change to the sample rates or to the instruments. All prose is in Simplified Technical English (ASD-STE100).

…749) Part of #563 Substrate had no way to measure how much telemetry it sends. This PR adds one, and builds it from the parts the benchmark suite already has. The managed collector sends its self-metrics to Cloud Monitoring only, and gives no data for each source. Thus you cannot see which component makes the volume. A second collector, the telemetry meter, counts the spans and the datapoints by `service.name`. ## Contents - **`benchmarking/telemetry/meter.yaml`** — the meter. A tee: it counts the telemetry by service, then sends it on to the managed collector. Thus one run gives the volume for each service and the cost of the load together. - **`benchmarking/telemetry/README.md`** — the method: install, measure, read, remove. - **`hack/install-ate.sh --otlp-endpoint`** — sends all control plane telemetry to the meter. Each component reads the endpoint from the `ate-otel-config` ConfigMap through `envFrom`, and `ate-controller` copies the value to the ateom worker pods, thus one patch is sufficient for all of them. - **`benchmarking/workloads/deploy.sh --otlp-endpoint`** — the same for actor containers, which substrate configures separately. - **`benchmarking/monitoring.yaml`** — scrapes the meter and cAdvisor. - **`benchmarking/observability.md`** — prerequisites and the scenario ladder. - **`docs/dev/best-practices/otel-collector.md`** — the volume model and the size guidance, corrected for the `ParentBased` defaults from #711. ## Correction The `glutton` ActorTemplate had no `env`. Substrate puts no OTLP configuration in an actor container, thus the exporter used the SDK default `localhost:4317`, no process listens on that address in the sandbox, and substrate discarded each actor span with no error message. The template now sets `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES`. `ActorTemplate.spec` is immutable, thus `deploy.sh --deploy` deletes the templates before it applies them. ## How I verified this On a GKE cluster with managed OTel. The sequence: ```bash # 1. Start from a clean state. ./hack/install-ate.sh --delete-benchmarks ./hack/install-ate.sh --delete-ate-system # 2. Install substrate from this branch. ./hack/install-ate.sh --deploy-ate-system # 3. Install the meter and Prometheus. kubectl apply -f benchmarking/telemetry/meter.yaml kubectl apply -f benchmarking/monitoring.yaml # 4. Send the control plane to the meter. METER=http://telemetry-meter.benchmarking.svc.cluster.local:4317 ./hack/install-ate.sh --deploy-ate-system --otlp-endpoint "${METER}" # 5. Install the workloads and locust, with the actors on the meter also. ./benchmarking/deploy_locust.sh --deploy --worker-count 10 \ --otlp-endpoint "${METER}" # 6. Read the volume. kubectl port-forward -n benchmarking svc/prometheus 9090:9090 ``` ## Result The idle floor, from one measurement on 10 workers: | Service | datapoints/min | |---|---| | `ate-controller` | 234.8 | | `ateapi` | 54.3 | | `atelet` | 44.9 | | `atenet-router` | 10.4 | | `ateom-gvisor` | 5.2 | | `glutton-actor` | 4.2 | | **Total** | **353.7** | The `glutton-actor` row is the correction above. Before it, actor telemetry could not arrive at all. Positive control for the same window: `otelcol_receiver_accepted_metric_points` 10.5/s, and each `otelcol_receiver_refused_*` counter at zero. Thus the zero values in the table are true zeros, and not data that did not arrive. Two values differ from the earlier manual measurement. `ate-controller` is now the largest source, and `ateom-gvisor` is no longer zero. The new imagecache counters from #831 are a probable cause. This difference is the reason to keep results out of the repository. ### Traces Trace volume follows the root sample decision, not the rate of each component. `atelet` and `ateom-gvisor` sent zero spans in each run, because they inherit the decision that `ateapi` makes and start no trace of their own. At idle each component is near zero: a trace needs an operation, but a metric pushes on a timer. This run gives no span volume for a usual load. The actors crashed on suspend, and the failed retries made an artificial rate of 13.7k spans/s. That rate is not a measurement of substrate. It does show that the meter accepted the spans with no refusals. ## Follow-up A separate PR adds the ladder to `automation/tests.yaml` and writes the results as artifacts of each run. The open items are at the end of `observability.md`. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Co-authored-by: Krisztian F <103492698+krisztianfekete@users.noreply.github.com>

Krisztian F (krisztianfekete) mentioned this pull request Aug 3, 2026

Define a default tracing policy #584

Closed

Max Smythe (maxsmythe) requested a review from Da Huang (git286) August 4, 2026 06:19

Max Smythe (maxsmythe) added kind/cleanup Small fixes that are not bugs, for example a typo in a code comment area/observability labels Aug 4, 2026

(otel): per-component sampler defaults, overridable via OTEL_TRACES_S…

0a38ca9

…AMPLER

Krisztian F (krisztianfekete) force-pushed the feat/set-sane-tracing-defaults branch from 1ce7df1 to 0a38ca9 Compare August 4, 2026 10:07

extend tracing policy to agw as well

cbd165c

Jeff Luo (JeffLuoo) requested changes Aug 4, 2026

View reviewed changes

Da Huang (git286) previously approved these changes Aug 4, 2026

View reviewed changes

address review: keep default on missing ratio arg, nil-safe default i…

4378769

…n warn log

Krisztian F (krisztianfekete) dismissed Da Huang (git286)’s stale review via 4378769 August 4, 2026 14:14

Krisztian F (krisztianfekete) requested review from Jeff Luo (JeffLuoo) and Da Huang (git286) August 4, 2026 14:14

Jeff Luo (JeffLuoo) approved these changes Aug 4, 2026

View reviewed changes

Eitan Yarmush (EItanya) merged commit 15eecd0 into agent-substrate:main Aug 4, 2026
11 checks passed

Krisztian F (krisztianfekete) deleted the feat/set-sane-tracing-defaults branch August 4, 2026 15:31

Jeff Luo (JeffLuoo) mentioned this pull request Aug 12, 2026

benchmarking: add observability benchmarking for the OTel collector #749

Merged

2 tasks done

Da Huang (git286) mentioned this pull request Oct 5, 2026

Router Envoy logs request credentials #2205

Closed

Navigation Menu

Repository navigation

Define a production-ready default tracing policy - #711

Define a production-ready default tracing policy#711
Eitan Yarmush (EItanya) merged 3 commits into
agent-substrate:mainfrom
krisztianfekete:feat/set-sane-tracing-defaults

Krisztian F (krisztianfekete) commented Aug 3, 2026 •

edited

Loading

Uh oh!

Max Smythe (maxsmythe) commented Aug 4, 2026 •

edited

Loading

Uh oh!

Krisztian F (krisztianfekete) commented Aug 4, 2026 •

edited

Loading

Uh oh!

Max Smythe (maxsmythe) commented Aug 4, 2026

Uh oh!

Jeff Luo (JeffLuoo) Aug 4, 2026

Uh oh!

Krisztian F (krisztianfekete) Aug 4, 2026

Uh oh!

Jeff Luo (JeffLuoo) Aug 4, 2026

Uh oh!

Krisztian F (krisztianfekete) Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Krisztian F (krisztianfekete) commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Max Smythe (maxsmythe) commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Krisztian F (krisztianfekete) commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Max Smythe (maxsmythe) commented Aug 4, 2026

Uh oh!

Jeff Luo (JeffLuoo) Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Krisztian F (krisztianfekete) Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Jeff Luo (JeffLuoo) Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Krisztian F (krisztianfekete) Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Krisztian F (krisztianfekete) commented Aug 3, 2026 •

edited

Loading

Max Smythe (maxsmythe) commented Aug 4, 2026 •

edited

Loading

Krisztian F (krisztianfekete) commented Aug 4, 2026 •

edited

Loading