Repository navigation
Configure Envoy tracing from OTEL_EXPORTER_OTLP_ENDPOINT - #637
Merged
Jeff Luo (JeffLuoo) merged 3 commits intoJul 31, 2026
Merged
Jeff Luo (JeffLuoo) merged 3 commits into
Jeff Luo (JeffLuoo) merged 3 commits into
Conversation
Jeff Luo (JeffLuoo)
changed the title
Jeff Luo (JeffLuoo)
force-pushed
the
fix/kind-otel-endpoint-atenet-router
branch
from
July 30, 2026 18:02
0e9e478 to
4f6186c
Compare
Jeff Luo (JeffLuoo)
requested review from
Eitan Yarmush (EItanya) and
Julian Gutierrez Oschmann (juli4n)
July 30, 2026 18:05
Eitan Yarmush (EItanya)
left a comment
Eitan Yarmush (EItanya)
left a comment
Collaborator
There was a problem hiding this comment.
LGTM but will leave to Julian Gutierrez Oschmann (@juli4n) for final review
Jeff Luo (JeffLuoo)
added a commit
to JeffLuoo/substrate
that referenced
this pull request
Jul 30, 2026
- Qualify the default endpoint as GKE-specific rather than universal. - Redact the cluster IP from the sample kubectl output. - Correct the connection-pinning claim. GKE's collector sets max_connection_age: 10m with the comment "Force connection recycling to balance load among the backends", so pinning is bounded and self-correcting, not permanent. Reworded to say what actually remains: rebalancing is periodic, so a burst inside a window still lands unevenly. - Add "Why the Deployment Suits Substrate". Telemetry volume tracks the active workload, which is what an HPA measures, whereas a DaemonSet sizes itself to node count and keeps paying after a burst drains. It also stops reserving CPU and memory on the sandbox nodes provisioned for actors, and stays out of the cluster autoscaler's way. - Add a filelog receiver, hostPath mounts, and a logs pipeline to the DaemonSet. Node-local log collection is the one thing a gateway categorically cannot do, so it is now the headline reason to choose one. - Adopt managed OTel's config practices: per-signal batch sizes, memory limiter spike percentage, layered pod_association, deployment name from replicaset, running-pod filter, gRPC keepalive, and health_check. Add a self-metrics pipeline. Table of the GCP-specific processors deliberately left out, and a note that GKE spells the processor k8s_attributes while upstream is k8sattributes. - Update the endpoint instructions for agent-substrate#637: atenet-router's --otlp-collector-address now defaults to OTEL_EXPORTER_OTLP_ENDPOINT. - Frame the TLS gap as a TODO to align with the pod-certificate mTLS every other in-cluster hop already uses, rather than a settled constraint. Part of agent-substrate#563.
Jeff Luo (JeffLuoo)
requested review from
Eitan Yarmush (EItanya) and
Krisztian F (krisztianfekete)
July 31, 2026 14:37
Krisztian F (krisztianfekete)
approved these changes
Jul 31, 2026
1 of 2 tasks
The kind overlay patched OTEL_EXPORTER_OTLP_ENDPOINT for ate-api-server, ate-controller and atelet, but not atenet-router. The router kept the base manifest's gke-managed-otel address for both its own OTLP exporter and the --otlp-collector-address arg that configures Envoy tracing. The gke-managed-otel namespace does not exist on kind, which only deploys a collector into otel-system, so the router's spans and metrics silently failed to export and Envoy's tracing cluster pointed at an unresolvable address. Envoy's collector address is a container arg, and args has no strategic-merge key, so a merge patch would replace the whole list. Patch that element positionally with a JSON6902 test guard so the build fails loudly if the base arg order changes.
…TLP_ENDPOINT Envoy receives its whole configuration over xDS and never reads the router's environment, so the collector address for its tracer cluster had to be relayed through --otlp-collector-address. That left atenet-router with two settings for one collector -- the env var for its own spans, an arg for Envoy's -- and every overlay had to patch both. The kind overlay patched only the env var, leaving Envoy pointed at the gke-managed-otel namespace that does not exist on kind. Default the flag to OTEL_EXPORTER_OTLP_ENDPOINT, the same variable ate-apiserver and atelet already honor, and normalize the URL form it carries down to the bare host and port an xDS SocketAddress accepts. Envoy's collector now follows the pod's env var, so the base manifest arg and the kind overlay's positional JSON patch both go away. Dropping the base arg shifts --ateapi-client-cert from args index 13 to 12, so the token-client and kind-token-client overlays, which remove it positionally, move with it. https is rejected rather than silently downgraded: the tracer cluster is plaintext h2c, so honoring it would ship spans in the clear to an endpoint that asked for TLS. An explicit empty flag still disables Envoy-side tracing while leaving the router's own spans enabled.
--otlp-collector-address now defaults to OTEL_EXPORTER_OTLP_ENDPOINT, so an address Envoy's tracer cannot use no longer arrives only from a deliberate flag. Rejecting it returned an error from Run, which cobra turns into os.Exit(1): the container exits before serving and CrashLoopBackOffs, taking the xDS control plane for every Envoy in the mesh with it. An https collector is the case that matters. It is valid config for the variable's other reader -- serverboot's exporter dials with WithInsecure and never inspects the scheme -- so a cluster that sets it would find the router refusing to start over an endpoint that works fine for the router's own spans. No tracing setting is worth the data plane. Warn and run without Envoy-side tracing instead. setOtlpCollector returns nothing at all, so no future caller can reintroduce the startup failure, and it clears any previously configured collector rather than leaving a stale one behind. Spans are still never downgraded to plaintext, which was the point of rejecting https in the first place; Envoy simply reports none.
Jeff Luo (JeffLuoo)
force-pushed
the
fix/kind-otel-endpoint-atenet-router
branch
from
July 31, 2026 16:57
bf1df24 to
fead1ed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--otlp-collector-addressdefaults toos.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT"), unify all stacks to use environment variables.#563