ateom: export telemetry through an atelet unix-socket OTLP relay by chw120 · Pull Request #809 · agent-substrate/substrate

Chenyi Wang (chw120) · 2026-08-08T01:36:15Z

ateom runs inside the worker pod that hosts the actor, and exported OTLP straight to the collector over the pod's network. This adds a node-local relay: ateom pushes OTLP/gRPC over a unix socket that atelet serves and forwards to the collector, so a worker pod needs no network path of its own to export spans and metrics.

Four things motivate it:

Blast radius. The pod runs untrusted agent code, so allowing it egress to the collector makes the collector reachable to anything that escapes the sandbox. A unix socket cannot leave the node.
Connection count. Worker pods are heavily oversubscribed; N ateoms per node each held their own collector connection. They collapse into atelet's single per-node one.
Interference. ateom transparently redirects actor egress to its own atunnel listener, and its own outbound traffic has to stay clear of the rules it installs. A unix socket is not IP traffic.
Shutdown loss. Teardown frees the actor's network and then the pod goes away, which is when the spans describing teardown are still queued in the batch processor. atelet outlives the worker pod.

The relay forwards the OTLP request verbatim rather than decoding and re-exporting, so each ateom's own resource (service.name, service.instance.id) survives instead of being absorbed into atelet's.

It is best-effort: an ateom that finds no socket at startup logs it and exports directly to OTEL_EXPORTER_OTLP_ENDPOINT as before, so this is a no-op for a cluster running an older atelet. atelet likewise declines to serve a relay when no collector is configured, since it would accept spans only to drop them. Both halves stay off with --otlp-relay-socket="".

The socket lives in ateompath.BasePath, the hostPath already mounted at the same path into atelet and into every ateom pod, so no new volume or controller change is needed.

Also includes:

End-to-end tests covering the full serverboot-to-collector path.
Observability documentation updates for Jaeger tracing.

It's a good idea to open an issue first for discussion.

Tests pass
Appropriate changes to documentation are included in the PR

ateom runs inside the worker pod that hosts the actor, and exported OTLP straight to the collector over the pod's network. This adds a node-local relay: ateom pushes OTLP/gRPC over a unix socket that atelet serves and forwards to the collector, so a worker pod needs no network path of its own to export spans and metrics. Four things motivate it: - Blast radius. The pod runs untrusted agent code, so allowing it egress to the collector makes the collector reachable to anything that escapes the sandbox. A unix socket cannot leave the node. - Connection count. Worker pods are heavily oversubscribed; N ateoms per node each held their own collector connection. They collapse into atelet's single per-node one. - Interference. ateom transparently redirects actor egress to its own atunnel listener, and its own outbound traffic has to stay clear of the rules it installs. A unix socket is not IP traffic. - Shutdown loss. Teardown frees the actor's network and then the pod goes away, which is when the spans describing teardown are still queued in the batch processor. atelet outlives the worker pod. The relay forwards the OTLP request verbatim rather than decoding and re-exporting, so each ateom's own resource (service.name, service.instance.id) survives instead of being absorbed into atelet's. It is best-effort: an ateom that finds no socket at startup logs it and exports directly to OTEL_EXPORTER_OTLP_ENDPOINT as before, so this is a no-op for a cluster running an older atelet. atelet likewise declines to serve a relay when no collector is configured, since it would accept spans only to drop them. Both halves stay off with --otlp-relay-socket="". The socket lives in ateompath.BasePath, the hostPath already mounted at the same path into atelet and into every ateom pod, so no new volume or controller change is needed. Also includes: - End-to-end tests covering the full serverboot-to-collector path. - Observability documentation updates for Jaeger tracing.

Chenyi Wang (chw120) · 2026-08-08T01:42:11Z

Benjamin Elder (@BenTheElder) FYI, this is the PR for collecting ateom traces. Please review it when you have time. Thank you.

Benjamin Elder (BenTheElder)

Human vetted, AI first pass.

Benjamin Elder (BenTheElder) · 2026-08-08T02:25:47Z

+	}
+	// A socket left behind by a previous atelet would make Listen fail with
+	// EADDRINUSE even though nothing holds it.
+	if err := os.RemoveAll(s.sockPath); err != nil {


🤖 should-fix 🟡 – RemoveAll is a lot more than this needs, and the blast radius on a misconfigured path is the node's whole ateom state. The comment explains the goal as clearing a stale socket so Listen does not hit EADDRINUSE — os.Remove does exactly that and fails on a directory, which is the right answer. RemoveAll instead succeeds: --otlp-relay-socket=/var/lib/ateom-gvisor (the BasePath itself, an easy slip since the socket lives under it) would have atelet silently delete the image cache, every actor bundle, the staged runsc binaries and the local checkpoints, at startup, before anything else runs.

Stop already uses os.Remove on the same path, so the two ends of the socket's lifecycle disagree about which call is appropriate.

Thanks Benjamin Elder (@BenTheElder) for review. Updated the code to use os.Remove.

Benjamin Elder (BenTheElder) · 2026-08-08T02:25:47Z

+	// gRPC resolves a "unix://" target to a unix socket dialer natively, so the
+	// OTLP exporters above this connection are unchanged: OTLP is gRPC, and gRPC
+	// needs only a reliable byte stream.
+	conn, err := grpc.NewClient("unix://"+sockPath,


🤖 nit 🟢 – This only builds a valid target for an absolute sockPath. With a relative one, unix://otlp.sock parses otlp.sock as the authority with an empty path, so the dial fails — and it fails lazily, at first export, which is precisely the outcome the os.Stat check above exists to avoid. Stat would have passed, since a relative path resolves fine against the working directory, so the deterministic startup fallback is bypassed and the telemetry is quietly lost instead.

grpc.NewClient("unix:"+sockPath, ...) — the single-colon form — takes everything after the colon as the path and handles both shapes, or the flag could reject a non-absolute path up front.

Added the validation.

Krisztian F (krisztianfekete)

I'd hold off on this. It's the ateom part of the same relay we're designing in #761. That relay has to rewrite telemetry identity, and it needs a network listener regardless since a unix socket won't work with microvm guests.

Can we design that end first and let ateom telemetry leverage that, I am fine with the socket itself, but let's not codify stuff we are about to undo. cc. Da Huang (@git286)

Da Huang (git286) · 2026-08-10T14:39:21Z

I'd hold off on this. It's the ateom part of the same relay we're designing in #761. That relay has to rewrite telemetry identity, and it needs a network listener regardless since a unix socket won't work with microvm guests.

Can we design that end first and let ateom telemetry leverage that, I am fine with the socket itself, but let's not codify stuff we are about to undo. cc. Da Huang (Da Huang (@git286))

Krisztian F (@krisztianfekete) I think these are two different hops.

#761 is actor → ateom/atelet: untrusted client inside the sandbox, identity has to be rewritten, and it needs a network listener because a unix socket is invisible to a micro-VM guest. This PR is ateom → atelet: ateom is a normal container in the worker pod with the BasePath hostPath mounted, whatever runtime it drives, and its identity is already correct. The socket will work for ateom -> atelet no matter it's gvisor or microvm, right?

Krisztian F (krisztianfekete) · 2026-08-10T15:17:32Z

I'd hold off on this. It's the ateom part of the same relay we're designing in #761. That relay has to rewrite telemetry identity, and it needs a network listener regardless since a unix socket won't work with microvm guests.
Can we design that end first and let ateom telemetry leverage that, I am fine with the socket itself, but let's not codify stuff we are about to undo. cc. Da Huang (Da Huang (Da Huang (@git286)))

Krisztian F (Krisztian F (@krisztianfekete)) I think these are two different hops.

#761 is actor → ateom/atelet: untrusted client inside the sandbox, identity has to be rewritten, and it needs a network listener because a unix socket is invisible to a micro-VM guest. This PR is ateom → atelet: ateom is a normal container in the worker pod with the BasePath hostPath mounted, whatever runtime it drives, and its identity is already correct. The socket will work for ateom -> atelet no matter it's gvisor or microvm, right?

You're right, I take back the microvm concern!

The one thing I'd still ask for is scoping. Forwarding as-is is the correct behaviour for ateom telemetry, but right now that covers relay itself rather than about this particular traffic. Whether #761 rewrites at ateom or at atelet, this relay is on this path too, so it will eventually has traffic that has to be modified alongside traffic that must not be. Could we limit this to ateom source now, so adding the rewriting path later is an extension rather than a redoing it?

internal/otlprelay imports the OTLP collector protos directly, but go.mod still carried the module as indirect: go mod tidy was never rerun after the relay landed, so hack/verify/go-modules.sh regenerates, finds a diff and fails run-tests. Nothing else moves. vendor/modules.txt already marked the module explicit, which is why every local build and test stayed green and only CI caught it.

Addresses review on agent-substrate#809: Krisztian's scoping request and Benjamin's two bugs. Scope the relay to ateom sources. Forwarding a request untouched is not a property of relaying in general; it is correct only for a source whose resource is already correct. ateom's is, so the relay now says so and refuses everything else with PermissionDenied, a resource declaring no service.name included. The case this excludes is actor telemetry: actors share a hostname ("runsc") and an interior IP, so their series merge unless identity is injected from outside the actor (agent-substrate#761), and rewriting in flight is the negation of pass-through. The two do not share this socket today -- agent-substrate#761 is actor->ateom and this relay is ateom->atelet -- but naming the contract while it still holds makes that path an added branch later rather than a re-argument about whether pass-through was ever safe. A batch is refused whole rather than having the offending resource dropped, since a partial success the sender reads as success loses telemetry silently. The allowlist duplicates the serviceName constants in cmd/ateom-*, which are package main and cannot be imported, so a test reads them back out of the source: a typo there would otherwise refuse every real ateom export while the package's own tests, sharing the typo, kept passing. Remove the stale socket with os.Remove rather than os.RemoveAll. The path comes from a flag and lives in BasePath, the host directory that also holds every ateom's own socket, so a value naming the directory would have had RemoveAll empty it. os.Remove refuses a populated directory, turning that into a startup error instead of data loss. Reject a relative socket path in both halves. gRPC does not resolve one: "unix://foo/r.sock" parses as authority "foo" and path "/r.sock". Because grpc.NewClient is lazy, that wrong target was accepted at startup and failed per export afterwards with the spans already gone, defeating the existence check in Dial that makes the direct-export fallback a startup decision. atelet and ateom would also resolve a relative path against different working directories, so it cannot name one socket for both halves. An absent socket still falls back quietly; only a malformed path errors.

Chenyi Wang (chw120) · 2026-08-11T00:01:19Z

I'd hold off on this. It's the ateom part of the same relay we're designing in #761. That relay has to rewrite telemetry identity, and it needs a network listener regardless since a unix socket won't work with microvm guests.
Can we design that end first and let ateom telemetry leverage that, I am fine with the socket itself, but let's not codify stuff we are about to undo. cc. Da Huang (Da Huang (Da Huang (Da Huang (@git286))))

Krisztian F (Krisztian F (Krisztian F (@krisztianfekete))) I think these are two different hops.
#761 is actor → ateom/atelet: untrusted client inside the sandbox, identity has to be rewritten, and it needs a network listener because a unix socket is invisible to a micro-VM guest. This PR is ateom → atelet: ateom is a normal container in the worker pod with the BasePath hostPath mounted, whatever runtime it drives, and its identity is already correct. The socket will work for ateom -> atelet no matter it's gvisor or microvm, right?

You're right, I take back the microvm concern!

The one thing I'd still ask for is scoping. Forwarding as-is is the correct behaviour for ateom telemetry, but right now that covers relay itself rather than about this particular traffic. Whether #761 rewrites at ateom or at atelet, this relay is on this path too, so it will eventually has traffic that has to be modified alongside traffic that must not be. Could we limit this to ateom source now, so adding the rewriting path later is an extension rather than a redoing it?

Thanks Krisztian F (@krisztianfekete) Da Huang (@git286) for review. Updated the code to scope it to only allow ateom sources, "ateom-gvisor" and "ateom-microvm".

Tim Bai (baizhenyu)

The overall change looks good to me, left some minor comments

Krisztian F (krisztianfekete) · 2026-08-11T15:28:32Z

+	// directly for its whole life, so the socket should exist before any worker
+	// pod on this node boots.
+	if relay, err := otlprelay.NewServer(ctx, *otlpRelaySocket); err != nil {
+		serverboot.Fatal(ctx, "Failed to create the OTLP relay", err)


Are we sure about this?
Everything else here degrades nicely, so Fatal might be too much. What do you think?

Replaced serverboot.Fatal with an error log.

Krisztian F (krisztianfekete) · 2026-08-11T15:33:32Z

+	// socketMode keeps the relay socket reachable by the ateom pods on the node
+	// (which do not necessarily share atelet's uid) while staying off-node by
+	// construction. The socket is inside BasePath, a root-owned host directory.
+	socketMode = 0o666


We get root from the controller, aren't we?
Would 0600 be enough here?

You're right. Updated.

Krisztian F (krisztianfekete) · 2026-08-11T15:39:43Z

+// The relay forwards the OTLP request message verbatim rather than decoding it
+// into SDK records and re-exporting. Pass-through keeps each ateom's own
+// resource (service.name, service.instance.id, pod attributes) intact, so its
+// spans stay attributed to ateom instead of being absorbed into atelet's. That
+// holds only for a source whose resource is already right, so the relay carries
+// ateom telemetry only; see ateomServices.


Could we word the guarantee as being about the ateom source, so the rewriting path is an extension later rather than a contradiction?

Update the description.

Krisztian F (krisztianfekete) · 2026-08-11T15:42:08Z


+### The ateom OTLP relay
+
+ateom is the one component that does not talk to the collector itself. It exports over a unix socket at `/var/lib/ateom-gvisor/atelet-otlp.sock`, which `atelet` serves and forwards to the collector on the node's network ([`internal/otlprelay`](../internal/otlprelay)):


As per this comment: #761 (comment), by Da Huang (@git286) the GKE pipelines needs checking as after this change ateom telemetry will be passing trough atelet connections, so k8sattributes might break.

Checked the GKE pipeline and Downward API config:

ate-controller already injects k8s.namespace.name, k8s.pod.name, k8s.pod.uid, and service.instance.id directly into OTEL_RESOURCE_ATTRIBUTES via the Kubernetes Downward API on every worker pod (see workerpool_apply.go:31).

Because the relay forwards OTLP requests and resources verbatim without re-encoding, these k8s.* resource attributes arrive at the collector intact directly on the payload, without relying on collector-side connection IP-based k8sattributes association. Added a clarifying note about this in docs/observability.md.

Krisztian F (krisztianfekete) · 2026-08-11T15:45:44Z

+		if specific == "" {
+			continue
+		}
+		if resolved != "" && resolved != generic && specific != resolved {


resolved != generic is false on the first non-empty pass, so the guard can only ever fire when both signal-specific vars are set.

Added more checks.

Krisztian F (krisztianfekete) · 2026-08-11T15:48:26Z

+			return nil, err
+		}
+	}
+	return t.upstream.Export(ctx, req)


gRPC won't pass the incoming metadata along to the upstream call, so if an ateom has OTEL_EXPORTER_OTLP_HEADERS set the headers will be dropped so we cannot do auth I guess?

Same applies for metrics below as well.

Added forwardContext() to extract incoming metadata and propagate it as outgoing metadata on the upstream call.

I've been thinking more on this, maybe it's better the other way round and we should read OTEL_EXPORTER_OTLP_HEADERS from atelet's env and attach those, and drop or allow-list the client's. Fixes the case I raised without making the rely forward anything problematic. What do you think?

Krisztian F (krisztianfekete) · 2026-08-11T15:50:51Z

+	// Lazy by design: grpc.NewClient does not block on the collector being up,
+	// so atelet startup does not depend on the collector's readiness.


What happens when the collector is down?

Can we set compression here?

grpc.NewClient is non-blocking. If the collector is down, transport errors (Unavailable) propagate over UDS to ateom's OTel SDK batch processor to retry/drop asynchronously without blocking actor execution; gRPC reconnects automatically once it recovers.

Added upstream gzip compression support via OTEL_EXPORTER_OTLP_COMPRESSION (and signal-specific overrides).

…ack, and permissions - Forward incoming gRPC metadata (headers, auth tokens) in trace and metric relays to upstream - Add upstream compression support (gzip / none) - Fix signal-specific and generic OTLP endpoint conflict resolution in upstreamTarget - Restrict socket mode to 0600 - Add substrate.otlp.relay resource attribute in serverboot for fleet-wide fallback observability - Make NewServer failure non-fatal in atelet - Ensure test isolation in e2e test with unroutable generic endpoint - Update documentation on ateom pass-through, k8sattributes preservation, and Phase 2 egress lockdown

Krisztian F (krisztianfekete) · 2026-08-12T08:43:41Z

+	if opts.ExporterConn != nil {
+		relayStatus = "relay"
+	}
+	res, err := newResource(ctx, opts.ServiceName, attribute.String("substrate.otlp.relay", relayStatus))


Nice! Two things on the name though:

our custom attributes live under ate.* and nothing uses substrate.*, so use ate.otlp.relay please.

since newResource is shared, every component gets this, so ateapi, atelet, atecontroller and atenet all set direct forever for a relay they'll never use. Worth setting it only on the ateom path if we can.

Thanks Krisztian F (@krisztianfekete) for review.

Updated, and only ateom-gvisor and ateom-microvm will have this attribute.

Krisztian F (krisztianfekete) · 2026-08-12T08:46:19Z

+func TestResourceRelayAttribute(t *testing.T) {
+	resDirect, err := newResource(context.Background(), "ateom-gvisor", attribute.String("substrate.otlp.relay", "direct"))
+	if err != nil {
+		t.Fatalf("newResource: %v", err)
+	}
+	if got := resourceAttrs(resDirect)["substrate.otlp.relay"]; got != "direct" {
+		t.Errorf("substrate.otlp.relay = %q, want direct", got)
+	}
+
+	resRelay, err := newResource(context.Background(), "ateom-gvisor", attribute.String("substrate.otlp.relay", "relay"))
+	if err != nil {
+		t.Fatalf("newResource: %v", err)
+	}
+	if got := resourceAttrs(resRelay)["substrate.otlp.relay"]; got != "relay" {
+		t.Errorf("substrate.otlp.relay = %q, want relay", got)
+	}
+}


This isn't really testing resource.New, and it would still pass if InitTracing stopped setting it, right?
Can we cover the the nil conn vs non-nil conn decision, which only the otlprelay e2e covers today, and only for relay?

Updated the tests.

Krisztian F (krisztianfekete) · 2026-08-12T08:47:58Z

+	// Export through atelet's node-local relay when it is there, so telemetry
+	// never touches the worker pod's network. A nil conn means it is not, and
+	// both providers fall back to dialing the collector directly.
+	relayConn, err := otlprelay.Dial(ctx, *otlpRelaySocket)
+	if err != nil {
+		serverboot.Fatal(ctx, "Failed to connect to the OTLP relay", err)
+	}
+	if relayConn != nil {
+		defer relayConn.Close()
+	}


Can we do the same we have now for atelet to avoid crashes?

Sure, changed to log.

Krisztian F (krisztianfekete) · 2026-08-12T08:48:13Z

+	// Export through atelet's node-local relay when it is there, so telemetry
+	// never touches the worker pod's network. A nil conn means it is not, and
+	// both providers fall back to dialing the collector directly.
+	relayConn, err := otlprelay.Dial(ctx, *otlpRelaySocket)
+	if err != nil {
+		serverboot.Fatal(ctx, "Failed to connect to the OTLP relay", err)
+	}
+	if relayConn != nil {
+		defer relayConn.Close()
+	}
+


See https://github.com/agent-substrate/substrate/pull/809/changes#r3764908115

Krisztian F (krisztianfekete) · 2026-08-12T08:52:33Z

+// checkAteomSource rejects a missing service.name along with an unrecognized
+// one: an unidentified source is the one the relay cannot vouch for.
+func checkAteomSource(r *resourcepb.Resource) error {
+	name := resourceServiceName(r)
+	if ateomServices[name] {
+		return nil
+	}
+	return status.Errorf(codes.PermissionDenied,
+		"the OTLP relay carries ateom telemetry only, got service.name %q; a source whose identity has to be rewritten (#761) must not be forwarded verbatim",
+		name)
+}


Note that OTel env vars can override these, so if someone sets this on an ateom pod every export gets PermissionDenied, which the SDK doesn't retry. Probably fine though?

Good concern. Added a warning.

Krisztian F (krisztianfekete) · 2026-08-12T08:56:17Z

+// It sits directly under BasePath, which is the host directory already mounted
+// at the same path into atelet and into every ateom pod, so no new volume is
+// needed for ateom to reach it.
+func AteletOTLPSocketPath() string {
+	return filepath.Join(
+		BasePath,
+		"atelet-otlp.sock",
+	)
+}
+


Wondering if a subdir only atelet writes, mounted read-only into the pods, would be better here?

It would be better, but I think it would be better to have another PR updating it, since it isn't specific to this socket. Instead, added more comments. WDYT?

Fair, could you please open an issue for it rather than leaving it as a comment though so we can track this?

Good point! This is part of the broader host socket/filesystem isolation discussion already tracked in #778 (and #853). We can address isolating atelet-owned sockets into dedicated read-only mounts as part of that work.

Probably a followup improvement: we need more protection in case actor escape. For example, bounded the memory/message size, limit the max concurrent streams etc.

…ay attribute to ateom Addresses Krisztian's second round on agent-substrate#809. Attach atelet's own OTEL_EXPORTER_OTLP_HEADERS upstream and drop the client's. The previous commit forwarded the incoming metadata verbatim, which answered the question it was asked -- an ateom with headers set could authenticate -- by handing the answer to the wrong party. The upstream leg is atelet's connection to the collector, so its credentials are atelet's, exactly as they were for the SDK exporter the relay stands in for. Forwarding let anything that reached the socket choose what atelet presents upstream, and a worker pod runs untrusted code. This is not the same trade-off as the resource: a claimed service.name that is wrong misfiles a series, while a header is an actual credential. Verbatim now describes the payload, not the call around it. Headers are held per signal rather than per connection, since unlike the endpoint and the compression they are per-call metadata and TRACES_/METRICS_ may legitimately differ; per the OTLP spec the signal-specific variable replaces the generic one rather than merging. An unparseable value fails NewServer instead of becoming a per-export rejection against a collector that refuses unauthenticated calls, and ateom then finds no socket and exports directly. Nothing is allow-listed through: atecontroller injects only OTEL_EXPORTER_OTLP_ENDPOINT into worker pods, so no ateom has a header to lose, and an allowlist can be added when one does. Parse errors name the offending entry by position and the log names the header keys only, because the values are credentials. Report the first rejection of each service.name. resource.WithFromEnv() runs last in newResource, so OTEL_SERVICE_NAME or an OTEL_RESOURCE_ATTRIBUTES entry on a worker pod overrides the ateom's own name, the allowlist stops recognizing it, and every export is refused with PermissionDenied -- which the SDK does not retry. The ateom keeps running and its telemetry silently stops, with nothing on the collector side to distinguish that from an idle node. The refusal is still correct; what was missing was anyone saying so, since the whole diagnosis lives on the relay side and was being discarded. Both the log and the gRPC error now name the two environment variables that produce this, so it is reachable from the node and from the ateom's own error handler. Dedup is keyed by name because a refused exporter retries on its own schedule for the pod's whole life, and a node runs many of them: unbounded logging would be the worse failure. Traces and metrics share one gate so a misnamed ateom is reported once rather than per signal. Rename substrate.otlp.relay to ate.otlp.relay and set it only where it means something. Custom attributes live under ate.*, and newResource is shared, so every component was carrying "direct" forever for a relay it will never be offered. Capability and connection are now separate inputs: nil alone cannot distinguish an ateom that tried and fell back from an atecontroller that never had a relay, and the fallback is precisely the state the attribute was added to make observable -- collapsing the two would hide a node whose atelet never came up. Only the ateoms set RelayCapable; calling InitMetricsPushOnlyVia implies it, since only a caller holding a relay reaches for that function. The key is spelled locally rather than taken from internal/ateattr: serverboot sits at the bottom of the dependency graph with one agent-substrate import and every main links it, while ateattr would pull in pkg/api/v1alpha1, ateapipb, internal/resources and ateletpb. Worth revisiting if that cost drops. Degrade instead of exiting when the relay cannot be dialed, matching atelet. Both ends of this decision already agreed and the ateoms disagreed: Dial treats an absent socket as a fallback rather than an error, and atelet logs and keeps going when it cannot serve the relay at all. What a failed dial loses is the node-local export path, not the ability to run actors, so failing the worker pod over its telemetry route would turn a misconfigured flag into an outage. Test the decision rather than the plumbing. The relay attribute test asserted what it had just passed in and would have kept passing if InitTracing stopped setting the attribute; it now covers all four combinations of capability and connection, two of which assert the attribute is absent, and reads the metric resource back off the provider through a ManualReader so a wiring mistake in newMeterProvider fails there and not in the unit test beside it. The e2e gains the direct half it never had: no socket, nil conn, and the span still reaches the collector carrying "direct". The metadata test asserted the behaviour this commit removes and is replaced by its inverse plus one for the attached headers. Record why the socket is not in a read-only subdirectory. It would be an improvement, but worker pods mount BasePath itself writable, so it needs its own volume and a controller change -- against this PR's premise of reusing the existing mount -- and the pods keep CAP_SYS_ADMIN, so it guards against mistakes rather than malice. It is also not specific to this socket: CredentialBrokerSocket and the image cache share the same writable directory, so the fix belongs to the mount, not to the newest thing under it.

Krisztian F (krisztianfekete) · 2026-08-13T09:48:28Z

+		}
+		// Percent-decoding is what makes a value containing "," or "=" (a base64
+		// token, say) expressible in this format at all.
+		decoded, err := url.QueryUnescape(strings.TrimSpace(value))


Can we use url.PathUnescape here?

Krisztian F (krisztianfekete) · 2026-08-13T09:51:47Z

+	if _, dup := g.logged.LoadOrStore(name, struct{}{}); !dup {
+		slog.WarnContext(ctx, "OTLP relay rejected telemetry from an unrecognized source; it is being dropped, not retried. If this is an ateom, check whether OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES on the worker pod is overriding its service.name",
+			slog.String("service.name", name),
+			slog.Any("allowed", allowedServices()),
+			slog.String("note", "logged once per distinct service.name"))
+	}


service.name is client-side, so not really bounded. Can we adopt the lru.New(256) pattern from atelet?

Krisztian F (krisztianfekete) · 2026-08-13T09:52:37Z

+
+The socket sits in the `BasePath` hostPath already mounted into both, so nothing new is mounted. `atelet` is a DaemonSet, so every ateom on a node shares one relay, and the many per-pod collector connections collapse into one per node. Four things motivate it: the worker pod runs untrusted agent code and will not need egress to the collector once direct fallback is phased out; the connection count drops; ateom's own telemetry stays clear of the transparent egress redirect it installs for the actor; and `atelet` outlives the worker pod, so spans still queued at teardown are not lost with it.
+
+The relay is best-effort. If the socket is absent when ateom starts — `atelet` not up yet, `--otlp-relay-socket=""`, or no collector configured for the relay to forward to — ateom logs it and exports directly to `OTEL_EXPORTER_OTLP_ENDPOINT` as before. That fallback is decided once at startup, not per export, and is stamped on telemetry as the `substrate.otlp.relay` resource attribute (`relay` vs `direct`).


Suggested change

The relay is best-effort. If the socket is absent when ateom starts — `atelet` not up yet, `--otlp-relay-socket=""`, or no collector configured for the relay to forward to — ateom logs it and exports directly to `OTEL_EXPORTER_OTLP_ENDPOINT` as before. That fallback is decided once at startup, not per export, and is stamped on telemetry as the `substrate.otlp.relay` resource attribute (`relay` vs `direct`).

The relay is best-effort. If the socket is absent when ateom starts — `atelet` not up yet, `--otlp-relay-socket=""`, or no collector configured for the relay to forward to — ateom logs it and exports directly to `OTEL_EXPORTER_OTLP_ENDPOINT` as before. That fallback is decided once at startup, not per export, and is stamped on telemetry as the `ate.otlp.relay` resource attribute (`relay` vs `direct`).

Krisztian F (krisztianfekete) · 2026-08-13T09:54:04Z

+// It sits directly under BasePath, which is the host directory already mounted
+// at the same path into atelet and into every ateom pod, so no new volume is
+// needed for ateom to reach it.
+func AteletOTLPSocketPath() string {
+	return filepath.Join(
+		BasePath,
+		"atelet-otlp.sock",
+	)
+}
+


Fair, could you please open an issue for it rather than leaving it as a comment though so we can track this?

…nescape for headers - Use url.PathUnescape instead of url.QueryUnescape when parsing OTLP headers so '+' characters (such as in Base64 tokens) are preserved rather than converted to spaces. - Replace sync.Map in sourceGate with k8s.io/utils/lru (bounded to 256 entries) to prevent unbounded memory growth from client-provided service names. - Update docs/observability.md to reference the ate.otlp.relay attribute.

Krisztian F (krisztianfekete)

LGTM!

Da Huang (git286) · 2026-08-14T14:54:05Z

-4. **Search and Inspect**: Copy the printed Trace ID from the CLI output and paste it into the Jaeger search box (top right), or select `ateapi` or `atelet` under the **Service** dropdown and click **Find Traces** to inspect detailed call stacks, DB transactions, state updates, and worker pod handoffs.
+4. **Search and Inspect**: Copy the printed Trace ID from the CLI output and paste it into the Jaeger search box (top right), or select `ateapi`, `atelet`, or `ateom-gvisor` under the **Service** dropdown and click **Find Traces** to inspect detailed call stacks, DB transactions, state updates, and worker pod handoffs.
+
+> ateom carries no manual spans — its only instrumentation is the `otelgrpc` interceptor on the gRPC surface `atelet` calls. So it produces a span for an actor lifecycle operation (`suspend`, `resume`) and nothing at all for a read like `kubectl ate get actor`. Its sampler is parent based, so a lifecycle command is traced end to end into ateom whenever `ateapi` roots a sampled trace, which the kind overlay makes unconditional; the per-component ratio never enters into it. `ateom-gvisor` appearing here at all is also the sign its [OTLP relay](#the-ateom-otlp-relay) is working, since that is the only path its telemetry has out of the worker pod.


ateom-gvisor appearing here at all is also the sign its OTLP relay is working, since that is the only path its telemetry has out of the worker pod.

Is it true? When the socket fails, the trace will still fallback to the network endpoint, right? Why trace show up means the relay is working?

Good catch. You're right. Updated the doc.

Benjamin Elder (BenTheElder) reviewed Aug 8, 2026

View reviewed changes

Max Smythe (maxsmythe) added the area/observability label Aug 8, 2026

Julian Gutierrez Oschmann (juli4n) requested review from Jeff Luo (JeffLuoo) and Da Huang (git286) August 8, 2026 15:14

Da Huang (git286) requested review from Tim Bai (baizhenyu) and removed request for Jeff Luo (JeffLuoo) August 10, 2026 13:06

Krisztian F (krisztianfekete) requested changes Aug 10, 2026

View reviewed changes

Benjamin Elder (BenTheElder) assigned Benjamin Elder (BenTheElder) and Bowei Du (bowei) Aug 10, 2026

Chenyi Wang (chw120) added 2 commits August 10, 2026 16:43

Tim Bai (baizhenyu) reviewed Aug 11, 2026

View reviewed changes

Comment thread internal/otlprelay/relay.go Resolved

Comment thread internal/otlprelay/client.go Resolved

Comment thread internal/otlprelay/relay.go Resolved

Comment thread internal/otlprelay/e2e_test.go Resolved

Da Huang (git286) mentioned this pull request Aug 11, 2026

[Design] Actor Otel Telemetry Relay #853

Open

Krisztian F (krisztianfekete) reviewed Aug 11, 2026

View reviewed changes

Krisztian F (krisztianfekete) reviewed Aug 12, 2026

View reviewed changes

Chenyi Wang (chw120) added 2 commits August 12, 2026 14:49

serverboot: fix typo in comment to satisfy misspell linter

20931cc

Krisztian F (krisztianfekete) reviewed Aug 13, 2026

View reviewed changes

Krisztian F (krisztianfekete) approved these changes Aug 14, 2026

View reviewed changes

Da Huang (git286) reviewed Aug 14, 2026

View reviewed changes

docs: clarify OTLP relay verification via ate.otlp.relay attribute

afd8353

Da Huang (git286) approved these changes Aug 14, 2026

View reviewed changes

Benjamin Elder (BenTheElder) merged commit c178f5b into agent-substrate:main Aug 14, 2026
11 checks passed

Tim Bai (baizhenyu) mentioned this pull request Sep 18, 2026

Actor resource-usage events: emit from ateom over the OTLP relay #1748

Closed

Krisztian F (krisztianfekete) mentioned this pull request Sep 24, 2026

serverboot: give the ateoms a LoggerProvider over the relay #1857

Merged


		### The ateom OTLP relay

		ateom is the one component that does not talk to the collector itself. It exports over a unix socket at `/var/lib/ateom-gvisor/atelet-otlp.sock`, which `atelet` serves and forwards to the collector on the node's network ([`internal/otlprelay`](../internal/otlprelay)):

		// Lazy by design: grpc.NewClient does not block on the collector being up,
		// so atelet startup does not depend on the collector's readiness.


		The socket sits in the `BasePath` hostPath already mounted into both, so nothing new is mounted. `atelet` is a DaemonSet, so every ateom on a node shares one relay, and the many per-pod collector connections collapse into one per node. Four things motivate it: the worker pod runs untrusted agent code and will not need egress to the collector once direct fallback is phased out; the connection count drops; ateom's own telemetry stays clear of the transparent egress redirect it installs for the actor; and `atelet` outlives the worker pod, so spans still queued at teardown are not lost with it.

		The relay is best-effort. If the socket is absent when ateom starts — `atelet` not up yet, `--otlp-relay-socket=""`, or no collector configured for the relay to forward to — ateom logs it and exports directly to `OTEL_EXPORTER_OTLP_ENDPOINT` as before. That fallback is decided once at startup, not per export, and is stamped on telemetry as the `substrate.otlp.relay` resource attribute (`relay` vs `direct`).

Conversation

Chenyi Wang (chw120) commented Aug 8, 2026

Uh oh!

Chenyi Wang (chw120) commented Aug 8, 2026

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Krisztian F (krisztianfekete) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Da Huang (git286) commented Aug 10, 2026

Uh oh!

Krisztian F (krisztianfekete) commented Aug 10, 2026

Uh oh!

Chenyi Wang (chw120) commented Aug 11, 2026

Uh oh!

Tim Bai (baizhenyu) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Krisztian F (krisztianfekete) left a comment •

edited

Loading