Actor resource-usage events: emit from ateom over the OTLP relay #1748

Context

Actor usage records are stdout events written by atelet's stats sweep (#1206), which also feeds the template-level metrics. ateom exports traces and metrics through atelet's unix-socket OTLP relay (#809). #1658 added OTLP log export (serverboot.InitLogging, internal/actorevent, events.yaml), used by ateapi for lifecycle events. First/final usage samples were descoped from #1206 because taking them from atelet added latency to resume and suspend.

Proposal

ateom emits the usage records and sends them as OTLP logs through the relay:

1. Periodic: each ateom samples its actors on a timer and emits one ate.actor.usage_sampled record per actor.
2. Lifecycle: a initial record when the workload becomes measurable, a final record inside CheckpointWorkload.

Every record goes to stdout through slog. When the OTLP exporter is enabled, a copy goes to the collector. Sampling is always on; the stream has its own off switch, and the OTLP copy its own gate.

Design

Emission lives in ateom. ateom holds the attribution and does the measurement; atelet only relays a copy back to itself. The final sample must be taken before checkpoint teardown, which only ateom can do. With ateom sampling on its own timer, the discovery RPC serves a cache, which removes the multi-actor cost problem (#1640 item 2), and a pending record carries the activation start, so a consumer reads the pending duration off it (#1640 item 5).

Epoch. A Run or Restore starts an activation; its epoch id is the unix-nano time of activeActor.Store, on every record and on WorkloadStatsSample. Only cpu_usage_usec becomes activation-relative, with a per-source baseline: zero for cgroup (it already restarts), the first successful read for guest-agent (its counters survive restore in guest RAM, so a raw per-epoch sum would double-count). The baseline is set once by compare-and-swap; reads below it clamp to zero. Memory current and working set stay absolute; peak stays raw, since a high-water mark minus a baseline means nothing. Lifetime CPU is the sum over epochs of each epoch's highest value, from periodic records alone; the final record is a precision improvement on gVisor and an epoch-close marker on micro-VM.

atelet's charge rule. Baselines are keyed by (uid, epoch). An epoch is charged its full first value when it has no baseline and started after atelet's process start; later increases are charged; a decrease charges nothing. Baselines survive missed sweeps and are pruned when the ateom directory is gone or a later epoch appears for the uid. A zero epoch on the wire (older ateom) needs no special path. The restart gap stays what it is today.

Transport. The relay, per #809: no network path from the worker pod, one collector connection per node, clear of ateom's egress rules, and atelet outlives the pod. The direct export stays a startup fallback; today the controller injects the endpoint and the NetworkPolicy is ingress-only, so closing that path is a #809 follow-up. After a checkpoint the ateom stays up, so ForceFlush (500ms, then proceed) buys promptness, not survival; LoggerProvider.Shutdown on ateom exit covers the queue.

Spoofing. A forged state record from a worker pod fakes any actor's state, and the relay cannot tell worker pods apart. A denylist of ateapi's scope or event names would protect only today's fields. Instead: consumers trust the resource, not the record (a lifecycle record is authoritative only under ateapi's resource, and the relay already refuses every non-ateom resource), and per-pod relay sockets under ateoms/<pod-uid>/ bind a connection to a pod (#741). SO_PEERCRED is not a substitute: everything runs as uid 0.

Plan

1. Relay LogsService. Own PR. Same source gate and verbatim forward as the other signals.
   1b. Consumer rule in events.yaml and observability.md. Doc-only.
2. LoggingOptions.ExporterConn and RelayCapable. ateom passes the relay connection and defers a nil-guarded Shutdown.
3. Event definition. ate.actor.usage_sampled in actorevent (with a scope parameter) and events.yaml. All attributes via ateattr: identity, ate.workerpool.*, class, source, the four measurements, observation time, epoch, kind (periodic/initial/final). New ateattr keys for measurements, observation time, epoch, and kind. Proto: epoch field on WorkloadStatsSample; doc change on the measurements block and fields 9 and 11.
4. Periodic emission. Sampling timer, always on, default 60s; the discovery RPC serves the cache from here. Emit interval flag, 0 = no records; OTLP copy gated separately. stdout copy uses a fixed-level async handler with drop accounting (not slog.Default()). Pending actors emit every key with zero measurements under source = UNSPECIFIED. Pool labels from a fieldRef on ate.dev/worker-pool plus POD_NAMESPACE (made unconditional). ateom sets the epoch on every sample; atelet re-keys baselines in the same PR. atelet's emitter is redundant after this step.
5. Lifecycle emission. gVisor: initial after the pause container's runsc start/restore, live read; final in CheckpointWorkload, live read, then ForceFlush. micro-VM: initial on a goroutine at publish (a live read is up to 25 vsock calls, the atelet: emit per-actor usage events from the stats sweep #1206 latency); final reuses the last cached sample, pending-shaped if none. No CheckpointWorkloadResponse echo.
6. atelet stops emitting. Remove statsEventEmitter; keep the metrics channel. Update docs/observability.md.
7. atelet timeouts drop to a plain RPC timeout.

Steps 1, 1b, 2 are independent and land before 4. Step 4 follows 3. Steps 5, 6, 7 follow 4.

Volume

About 1.4M records per day at 1,000 actors, on stdout regardless of OTLP. The OTLP copy is gated off by default; the emit interval flag is the stream's off switch.

Breaking change

The stdout record changes: fields move to ate.* keys, identity and pool labels are flat attributes instead of the logging.googleapis.com/labels group, and the message becomes the event body. Filters on the old shape stop matching. Lands with step 4; old shape stops at step 6.

Out of scope

Template metrics stay in atelet (#174). Relay provenance: #741. kubectl ate top. ateapi's lifecycle stdout copy.

Related

#550 · #896 · #1206 · #809 · #741 · #1658 · #1640 (items 2 and 5) · #1266