Actor resource telemetry: adapt to multi-actor workers #1640

Context

Adapt the actor resource-telemetry stack (#550: GetWorkloadStats / GetActiveWorkloadStats on both ateom runtimes, the atelet stats sweep with template-level metrics and per-actor usage events) to multi-actor workers (#1266, closed).

The stack was built on the one-actor-per-worker state machine. Once workers host N actors, worker-pod-level metrics (metrics-server, kubectl ate top workers) stop being a usable proxy for any actor's usage. This pipeline is the only actor-granular source, so it must be multi-actor-clean. Both ateom binaries now default --max-actors to 1000; the gate is no longer "before capacity flips" but "before anyone relies on the numbers at capacity > 1".

Status

What #1836 settled

The handlers iterate a hostedActor registry guarded by its own actorsMu (RWMutex), not the lifecycle mutex, so a poll landing mid-boot or mid-checkpoint answers without waiting. Each entry is loaded, sampled, and re-checked by lookupActor(uid): an actor re-hosted underneath the read reports pending under its new attribution; one gone from the registry drops out of that response. The keyed GetWorkloadStats(actor_uid) is a registry lookup; its codes are unchanged.

On the micro-VM runtime each hostedActor owns its guest client. The discovery read fans out to at most 32 guests at once under a 45s sweep budget that stays inside atelet's 55s call deadline; a guest not reached in time reports as pending rather than failing the response. This replaces the ceil(N / cap) x 50s formula the earlier plan asked atelet to derive: the ateom bounds its own sweep, so atelet's deadline stays a constant.

Both runtimes have tests for two hosted actors, including one pending and one measured in the same response.

Remaining work

1. atelet: pin the multi-entry fold. collect iterates resp.GetSamples() and folds each entry: pending entries (source UNSPECIFIED) contribute nothing to aggregates, CPU baselines, or events; measured entries aggregate under their own template key and emit one event each. The code is correct by reading, but every test in statspoller_test.go uses single-entry responses. Add tests where one worker returns two measured actors (same template: summed; different templates: two keys; two lastCPU baselines), and one measured plus one pending (pending contributes nothing, sampled_actors counts one).

2. atelet: reconcile the timeout comments. minActorStatsPollInterval (50s) and statsRPCTimeout (55s) are still documented as derived from the single-actor worst case, maxActorContainers x statsCallTimeout. The values stay correct because the ateom now caps its own sweep at 45s regardless of N; the comments should say so, and cite statsSweepBudget as the contract atelet's 55s covers.

Both fit in one small atelet PR. Nothing else blocks calling the pipeline multi-actor-complete.

Deferred: dead-guest visibility

With one actor per worker, a dead guest fails readyz and is replaced. With N actors, one dead guest sits behind healthy siblings and shows only as a pending entry the poller skips. #1836 widened this slightly: a guest slower than the 45s sweep budget also reports pending, so "pending" now covers booting, restoring, slow, and dead.

The signal is still worth having: atelet tracks first-pending time per actor uid (same shape as lastCPU, pruned to actors seen) and emits a warning event or a long-pending-actors-per-template gauge past a threshold. It is lowered to a follow-up rather than a gate: today's occupancy is 1 everywhere, readyz still catches a dead lone guest, and the per-actor usage events already make a pending actor visible by its absence. Tracked separately when there is a consumer.

Out of scope

• kubectl ate top workers: needs no multi-actor adaptation; its CPU/MEMORY are pod-level totals, and the plural display landed with multi-actor worker API #1283.
• Actor-level usage in the CLI (top actors, live read-through via an atelet relay and ateapi fan-out): excluded by decision; per-actor usage is on the events channel and, for one target, the keyed GetWorkloadStats. The earlier design is recorded here so it is not re-derived: ateapi pages by worker, one DialForNode per node, per-worker discovery reads, "metrics unavailable" rows on node timeout, no central sample cache.
• Utilization (usage / limit): with the Feature Request: Resource-usage telemetry at actor / ActorTemplate granularity #550 phase-2 rollups.
• Per-container breakdown for gVisor (sentry accounting): unchanged, per Phase 0: StatsWorkload RPC on ateom (proto, both runtime reads, identity retention) #594.
• memory_peak_bytes over-count for multi-container actors (sum of per-container peaks): the fix is an upstream kata sandbox-level stats RPC reading the /ateomchv parent; owned upstream.
• micro-VM host cgroup keying for N cloud-hypervisor processes: the epic's sizing and isolation work (ateom microvm: drop privileged: true, using atelet device plugin for KVM #1254), not telemetry.

Related

#550 (parent) · #594 (phase 0: #667 #739 #832) · #896 (phase 1: #899 #961 #1206; open for the lifecycle-bracket follow-up, now #1748) · #1266 (the epic, closed: API #1283, runtime #1836, gVisor leaves #1667, uVM cgroup prep #1254) · #1644 (wire shape)