Skip to content

atelet: pin the multi-actor stats fold and keep CPU baselines across pending sweeps - #1885

Merged
Jeff Luo (JeffLuoo) merged 1 commit into
agent-substrate:mainfrom
baizhenyu:atelet-multi-entry-fold
Sep 28, 2026
Merged

Jeff Luo (JeffLuoo) merged 1 commit into
agent-substrate:mainfrom
baizhenyu:atelet-multi-entry-fold

Conversation

@baizhenyu

@baizhenyu Tim Bai (baizhenyu) commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Closes the remaining items in #1640: the atelet side of multi-actor telemetry.

Since #1836 an ateom returns one GetActiveWorkloadStats entry per hosted actor. atelet's sweep already folded every entry, but nothing pinned the behavior at occupancy above one, and reviewing that fold surfaced three CPU-accounting defects: two in how baselines survive a pending sweep, one in the decrease rule.

What changed

Tests pin the multi-actor fold. One worker, several actors: same-template entries sum under one key, a different template gets its own, a pending sibling contributes nothing to aggregates or events, CPU baselines are per actor (independent advance, independent epoch reset), one probe serves the whole response. Each test was verified to fail against the corresponding mutation.

Three CPU-accounting defects found and fixed along the way:

  1. Undercount. A pending entry (boot, restore, or a micro-VM guest the ateom did not reach in its 45s budget) never wrote the actor's baseline, and lastCPU is rebuilt each sweep — so measured → pending → measured lost the whole interval. On the guest-agent source the counter survives a restore, so that was real CPU time dropped, and on a full micro-VM worker it would happen routinely. Pending now carries the baseline forward; only actors absent from every response are dropped.
  2. Overcount opened by fix 1. Carrying the baseline unconditionally meant that during a restore spanning one sweep (source still reports the actor measured at C1, destination reports it pending), goroutine order could let the stale C0 overwrite C1, and the next sweep charged the interval twice. The pending branch now writes only if no worker has measured the actor this sweep.
  3. Overcount on a guest-agent decrease. A CPU decrease was charged as the whole new value. That is right for a cgroup counter, which restarts at zero on restore. It is not always right for a guest-agent counter: a FULL or DATA_ON_GOLDEN restore resumes a guest at that guest's counter (after a golden restore, main billed the golden guest's CPU to the actor), a cold boot restarts it at zero, and a container exit lowers the sum. A sample cannot tell these apart, so a guest-agent decrease now charges nothing and re-baselines. A cgroup decrease still charges the new value, with or without a pending sweep between. Trade-off, new versus main: a guest-agent decrease loses the usage from the event to the first sample. Main counted it correctly only after a cold boot (a DATA restore, or a Run after a crash); in every other case (golden restore, FULL restore from an older snapshot, container exit) main overcounted it.

Remaining limit, documented in the proto: an epoch that starts above the last reported value cannot be detected from consecutive samples. Main has this for guest-agent restores between sweeps; carrying the baseline extends it to restores that span a pending sweep. The overcount is bounded by the golden guest's counter minus the actor's last value, so it needs an actor that used less CPU than the golden guest had at snapshot.

The test for (2) forces both fold orders deterministically: the sweep closes a probe's connection right after folding its response, so the fixture's closer releases the gated probe from Close — no sleeps, no polling.

Timeout rationale reconciled. The poll floor (50s) is one actor's worst-case micro-VM read: 25 containers at 2s each. It never prevented overlap, because sweeps are sequential; it bounds guest-agent load, and the comment now says so. The RPC timeout (55s) was documented as covering that single-actor worst case. With several actors per worker, the micro-VM ateom caps its own read at statsSweepBudget (45s, chosen to stay under atelet's 55s) and reports unreached guests as pending, so the timeout stays a constant; its comment now points at that budget.

Not in scope, recorded

  • Two workers both measuring one actor in one sweep double-charges. Pre-existing and unchanged here. It is rare: it needs the same probe skew as defect 2, with the destination read after its restore finished.
  • ateom.proto still says the discovery list holds "at most one entry until multi-actor workers land". Stale; fixed in a separate change.

Verification

gofmt, go vet, and race-enabled tests pass. Mutation checks: dropping the carry, the measured-wins guard, or either side of the per-source decrease rule each fails its test.

Comment thread cmd/atelet/statspoller.go Outdated
// that live with that runtime -- so a shorter interval could start a new poll
// into a guest agent still serving the previous one.
// to. Sweeps never overlap, so it bounds guest-agent load, not correctness,
// and sits just above the micro-VM ateom's own sweep budget (45s).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is 45s coming from? Same below L56

@baizhenyu Tim Bai (baizhenyu) Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

45s is statsSweepBudget in cmd/ateom-microvm/stats.go. #1836 added it so the micro-VM ateom stops its guest reads before atelet's 55s statsRPCTimeout expires. It has no other derivation.

The comments had the dependency backwards. The 50s floor comes first: it is one actor's worst-case micro-VM read, maxActorContainers (25) containers at statsCallTimeout (2s) each. The 55s timeout is that plus margin. Fixed both in 0f5a684: the floor now states the 25 × 2s derivation, and statsRPCTimeout says the ateom sets its 45s budget below it.

@JeffLuoo Jeff Luo (JeffLuoo) self-assigned this Sep 25, 2026
@baizhenyu
Tim Bai (baizhenyu) force-pushed the atelet-multi-entry-fold branch 2 times, most recently from ae70c02 to 9329dc8 Compare September 25, 2026 20:33
…pending sweeps

Ateoms now return one GetActiveWorkloadStats entry per hosted actor. The
atelet sweep already folded every entry, but nothing tested it above one
actor per worker, and reviewing the fold found CPU accounting defects.

A pending entry (boot, restore, teardown, or a guest the ateom did not
reach) never wrote the actor's CPU baseline, and lastCPU is rebuilt each
sweep, so measured -> pending -> measured lost the whole interval. Pending
entries now carry the baseline forward; only actors absent from every
response drop it.

Carrying it unconditionally could overwrite a value another worker measured
in the same sweep (a restore in flight: the source still reports the actor),
charging the interval twice on the next sweep. The pending branch now writes
only if no worker measured the actor this sweep.

A CPU decrease was charged as the whole new value. That is right for a
cgroup counter, which restarts at zero on restore, but not always for a
guest-agent counter: a FULL or golden restore resumes a guest at that
guest's counter, a cold boot restarts it at zero, and a container exit
lowers the sum. A guest-agent decrease cannot tell these apart, so it now
charges nothing and re-baselines. That loses the usage from the event to
the first sample. Main counted it correctly only after a cold boot (a DATA
restore, or a Run after a crash); in every other case main overcounted it.
An epoch that starts above the last value stays undetectable, as the proto
notes, and carrying the baseline extends that limit to restores that span a
pending sweep.

Tests cover the fold over a multi-actor worker, per-actor baselines,
per-entry events, baseline carry across a pending sweep, the decrease rule
per source with and without a pending sweep, and the two-worker restore
case in both fold orders. That last one orders the probes on the fold
itself, via the connection close that follows it, rather than a sleep.

The poll floor (50s) is one actor's worst-case micro-VM read, 25
containers at 2s each. Sweeps are sequential, so it bounds guest-agent
load, not overlap. The RPC timeout (55s) was documented as covering that
single-actor worst case. With several actors per worker, the micro-VM ateom
caps its read at a 45s budget and reports unreached guests as pending, so
the timeout stays constant. The comments now say so.
@JeffLuoo
Jeff Luo (JeffLuoo) added this pull request to the merge queue Sep 28, 2026
Merged via the queue into agent-substrate:main with commit 7999d6f Sep 28, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/node area/observability kind/bug Something isn't working / bugfixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants