Repository navigation
atelet: pin the multi-actor stats fold and keep CPU baselines across pending sweeps - #1885
Merged
Jeff Luo (JeffLuoo) merged 1 commit intoSep 28, 2026
Conversation
Tim Bai (baizhenyu)
force-pushed
the
atelet-multi-entry-fold
branch
5 times, most recently
from
September 25, 2026 19:30
f9a6fa8 to
ac4524b
Compare
Tim Bai (baizhenyu)
requested review from
Jeff Luo (JeffLuoo) and
Da Huang (git286)
September 25, 2026 19:35
Tim Bai (baizhenyu)
force-pushed
the
atelet-multi-entry-fold
branch
from
September 25, 2026 19:40
ac4524b to
628eb71
Compare
| // that live with that runtime -- so a shorter interval could start a new poll | ||
| // into a guest agent still serving the previous one. | ||
| // to. Sweeps never overlap, so it bounds guest-agent load, not correctness, | ||
| // and sits just above the micro-VM ateom's own sweep budget (45s). |
Collaborator
There was a problem hiding this comment.
Where is 45s coming from? Same below L56
Collaborator
Author
There was a problem hiding this comment.
45s is statsSweepBudget in cmd/ateom-microvm/stats.go. #1836 added it so the micro-VM ateom stops its guest reads before atelet's 55s statsRPCTimeout expires. It has no other derivation.
The comments had the dependency backwards. The 50s floor comes first: it is one actor's worst-case micro-VM read, maxActorContainers (25) containers at statsCallTimeout (2s) each. The 55s timeout is that plus margin. Fixed both in 0f5a684: the floor now states the 25 × 2s derivation, and statsRPCTimeout says the ateom sets its 45s budget below it.
Tim Bai (baizhenyu)
force-pushed
the
atelet-multi-entry-fold
branch
2 times, most recently
from
September 25, 2026 20:33
ae70c02 to
9329dc8
Compare
…pending sweeps Ateoms now return one GetActiveWorkloadStats entry per hosted actor. The atelet sweep already folded every entry, but nothing tested it above one actor per worker, and reviewing the fold found CPU accounting defects. A pending entry (boot, restore, teardown, or a guest the ateom did not reach) never wrote the actor's CPU baseline, and lastCPU is rebuilt each sweep, so measured -> pending -> measured lost the whole interval. Pending entries now carry the baseline forward; only actors absent from every response drop it. Carrying it unconditionally could overwrite a value another worker measured in the same sweep (a restore in flight: the source still reports the actor), charging the interval twice on the next sweep. The pending branch now writes only if no worker measured the actor this sweep. A CPU decrease was charged as the whole new value. That is right for a cgroup counter, which restarts at zero on restore, but not always for a guest-agent counter: a FULL or golden restore resumes a guest at that guest's counter, a cold boot restarts it at zero, and a container exit lowers the sum. A guest-agent decrease cannot tell these apart, so it now charges nothing and re-baselines. That loses the usage from the event to the first sample. Main counted it correctly only after a cold boot (a DATA restore, or a Run after a crash); in every other case main overcounted it. An epoch that starts above the last value stays undetectable, as the proto notes, and carrying the baseline extends that limit to restores that span a pending sweep. Tests cover the fold over a multi-actor worker, per-actor baselines, per-entry events, baseline carry across a pending sweep, the decrease rule per source with and without a pending sweep, and the two-worker restore case in both fold orders. That last one orders the probes on the fold itself, via the connection close that follows it, rather than a sleep. The poll floor (50s) is one actor's worst-case micro-VM read, 25 containers at 2s each. Sweeps are sequential, so it bounds guest-agent load, not overlap. The RPC timeout (55s) was documented as covering that single-actor worst case. With several actors per worker, the micro-VM ateom caps its read at a 45s budget and reports unreached guests as pending, so the timeout stays constant. The comments now say so.
Tim Bai (baizhenyu)
force-pushed
the
atelet-multi-entry-fold
branch
from
September 25, 2026 20:35
9329dc8 to
0f5a684
Compare
Jeff Luo (JeffLuoo)
approved these changes
Sep 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the remaining items in #1640: the atelet side of multi-actor telemetry.
Since #1836 an ateom returns one
GetActiveWorkloadStatsentry per hosted actor. atelet's sweep already folded every entry, but nothing pinned the behavior at occupancy above one, and reviewing that fold surfaced three CPU-accounting defects: two in how baselines survive a pending sweep, one in the decrease rule.What changed
Tests pin the multi-actor fold. One worker, several actors: same-template entries sum under one key, a different template gets its own, a pending sibling contributes nothing to aggregates or events, CPU baselines are per actor (independent advance, independent epoch reset), one probe serves the whole response. Each test was verified to fail against the corresponding mutation.
Three CPU-accounting defects found and fixed along the way:
lastCPUis rebuilt each sweep — so measured → pending → measured lost the whole interval. On the guest-agent source the counter survives a restore, so that was real CPU time dropped, and on a full micro-VM worker it would happen routinely. Pending now carries the baseline forward; only actors absent from every response are dropped.Remaining limit, documented in the proto: an epoch that starts above the last reported value cannot be detected from consecutive samples. Main has this for guest-agent restores between sweeps; carrying the baseline extends it to restores that span a pending sweep. The overcount is bounded by the golden guest's counter minus the actor's last value, so it needs an actor that used less CPU than the golden guest had at snapshot.
The test for (2) forces both fold orders deterministically: the sweep closes a probe's connection right after folding its response, so the fixture's closer releases the gated probe from
Close— no sleeps, no polling.Timeout rationale reconciled. The poll floor (50s) is one actor's worst-case micro-VM read: 25 containers at 2s each. It never prevented overlap, because sweeps are sequential; it bounds guest-agent load, and the comment now says so. The RPC timeout (55s) was documented as covering that single-actor worst case. With several actors per worker, the micro-VM ateom caps its own read at
statsSweepBudget(45s, chosen to stay under atelet's 55s) and reports unreached guests as pending, so the timeout stays a constant; its comment now points at that budget.Not in scope, recorded
ateom.protostill says the discovery list holds "at most one entry until multi-actor workers land". Stale; fixed in a separate change.Verification
gofmt,go vet, and race-enabled tests pass. Mutation checks: dropping the carry, the measured-wins guard, or either side of the per-source decrease rule each fails its test.