Repository navigation
ateapi: count golden snapshot builds by terminal outcome - #2459
Open
Luomu (cloudonly) wants to merge 1 commit into
Open
Luomu (cloudonly) wants to merge 1 commit into
Luomu (cloudonly) wants to merge 1 commit into
Conversation
Luomu (cloudonly)
force-pushed
the
golden-build-metrics
branch
3 times, most recently
from
October 17, 2026 02:06
45fd7db to
61d0e07
Compare
The template reconciler had no instruments, so nothing measured a golden image build: a template whose build failed terminally left its reason only in the stored status, and success was not counted at all. The reconciler now increments ate.template.golden.builds once per build that reaches a terminal state — success when the golden tag is recorded (including its recovery after a replica died mid-build), or the bounded failure reason when fail() commits — labeled by template atespace and name plus ate.template.golden.outcome. The counter fires only after the terminal status commit succeeds, and that commit is guarded by the observed version precondition and the terminal-state check, so retried reconciles and the second replica cannot double-count a build. Two inherent under-count windows remain (process death between commit and export; an ambiguous commit), which is the right side to fail on. A template deleted mid-build and a template that opted out both record nothing. Registers the instrument in the weaver registry and removes the golden snapshot pipeline blind spot from docs/metrics/substrate.yaml.
Luomu (cloudonly)
force-pushed
the
golden-build-metrics
branch
from
October 23, 2026 08:17
61d0e07 to
ce353e1
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2478 — the "golden snapshot pipeline" blind spot in
docs/metrics/substrate.yaml: "The template reconciler in ate-api has no instruments. Thus nothing measures a build of a golden image." Today a terminally-failed build leaves its reason only in the stored status an operator must fetch, and success is not counted at all.ate.template.golden.builds(counter,{build}) records exactly one datapoint per build that reaches a terminal state:saveGoldenTag, including the crash-recovery path (a replica that died mid-build never committed, so the recovering pass's commit is the one and only record);fail()'s terminal commit, labeled with the boundedate.template.golden.outcomevalue mapped from the terminal reasons (golden_tag_conflict,golden_actor_invalid,golden_actor_crashed,golden_actor_unexpected_state, andgolden_egress_policy_conflict).Labels are
ate.template.atespace+ate.template.name(operator-named catalog objects, same class asate.actor.crashes) + the outcome enum; no actor identity, no parallel failure counter. A template that opted out of the golden snapshot (#2328) builds nothing and records nothing; non-terminal errors retry and record nothing; a template deleted mid-build ends without a record (documented in the registry note).Exactly-once mechanics: the counter fires only after the terminal status commit succeeds, and that commit is doubly guarded — the observed uid+version precondition and the mutate-time terminal-state check, with the row-level
UPDATE ... WHERE uid AND versionas the atomic backstop — so retried reconciles, the second replica, and post-recovery passes cannot double-count. Two inherent under-count windows remain (process death between commit and export; an ambiguous commit whose retry short-circuits on the terminal state), which is the right side to fail on. An exhaustiveness test keeps every future terminal reason mapped to a bounded value, and a map miss logs a warning rather than emitting an empty label.Registry work: new
registry.ate.template.goldenattribute group (the five-member outcome enum), newmetric.ate.template.golden.buildsentry, weaverregistry check --futuregreen, blind-spot entry removed,docs/observability.mdmetrics table row added.Testing (test-first, linux/amd64 with the real PG testcontainer):
TestReconcileOneGoldenBuildsMetricdrives the real reconciler over the fake store — success records once across two reconcile passes (the retried terminal pass adds nothing), tag conflict and crashed-actor each record their bounded reason once, a disabled template records nothing; assertions pin unit, monotonic sum, exactly one datapoint, and exactly the three labels.TestGoldenBuildOutcomesExhaustivepins the reason→outcome map. Fullcmd/ateapi/internal/controlapipackage passes.Stacked on #2328 (the golden opt-out this builds on) — rebase onto main once it lands; the only overlap is
template_reconciler.go, and the hunks are disjoint (record sites vs. disabled-mode short-circuits). Also textually disjoint from #2453 (failure-detail strings) and #2438 (proto renames).Latest rebase (2026-10-23): rebased onto main
4ebd668datce353e12. Focused race regressions passed three shuffled runs; affected binaries and test packages compiled. Metrics registry verification passed. Earlier validation entries describe prior revisions. Cluster E2E was not rerun. Codex assisted with this rebase and validation.