Skip to content

ateapi: count golden snapshot builds by terminal outcome - #2459

Open
Luomu (cloudonly) wants to merge 1 commit into
agent-substrate:mainfrom
cloudonly:golden-build-metrics
Open

Luomu (cloudonly) wants to merge 1 commit into
agent-substrate:mainfrom
cloudonly:golden-build-metrics

Conversation

@cloudonly

@cloudonly Luomu (cloudonly) commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Closes #2478 — the "golden snapshot pipeline" blind spot in docs/metrics/substrate.yaml: "The template reconciler in ate-api has no instruments. Thus nothing measures a build of a golden image." Today a terminally-failed build leaves its reason only in the stored status an operator must fetch, and success is not counted at all.

ate.template.golden.builds (counter, {build}) records exactly one datapoint per build that reaches a terminal state:

  • success — the terminal commit of the golden tag in saveGoldenTag, including the crash-recovery path (a replica that died mid-build never committed, so the recovering pass's commit is the one and only record);
  • failure — fail()'s terminal commit, labeled with the bounded ate.template.golden.outcome value mapped from the terminal reasons (golden_tag_conflict, golden_actor_invalid, golden_actor_crashed, golden_actor_unexpected_state, and golden_egress_policy_conflict).

Labels are ate.template.atespace + ate.template.name (operator-named catalog objects, same class as ate.actor.crashes) + the outcome enum; no actor identity, no parallel failure counter. A template that opted out of the golden snapshot (#2328) builds nothing and records nothing; non-terminal errors retry and record nothing; a template deleted mid-build ends without a record (documented in the registry note).

Exactly-once mechanics: the counter fires only after the terminal status commit succeeds, and that commit is doubly guarded — the observed uid+version precondition and the mutate-time terminal-state check, with the row-level UPDATE ... WHERE uid AND version as the atomic backstop — so retried reconciles, the second replica, and post-recovery passes cannot double-count. Two inherent under-count windows remain (process death between commit and export; an ambiguous commit whose retry short-circuits on the terminal state), which is the right side to fail on. An exhaustiveness test keeps every future terminal reason mapped to a bounded value, and a map miss logs a warning rather than emitting an empty label.

Registry work: new registry.ate.template.golden attribute group (the five-member outcome enum), new metric.ate.template.golden.builds entry, weaver registry check --future green, blind-spot entry removed, docs/observability.md metrics table row added.

Testing (test-first, linux/amd64 with the real PG testcontainer): TestReconcileOneGoldenBuildsMetric drives the real reconciler over the fake store — success records once across two reconcile passes (the retried terminal pass adds nothing), tag conflict and crashed-actor each record their bounded reason once, a disabled template records nothing; assertions pin unit, monotonic sum, exactly one datapoint, and exactly the three labels. TestGoldenBuildOutcomesExhaustive pins the reason→outcome map. Full cmd/ateapi/internal/controlapi package passes.

Stacked on #2328 (the golden opt-out this builds on) — rebase onto main once it lands; the only overlap is template_reconciler.go, and the hunks are disjoint (record sites vs. disabled-mode short-circuits). Also textually disjoint from #2453 (failure-detail strings) and #2438 (proto renames).

Latest rebase (2026-10-23): rebased onto main 4ebd668d at ce353e12. Focused race regressions passed three shuffled runs; affected binaries and test packages compiled. Metrics registry verification passed. Earlier validation entries describe prior revisions. Cluster E2E was not rerun. Codex assisted with this rebase and validation.

@cloudonly
Luomu (cloudonly) force-pushed the golden-build-metrics branch 3 times, most recently from 45fd7db to 61d0e07 Compare October 17, 2026 02:06
@bowei Bowei Du (bowei) added area/api-machinery area/observability kind/feature An enhancement / feature request or implementation labels Oct 19, 2026
The template reconciler had no instruments, so nothing measured a
golden image build: a template whose build failed terminally left its
reason only in the stored status, and success was not counted at all.
The reconciler now increments ate.template.golden.builds once per build
that reaches a terminal state — success when the golden tag is
recorded (including its recovery after a replica died mid-build), or
the bounded failure reason when fail() commits — labeled by template
atespace and name plus ate.template.golden.outcome.

The counter fires only after the terminal status commit succeeds, and
that commit is guarded by the observed version precondition and the
terminal-state check, so retried reconciles and the second replica
cannot double-count a build. Two inherent under-count windows remain
(process death between commit and export; an ambiguous commit), which
is the right side to fail on. A template deleted mid-build and a
template that opted out both record nothing.

Registers the instrument in the weaver registry and removes the golden
snapshot pipeline blind spot from docs/metrics/substrate.yaml.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api-machinery area/observability kind/feature An enhancement / feature request or implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ateapi: nothing measures a golden snapshot build (template reconciler has no instruments)

2 participants