Skip to content

feat(ci): upstream-compat schedule + CI budget record (DEC-082) - #59

Merged
omar-nahhas merged 1 commit into
mainfrom
feat/upstream-compat-and-ci-budget
Aug 5, 2026
Merged

omar-nahhas merged 1 commit into
mainfrom
feat/upstream-compat-and-ci-budget

Conversation

@omar-nahhas

Copy link
Copy Markdown
Contributor

Closes C8. Scoped to the four plugins we are working on.

The finding that matters

The reusable workflow defaults a0_image to the private fork, and none of the four plugins overrides it. So every PR has been proving the plugin works on the fork — while upstream, the only thing anyone actually runs, was verified nowhere.

That is not a coverage gap so much as coverage aimed at the wrong image.

Two changes, deliberately separate

1. The PR gate should test what customers run. Each plugin gets a0_image: docker.io/agent0ai/agent-zero:latest in its own .devkit.yml — a per-repo override rather than flipping the devkit default, so this touches only our four. A break is then caught before merge, by the author, with the diff in front of them. (Separate PRs, one per repo.)

2. upstream-compat catches upstream moving underneath us — which no PR can catch, because nothing in the repo changed.

Why this schedule is not the one we just deleted

DEC-079 removed a nightly that opened a PR every night regardless of need and then failed silently behind an approval gate for three weeks. This one is different in the ways that caused that failure:

old nightly upstream-compat
trigger calendar upstream digest actually moved
cost 189 runs → 29 merges ~295 runner-min/mo vs ~1,275 naive
output a PR per night no PRs, no commits
failure visible? no — silent for 3 weeks one issue, auto-closes when green

Measured, not assumed: upstream ships ~1.5×/week, full fan-out is 42 min. Digest-gating gets the same ≤24h detection for ~76% less.

Three structural guards: the probe hard-fails rather than shrugging, an empty matrix is an error not a vacuous pass, and the report job asserts the matrix ran if the probe asked for it.

Runs pin the resolved digest, not :latest — so a failure is reproducible against the exact image that broke it, not whatever :latest points at by the time someone looks.

CI budget

Record lives in a long-lived per-repo issue, upserted in place — not a committed file. A committed record means a commit per run whose only delta is a timestamp, which is precisely the churn that got devkit-sync deleted.

A trap worth knowing: runs/{id}/timing returns total_ms: 0 on every run for these public repos. A budget built on it would report 0 min forever and look like it worked. It is deliberately unused, and the record says so on its face rather than presenting a confident zero.

Not yet proven

The workflows have never executed on Actions. Highest-risk untested detail is the plugin/ checkout and /plugin mount path in upstream-compat.yml — it runs in the devkit's workspace, not the consumer's, so that path differs from the proven plugin-e2e.yml body. I will exercise it via workflow_dispatch once merged, before claiming it works.

Upstream A0 is the only thing anyone actually runs, and nothing tested it. The
reusable workflow defaults a0_image to the private fork and none of the four
plugins overrode it, so every PR proved the plugin worked on the fork while the
target that matters went unverified. Coverage pointed at the wrong image.

upstream-compat re-runs the four suites when UPSTREAM moves — something no PR can
catch, because nothing in the repo changed. Gated on the manifest digest actually
moving: measured, upstream ships ~1.5x/week against a 42-minute fan-out, so
digest-gating costs ~295 runner-min/month instead of ~1,275 for the same <=24h
detection. Runs pin the resolved digest, not :latest, so a failure is reproducible
against the exact image that broke it.

Deliberately unlike the nightly just deleted in DEC-079: that opened a PR every
night regardless and then failed silently behind an approval gate for three weeks.
This opens no PRs, commits nothing, and reports one issue that auto-closes when
green — so an open issue always means act. Three guards keep it honest: the probe
hard-fails rather than shrugging, an empty matrix is an error not a vacuous pass,
and the report job asserts the matrix ran if the probe asked for it.

CI budget record lives in a long-lived per-repo issue upserted in place, NOT a
committed file — a committed record produces a commit per run whose only delta is
a timestamp, which is exactly the churn that got devkit-sync deleted.
Worth knowing: runs/{id}/timing returns total_ms 0 on every run for these public
repos, so a budget built on it would report 0 min forever and look like it worked.
It is deliberately unused and the record says so on its face.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant