Repository navigation
ci(check): restore the base commit's Turbo cache first and report every check job - #279
Merged
Merged
Conversation
…ry check job `check` restored `turbo-Linux-dist-<sha>` through prefix restore-keys, so a PR got the newest entry its ref could read rather than its base commit's. #269 (one changeset file) restored an older main entry and re-ran 53 of 112 tasks; turbo's own dry-run hashes are identical between that base and head, so the key was never invalidated, the restored entry was simply another commit's. - check restores `turbo-<os>-dist-<base>` in its own step (first parent, or merge-base with main on dispatch), and falls back to the prefix only on a miss; actions/cache searches the current ref before main, so one step with restore-keys would still prefer this ref's stale entry. - Turbo writes run summaries (TURBO_RUN_SUMMARY); the cache is pruned to this run's task hashes and saved only when the Gate passed and missed a task. - cache-sweep (main pushes only, the one job with actions: write) deletes turbo entries by cache id: main keeps its newest three, other refs one, closed pull requests none. - .github/actions/job-report writes each job's summary (duration, Turbo cached/total, slowest missed tasks, restored key) and, on failure, an annotation titled with the reason code: GATE_OK, TURBO_TASK_FAILED, GATE_NON_TURBO_FAILED, CHECK_SETUP_FAILED, CANCELLED, SWEEP_OK/FAILED. - workflow_dispatch input `cache: cold` skips the restore to measure cold. Adds the CI wall-clock plan (docs/plans) this change is unit U1 of.
…cache - The save guard compared only the fallback restore's exact hit, which is empty whenever the base entry restored, so a same-sha rerun tried to save over its own key. It now skips when either restore matched this run's key. - cache-sweep lists and deletes through `gh cache list` / `gh cache delete <id>` (still by id) and checks out only the job-report action. - job-report names its reason vocabulary itself and reports NO_REASON_CODE instead of an empty title when a caller passes none.
cancelled() is valid only in if: conditionals, so the workflow failed to compile on every push of this branch (actionlint: calling function "cancelled" is not allowed here).
…rden the sweep - .github/scripts/turbo-summary.sh reads missed and failed tasks from Turbo's run summaries and exits 2 naming the first missing field. The verdict maps that to GATE_VERDICT_UNPARSEABLE with a next action and fails the job, instead of reading a format change as "nothing missed, nothing failed". turbo-summary.test.sh feeds it summaries missing fields and runs in check before the Gate. - A lookup-only probe of this commit's own entry replaces the key comparisons: a re-run never re-tars and re-saves an entry that exists. A cache: cold dispatch never saves. The summary says "save: attempted" or "skipped (<why>)". - cache-sweep: a failed delete is a SWEEP_DELETE_FAILED warning plus a count, and a failed PR-state lookup keeps that PR's entries; only a failed listing fails the job. - The e2e comment claimed a cancelled job skips if: always() steps; a cancelled run ran them. It now says what timeout buys.
… with the repo scripts #275 runs every scripts/*.test.ts in check, with Deno. The summary reader moves there as scripts/turbo-summary.ts (run with node in the verdict) beside a Deno test that feeds it summaries missing the exit code, the cache status, or the whole summary. The bash reader and its own test step are gone.
…eno test The previous commit removed the bash reader without the workflow and script changes that replace it.
…h install @effect/platform-node, platform-node-shared, opentelemetry and vitest 4.0.3 were published at 10:03 UTC with a peer effect@^4.0.3 that is not on the registry. The bake resolves registry dependencies fresh, so npm resolved forever and every e2e leg of run 38043976128 timed out in global setup. Both bake installs now pass --min-release-age=1, and each runs under a 300 s timeout whose expiry names the fixture and step on stderr.
The bake guest is node:24-alpine, whose timeout is busybox: a killed install exits with its own signal status (143 or 137), never GNU's 124, so the reason line never printed. The script now decides from elapsed seconds, and timeout -k 10 kills an npm that ignores TERM.
This was referenced Oct 10, 2026
Merged
ryanleecode
approved these changes
Oct 10, 2026
ryanleecode
left a comment
ryanleecode
left a comment
Contributor
There was a problem hiding this comment.
Root verified at 1ab873d: all checks green, 0 threads, no suppressions. Over-10m e2e/check legs are inherited (Stream K).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unit U1 of the CI wall-clock plan (
docs/plans/2026-10-10-0702-perf-ci-wall-clock-plan.md,### U1.). Scope granted by unit it_b2aa4120:.github/workflows/*.AGENTS.mdstill lists.github/workflows/as Read-only and is left unchanged.What changes
checkrestores the Turbo cache entry of the commit's base by exact key. The base is the first parent, or the merge-base with main on dispatch. It falls back to the newest prefix match this ref can read only when the base commit has no entry. Before this, one restore step withrestore-keysreturned the ref's newest entry, which could be stale, ahead of main's exact base entry. That cost 13.7-18.4-minute Gates on PRs whose base entry existed.TURBO_RUN_SUMMARY=true. A verdict step derives a reason code from the step outcomes and from Turbo's run summaries, read byscripts/turbo-summary.ts. The codes areGATE_OK,TURBO_TASK_FAILED,GATE_NON_TURBO_FAILED,CHECK_SETUP_FAILED,CANCELLEDandGATE_VERDICT_UNPARSEABLE.GATE_VERDICT_UNPARSEABLEwith a next action and fails the job. It never reports a silentGATE_OK.scripts/turbo-summary.test.tsruns in "Repo scripts tests" and feeds the reader summaries missing the exit code, the cache status, or the whole summary. If the exit-code guard is removed, the test fails withgot {"kind":"read",...}.cache: colddispatch never saves. Before saving, restored entries that this run's task hashes do not reference are pruned, the same way asmutation.yml's build job. The summary line sayssave: attempted (...)orsave: skipped (<why>), never "saved".cache-sweepjob, which runs on main pushes only and is the only job withactions: write. It deletes Turbo entries by id: main keeps its newest 3, every other ref its newest 1, closed PRs 0.SWEEP_DELETE_FAILEDwarning plus a count.SWEEP_PR_STATE_UNKNOWNwarning..github/actions/job-report. It gives every job it is wired into a step summary: outcome, reason code, duration, Turbo cached/total, and the slowest missed tasks. On failure it adds an annotation titled with the reason code, whose message is the next action.workflow_dispatchinputcache: coldskips both restores, to measure a cold Gate.if: always()steps. Run 38041980554 disproved that, and the comment now says whattimeoutbuys.Coverage
pnpm check:ci, same planted two-plan check, same e2e matrix.turbo run … --dry=jsongives 112 Gate tasks and 17gate:disttasks at both baseadd6faa73and head, with 0 differing task hashes.Before / after (minutes, first job start to last job end)
6a91cdf74541aa86541aa86d1c3489checkcheck: 20.8 min, with the Gate at 19.4 min and both restores skipped. Before this change, a cache miss took 17.2-18.4 min of Gate (run 38026672592). U1 removes the stale-restore misses but does not make a true cold run faster; that is U6's lever.541aa86predates the never-save-when-cold fix.E2E timeouts are repo-wide, not this diff
Since about 10:10Z, every e2e leg times out at exactly 20.0 min in "E2E lane":
ee2c4cfb0): all 5 legs;The same tree
541aa86passed every leg at 09:59-10:13Z. In the cold run's exported traces (rest-2),e2e.setup.packfinishes in 21 s andboot_sandboxandawait_readinesssucceed at 24 s. After that, no span closes until the timeout. The guest bake job never completes. This diff touches no e2e step except a comment.Agent surface
GATE_OK, warm,d1c3489, job 114192840110: "Turbo cache restored:turbo-Linux-dist-ee2c4cfb0…(base commit); save: skipped (every task was cached)".GATE_NON_TURBO_FAILED, dprint plant indocs/, run 38041980554: annotation "A Gate script outside Turbo failed: format:check (fix with pnpm format), lint:conventions, or gate:repo. Reproduce with: pnpm check:ci". Turbo was still 112/112 cached.TURBO_TASK_FAILED, type-error plant inpackages/ignorers/angular/tests, run 38042401509: annotation "Turbo task failed: @systemfsoftware/stryker-ignorer-angular#test …#typecheck. Reproduce with: pnpm exec turbo run …". 109/112 were cached.541aa8621, whose tree is byte-identical to6a91cdf74.Residual risks
checkthat hits itstimeout-minutes: 45has not been observed. GitHub documents that a cancelled job runsif: always()steps within a 5-minute window, so the verdict and job report should run, but no run proves it for the timeout case. Superseded-push cancellation is proven: in run 38041980554, the legs'always()steps ran after "E2E lane" was cancelled.cache-sweepruns only on main pushes, so it has not run on this branch. It lists at most 1000 entries.