Repository navigation
CI costs about 110 runner-minutes for every code PR, whatever it changes: run the guard tests first, build the test projects once, and choose test classes by what changed #5459
Description
Claude posting for Erik Darling.
A code PR's CI run costs a median of 109 runner-minutes (p90 144), and 20 of the last 20 PR runs ran both full suites. A 3-file Darling change (#5456) cost the same as a schema migration. Below are the numbers, where the time goes, which tests actually fail, and five changes in order of payoff.
All numbers are measured unless marked inferred. The sources:
- the job and step timings of the last 20 successful PR runs and 10 dev push runs (Sep 30 – Oct 7);
- every
build.ymlrun since Sep 7, with the failed jobs' logs; - per-class times from one full local run on a Windows 11 ARM VM, plus one macOS run for the live-store classes the VM skips.
Where a run's time goes (median of 20 PR runs)
| Job | Instances | Minutes each | Runner-minutes |
|---|---|---|---|
| Darling PG tests | 3 (6 in 8 of 20 runs) | 12–16 | about 40 (up to about 80) |
| Lite tests | 4 | 9–11 | about 40 |
build (also runs the Darling tests that need no store) |
1 | 14.4 | 14.4 |
| Darling whole-tree guards | 1 | 0.3 (p90 11) | up to 11 |
| Linux build, gate, result jobs | about 3 |
Each shard repeats the same setup.
- Per run:
Darling.Testsis built in each PG shard (3 × 3.1 min),Lite.Testsfive times (5 × 1.9 min) and .NET is set up nine times (9 × 0.5 min). Restores, checkouts and the upgrade-fixture runtime build add more. - In total: about 28 runner-minutes, about 25% of a run, are setup repeated per shard. With 6 PG shards it's more.
The test work itself is smaller than the runner time it takes.
| Suite | Test time per class, summed | One process on the VM, wall time |
|---|---|---|
| Lite.Tests (8,810 tests, 702 classes) | 20.4 min | 7.4 min |
| Darling.Tests without a store (about 25,000 tests) | 11.1 min | 3.7 min |
| Darling live-store classes (535 classes, 2,254 tests) | 41.8 min (macOS) | — |
- Top-heavy? No. The top 50 classes are only 41% of test time.
- What follows: trimming slow classes alone won't move the total much.
Selection today: area filters that overlap
- The mechanism:
build.ymlchooses bydorny/paths-filterarea. - The overlap: the
litefilter includesDarling/Darling.Tests/**andDarling/PerformanceMonitor.Darling.Service/**, and thedarlingfilter includesLite/**/*.cs,*.xamland*.csproj. - The effect: almost any code change selects every suite. Each overlap exists because some test reads the other app's files (PR CI skips Lite.Tests when only Darling changes, so cross-app parity guards do not gate merges #2839, Say where the data starts on every time-ranged tab, in Lite and both Darling viewers #4966). But the filters run whole suites to cover a handful of cross-reading classes.
Which tests fail CI
The counts, over 30 days (Sep 7 – Oct 7):
| Runs | Count |
|---|---|
All build.yml runs |
5,797 |
| Failed | 1,238 |
| Failed only because of the draft or merge-train gate, by design | 502 |
| Failed with a real test or build failure | 711 |
The cheapest tests fail CI most often.
- Guard-style classes (hygiene, census, convention, adoption, ratchet and source-pin checks): 173 classes, 5.5% of test time, 18% of all class failures.
- The five most-failed classes take under 13 s together:
| Class | Failed jobs in 30 days | Seconds per run |
|---|---|---|
DocCommentHygieneTests |
97 | 6.9 |
LivePostgresCollectionHygieneTests |
65 | 0.5 |
RepoFileAdoptionTests |
50 | 3.4 |
TsqlConventionGuardTests |
50 | 1.5 |
LiveCleanupConversionRatchetTests |
33 | 0.4 |
- The cost of that: each of those failures cost a full run of about 110 minutes. For example, a PR whose one failure was a stacked doc-comment summary in a test file used a full run to find it.
Most classes never fail.
- 2,053 of 2,819 classes, 65% of test time, failed no CI job in 30 days.
- That isn't a reason to delete them: they guard against regressions. It makes them the natural candidates for change-based selection, and for running only on dev pushes and in the nightly.
Flakes are undercounted here. A run that failed and then passed on re-run is recorded as a success, so it's missing from the failed-run list.
- Known flakes: Flaky test: a large plan read can go over the statement filter's time budget on a slow CI runner #5440, Flaky test: TrendPayloadBudgetLiveTests fails now and then in a local run, then passes #5425, Flaky test: the retired baseline sweep test finds its connection closed after the first attempt #5416, and today's
StatementFilterAlertTests.Engine_AFourMegabyteReport_ReachesTheDelivererFilteredNotWithheldWholeon PG shard 4, which passed on re-run with no change. - A proper count needs the per-attempt history of each run.
Proposed changes, in order of payoff
- Run the guard tests first, and fail fast.
- How: one early job builds the test projects once and runs only the guard-style classes, which take about 4 test-minutes. The shard jobs
needs:it. - What it saves: a convention miss then costs about 5–8 runner-minutes instead of about 110 (inferred from the times above).
- Optional: make the same classes easy to run locally before a push.
- How: one early job builds the test projects once and runs only the guard-style classes, which take about 4 test-minutes. The shard jobs
- Build the test projects once and share the output.
- How: the early job uploads the built
Darling.TestsandLite.Testsfolders as an artifact. Each shard downloads it, instead of restoring and building. - What it saves: most of the about 28 repeated runner-minutes per run (inferred). Measure the artifact upload and download time first.
- How: the early job uploads the built
- Narrow the overlapping filters to the cross-reading classes.
- How: list the test classes that read the other app's files. Lite.Tests already tags some
Reads=Darling; add the Darling.Tests side. When only the other app changed, run just those classes. - What it saves: a Darling-only change then skips about 40 runner-minutes of Lite shards, and the reverse.
- How: list the test classes that read the other app's files. Lite.Tests already tags some
- Choose test classes by what changed, from coverage.
- How: the nightly records which source files each test class runs, as a class-to-file map. A PR then runs:
- the classes that touch its changed files;
- the guard stage from step 1;
- every class whose own file changed.
- The safety net: the full suite still runs on every dev push and in the nightly, so a missed test is caught within one merge, not shipped.
- Keep full runs for: changes to
build.yml,Directory.Build.propsand the shared projects, until the map has proven itself.
- How: the nightly records which source files each test class runs, as a class-to-file map. A PR then runs:
- Then review value, with real flake counts.
- How: count reruns that passed with no change, per class, from the attempt history.
- Flaky classes: fix them or quarantine them.
- Never-failed slow classes: for the costliest, decide case by case whether they belong in the nightly only. The costliest never-failed classes today are:
DarlingPasswordKeyStoreLiveTests(70 s);- Lite's
McpWindowNoticeSystemHealthToolTests(56 s); WebDataStartNoteLiveTests(54 s);FinOpsStorageGrowthViewLiveTests(53 s);ComposeTimeoutPhaseLiveTests(47 s, 2 tests).
Not covered
- Byte and cost totals: GitHub minutes billing isn't read here.
- The release workflow and its signing steps.
- Whether the PG shards' live-store time on a hosted runner differs from the macOS numbers. The shard logs give the shard total but not per-class times.
Activity
Revert "Plant one stacked summary so the proof run has a guard failure (
Guard stage: the census checks xunit's own tagged set, the guard job …
Guard stage review round 2: the reflection matches xunit's discovery,…
A guard miss fails the pull request in about 6 minutes and skips ever…
The Guard build artifacts also carry the Darling service output and t…
The CI shards reuse the Guard tests job's build instead of building t…
The no-store Darling run keeps only the Guard stage while the Postgre…
The install-tree wait test's holder process outlives a slow PowerShel…
DuckDB trim test waits for a trim cycle that really ran, not for one …
The CPU accountant's tests burn thread CPU time, not wall time, so a …
One selection script for the test jobs, with a failure-corpus replay …
QueryStoreIntervalWideGridLiveTests waits out closed backends instead…
Revert "QueryStoreIntervalWideGridLiveTests waits out closed backends…
17 remaining items
Pull requests skip 27 slow, never-failed test classes unless the chan…
A managed store near a WAL size step no longer rewrites its settings …
Nightly map shards and the AltCover probe wait for build and darling-…
Shadow mode for the test-class map: a pinned map, the selection in th…
erikdarlingdata commentedon Oct 9, 2026
Claude posting for Erik Darling.
Shadow mode has run on 76 pull request runs and the map missed nothing. But 68 of those 76 selections were FULL. Only one run put a failing class in front of a real selection. I did not switch to live selection. Numbers first, then why.
The numbers (build.yml pull_request runs created since 2026-10-08 09:00:20Z, 188 runs)
proof/5603-*runs and one empty cancelled run (no shadow jobs)Why the 68 FULL selections were FULL:
.github/workflows/build.yml30,Darling/Darling.Tests/Darling.Tests.csproj9,PerformanceMonitor.Alerting/AgentJobStepQuery.cs5.The keep-full check counts files changed on dev since the map was built, not just the pull request's own files. Ten pull requests had neither build.yml nor the Darling test project in their own file list: #5564, #5566, #5572, #5576, #5579, #5580, #5588, #5590, #5593 and #5599. Each still got a FULL run from a file that changed on dev after the nightly. On a day when CI itself is being changed, the map is mostly unused. Only 8 of the 56 runs that had a map got a real selection.
The hand check
For the 41 not-green runs where the shadow job ran, I took the failed classes from each run's own failed-job logs and compared them to the row. Every failed class in the shard logs matched the script's list. One run has a failure and a real selection: 37785672153 (#5566).
ScreenReaderNamesFollowUpTestsandScreenReaderRowNamesTestsfailed in Lite shard 0. Both were in the selection ("covers a changed file"). The script and I agree there is no miss.Where I disagree with the script:
shadow-checkreads only the*-tests-timing-*shard artifacts. In 5 runs the row says no failing class, yet the Guard tests job had one: 37767253953 (CommentFilterAdoptionTests), 37793985305 (WpfStaGateCensusTests), 37849995332 (RepoFileAdoptionTests), 37867831747 (LiveCleanupConversionRatchetTests,McpPayloadContractCensusTests), 37869178958 (those two plusAggregateJobGraceReopenTests). In 2 more runs only "Darling whole-tree guards" failed: 37843503108 and 37836917303. All 7 were FULL, so no miss was hidden. Guard classes carry Stage=Guard, so the map always selects them. Even so, a live miss check would not report a guard failure either.ScreenReaderRowNamesTestsas failed, so the row read a failed shard's report from attempt 1. A live check could then flag a class that passed.Back-test
I ran
ci-select.py --replay --mapover.github/ci-history/failures.jsonlwith the newest map (nightly run 37815034195, commit e161e5c). It read 731 rows (312 FULL) and checked 2,035 failing classes. 18 were accepted and 55 were misses across 37 classes, all in pull request rows. Those 37 classes failed 161 times in the corpus, so the map covered the other 106. The replay treats the map as current, with no drift. Most misses are live-store classes that failed on a pull request that never touched them. Examples arePgServerLogTailRotationLiveTests,StoreStatementStatsLiveTestsandStatementFilterAlertTests, which this issue names as a known flake. The biggest cluster isDarlingAnomalyBaselineTests, 7 times on 2026-09-24. It readsLite/Analysis/AnomalyDetector.cs, and those rows changed onlyLite.Tests/AnomalyDetector*Tests.cs. That looks like a red base, not a missing link. I did not prove it for each of the 55.Why I stopped short of live selection
The literal switch condition (0 misses, one run with a real selection and a failure) is met. I held back because:
build,darling-pgandlite-testsand adds a wait on the selection job. The pull request that makes the change cannot test it, because it changes build.yml and so selects FULL.If you want it anyway, the smallest version is this. The three jobs download
test-map-shadow-selectionand filter their class list after the hash cut. They keep the Guard stage and the FULL fallback. The map selection replaces the Cost=Slow skip when it is not FULL. Say so and I will build it. I opened no pull request.Shadow check: stale reports are decided per artifact folder, so a sha…
Test map: shared live-store fixtures are linked, keep-full counts onl…
A pull request into main always runs every test class (#5459)
Test map selection: nested classes and CI-only gated classes are kept (…
Test map selection: comment wording only (#5459)
Test map selection: state the gate scan's limits and measured cost; r…
Pull request CI runs only the test classes the test map picks (#5459) (…
erikdarlingdata commentedon Oct 9, 2026
Fixed on dev by #5624 (d7a321e), with Erik's OK to go live. Pull request CI now runs only the test classes the test map picks for the diff. It runs everything when the selection is full, missing, unreadable or empty, and for any pull request into main. Test classes gated on PostgreSQL environment variables always run. The nightly still runs the full suite. Claude posting for Erik Darling.