fix(tier1): count from the artifact when the cache eats the summary; content-keyed reach; anti-stall ratchet (#439) - #442
Merged
Merged
Conversation
…s stalling (#439) Three ways the gate lied on a build that was green. A fully cached `zig build test` on zig 0.17 prints "Build Summary: 4/4 steps succeeded" and no "N/N tests passed" line, because the run step never re-ran. The sed came back empty and the tests check FAILED - the field workaround was `rm -rf .zig-cache/h`. Ask the artifact instead: a test binary run with no arguments prints "All N tests passed." every time, cached or not, and running it re-proves the suite besides. The same fallback covers the two filtered builds in the invariants check, which cache and go quiet identically on a second run. Picking that artifact by mtime is the second bug. `.zig-cache/o` holds one binary per distinct build and every `-Dtest-filter` build lands there too, so after the invariants check - or any hand-run filtered build - the newest binary holds only the tests that filter matched, and reachability reported the other 952 declared tests as "not compiled in". Selection moves to content: `-Dtest-filter` drops non-matching tests at compile time, names and all, so the build we want is the one whose declared-name set matches what its filters select. mtime only breaks ties. And test_count_baseline was 656 against a suite of 994, a floor low enough to miss a whole module falling out of the test root. It is now 994, still a floor nothing raises on its own, but `count` exits 3 - a warning, never a failure - once the suite runs more than test_count_slack (25) ahead of it, so it cannot silently sit 350 behind again. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #439 — all three defects in the tier-1 gate, each with a demonstrated proof (details in the commit; 4 green end-to-end runs, cold 61s / warm 21s).
zig build testprints noN/N tests passedline, so thetestscheck red-flagged green builds. New shared resolver (tier1_test_binary.py) finds the right.zig-cache/oartifact and readsAll N tests passed.off it directly. Same fallback wired intoinvariants(its filtered builds cache and go silent identically). Proof: old script FAILs on a hot cache, new one reports 994 both hot and cold.test_count_baseline656 → 994 (measured), plustest_count_slack: 25— a count more than slack above the baseline exits 3 and the verdict line carriesWARN: the ratchet has stalledwhile staying green, so the baseline can never silently sit 350 behind again. Floor semantics unchanged (994 baseline / 993 observed still FAILs); nothing auto-raises. Full probe matrix in the commit.-Dtest-filterdrops tests at compile time, so the declared-test byte-strings a binary contains identify which build it is — full (961 names), floor (0), invariants (9). Reachability now selects by content and explicitly logs when it ignores a newer-but-filtered artifact. A/B on an identical cache: old reports "952 tests NOT compiled in" (exactly 961−9 — the mechanism), new correctly finds all 961.Honest cost: warm tier-1 is ~21s (was ~5s-but-failing) because the fallback genuinely re-executes the suite instead of trusting a run that never happened — still inside the script's stated ~30s budget. The cheaper
--listenmetadata-query route is noted in AGENTS.md as the escape hatch, deliberately not taken (binding the gate to a wire format is the failure class that broke it here).No Zig touched; golden harness unaffected. Three agents independently hit these false-FAILs this week — this restores trust in the gate that has caught real regressions before.