feat(goal): skip a re-verify the workspace cannot have changed (#412) - #434
Merged
Conversation
A /goal (or /loop) run is plan-act-VERIFY and re-verifies on every continuation, so the obvious move for a model that has just been told "completion is blocked, run eval" is to run eval again. When it has edited nothing since the last RED, that re-run costs the whole --eval command, a --judge model call, and 1500 bytes of output tail to re-derive a verdict that could not have changed. verify_fingerprint.zig fingerprints the verification before running it, and a fingerprint identical to the one the last verification FAILED on means the verifier is not run at all: - the fold covers every input to "could this produce a different result": the eval command text, `git status --porcelain -z -uall`, `git diff --binary HEAD`, and the CONTENTS of every untracked file the status listed. The last one is load-bearing - editing an untracked file leaves the porcelain line and the diff byte-identical, so without it the guard would skip a verify over real work. Parts are length-prefixed, so a byte moved across a file boundary is a change, not a collision. - the attempt still counts (eval_iter advances). A model that keeps calling eval without editing has to converge on the iteration cap, not spin for free. - the steer names the actual blocker instead of manufacturing a verdict: the workspace has not changed since the last failed verification, edit source files or tests first. Completion stays blocked either way, since a skipped run verifies nothing. FAIL-OPEN everywhere: no repo, no git, a timed-out probe, a truncated stream, an unreadable file, an absurd untracked count - each yields an unknown fingerprint, which never matches, so the verifier runs. The guard can cost a skipped re-run only when it is certain nothing moved; it can never invent a pass. A command that could not RUN disarms it too - "edit source files" is not the fix for that, and a stale fingerprint must not suppress the retry. The fold and the decision are pure and unit-tested without a repo (identical / tracked change / untracked-only change / boundary collision / fail-open matrix). The end-to-end test drives the real runEval against a real single-commit fixture repo and counts VERIFIER INVOCATIONS: six attempts, four spawns, the two no-progress ones free, and a tracked edit, a new untracked file and an edit to that untracked file each re-arm it. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
The fingerprint folds untracked file contents, and runEval appends to .graff/eval-log.tsv on every call — in a repo that does not gitignore .graff/, the tree provably moved each iteration and the guard failed open (measured: strictly worse than no guard, +prompt growth, zero savings). graff's own state dir is now excluded from the untracked scan. Found by the integration batch comparison. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #412 — the git-fingerprint no-progress guard, adopted from the prime-agent analysis and landed where graff's goal/loop verification actually lives: the
evaltool.Mechanism
Before running a previously-RED verification, fingerprint what could change its result: the
--evalcommand text (a re-pointed check is a different check),git status --porcelain -z -uall,git diff --binary HEAD, and the contents of every untracked file (status + diff are byte-identical however an untracked file changes — both a unit test and the E2E pin this). Identical fingerprint → the verifier is not run at all: no command spawn, no--judgemodel call, no output tail. The attempt still counts (a stuck loop converges on its cap instead of spinning for free) and the model is steered at the real blocker: the workspace has not changed — edit something first.Fail-open everywhere: no repo, spawn error, nonzero exit, timeout, truncated probe, >512 untracked files, >8MiB untracked bytes, unreadable file — all read as "changed", the verifier runs. A verifier that could not run disarms the guard ("edit source files" is not the fix for an unrunnable command). Fingerprinting happens in a private arena — only 32 bytes outlive the capture.
Evidence
expected 1, found 2), not state flags.verify_fingerprint.zig(274 lines) wired viatest_hooks.zigwith reachability proven.v0.0.241 (unreleased)section;v0.0.240's header gets its ship date.Noted, not owed here:
scripts/eval/tier1-manifest.jsonstill saystest_count_baseline: 656vs a 1000-test suite — the ratchet was already ~338 behind on main; flagged for the next release cut.Part of the prime-agent adoption batch (#409-#421).