chore(eval): promote the live A/B token-economics harness into the repo - #449
Merged
Conversation
The instrument that produced the +240 tok/call prompt-delta finding and the three-cap-stack finding lived only in a session scratchpad, which made every number it produced unreproducible the moment that session ended. It moves to scripts/eval/live-ab/ largely intact, with the changes needed to survive a clone: - the two arms' binaries come from GRAFF_EVAL_BEFORE / GRAFF_EVAL_AFTER instead of hardcoded worktree paths, resolved lazily so running a single arm needs only that arm's variable, and failing with an actionable message rather than a traceback; - model and timeout are overridable (GRAFF_EVAL_MODEL, GRAFF_EVAL_TIMEOUT), with one model per invocation so the two arms cannot silently diverge; - one_run now WRITES .harness/settings.json into each run copy rather than relying on a committed dotfile. Disabling the companion MCP server is a measurement invariant (its schemas move the input-token number), and the repo .gitignore excludes .harness/, so a shipped copy would not have survived a clone at all; - run residue (.git, __pycache__, .pytest_cache) is stripped from the fixtures, and runs/ + results.json are ignored as outputs. RESULTS-2026-08-05-prime-batch.md is kept verbatim as the measured baseline those findings came from. Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The instrument behind the recent token-economics findings (the +240 tok/call system-prompt delta, and the three-cap-stack observation that motivated #440) lived only in a session scratchpad. Every number it produced became unreproducible the moment that session ended. This makes it durable.
The harness moves to
scripts/eval/live-ab/largely intact, with the changes needed to survive a clone:GRAFF_EVAL_BEFORE/GRAFF_EVAL_AFTERreplace hardcoded worktree paths, resolved lazily so running a single arm needs only that arm's variable, and failing with an actionable message rather than a traceback.GRAFF_EVAL_MODEL,GRAFF_EVAL_TIMEOUT), with one model per invocation so the two arms cannot silently diverge.one_runnow writes.harness/settings.jsoninto each run copy instead of relying on a committed dotfile. Disabling the companion MCP server is a measurement invariant, since its schemas move the input-token number, and the repo.gitignoreexcludes.harness/, so a shipped copy would not have survived a clone at all. This was caught by the staging step, not assumed..git,__pycache__and.pytest_cacheare gone from the fixtures;runs/andresults.jsonare ignored as outputs, not instrument.RESULTS-2026-08-05-prime-batch.mdis kept verbatim as the measured baseline the findings came from, andREADME.mddocuments the four conditions that have to hold for a comparison to mean anything (same model, both ReleaseSafe, companion MCP off, fresh fixture per run) plus how to read the reports without overclaiming at n=3.Verified: all three Python entry points byte-compile; the unset-environment path produces the intended message rather than a traceback; the fixture's single-
FAILED-line invariant still holds (line 1036 of 1151).Not verified: no live A/B was re-run against this copy, since that costs model spend. The next change that needs a number is the natural place to exercise it end to end.