chore(eval): promote the live A/B token-economics harness into the repo

scripts/eval/live-ab/.gitignore

@@ -0,0 +1,4 @@
+runs/
+results.json
+promptcap/
+__pycache__/

scripts/eval/live-ab/README.md

@@ -0,0 +1,92 @@
+# Live A/B eval harness
+
+Measures what a change actually costs or saves on real agent runs: api calls,
+input tokens, cached tokens, output tokens, tool calls, wall time, and whether
+the task was solved. Two graff binaries run the same fixtures with the same
+model, N times each, on a fresh copy of the fixture per run.
+
+This exists because token-economics claims are easy to assert and hard to
+prove. Every number in a `RESULTS-*.md` here is regenerable from the run
+artifacts the harness leaves behind.
+
+## Running it
+
+Build the two binaries you want to compare (both `-Doptimize=ReleaseSafe`, or
+wall time is meaningless), then:
+
+```sh
+export GRAFF_EVAL_BEFORE=/path/to/base-worktree/zig-out/bin/graff
+export GRAFF_EVAL_AFTER=/path/to/change-worktree/zig-out/bin/graff
+
+python3 run_eval.py                      # all tasks, both arms, 3 runs each
+python3 run_eval.py f3_goalloop          # one task
+python3 run_eval.py f3_goalloop after 5  # one task, one arm, 5 runs
+python3 make_results.py > RESULTS.md     # regenerate the report
+```
+
+Optional: `GRAFF_EVAL_MODEL` (default `gpt-5.6-sol`), `GRAFF_EVAL_TIMEOUT`
+(default 300s hard kill).
+
+Results accumulate in `results.json` and are merged by `(task, binary, run)`,
+so a re-run of one cell replaces just that cell. Raw stdout, stderr and the
+post-run fixture tree stay under `runs/<task>/<arm>/<n>/`. Both are gitignored:
+they are outputs, not instrument.
+
+## Measurement validity
+
+Four things have to hold or the comparison says nothing:
+
+1. **Same model on both arms.** The harness enforces one `MODEL` for the whole
+   invocation.
+2. **Both arms ReleaseSafe.** A Debug binary is roughly an order of magnitude
+   slower and makes every wall-time column a lie.
+3. **Companion MCP servers off.** `one_run` writes `.harness/settings.json`
+   with `{"skills":{"codedbpro":false}}` into every run copy, so tool schemas
+   cannot drift between the arms and silently move the input-token number.
+   This lives in the harness rather than in each fixture because the repo
+   `.gitignore` excludes `.harness/`, so a committed copy would not survive a
+   clone. `empty-mcp.json` is here for the same purpose when you need to
+   override MCP config wholesale (point `GRAFF_MCP_CONFIG` at it).
+4. **Fresh fixture per run.** `one_run` deletes and re-copies the fixture, so
+   run 2 never inherits run 1's edits or `.graff/` state.
+
+`--no-telemetry` is passed throughout.
+
+## The fixtures
+
+| task | fixture | what it exercises |
+|---|---|---|
+| `f1_bugfix` | `f1-bugfix` | Ordinary agentic work: run tests, find a planted median bug, fix it, get 11 tests green. The broad regression check. |
+| `f2_biglog` | `f2-biglog` | Big tool output, natural phrasing. A 168 KB build log with exactly one `status=FAILED` line. Models usually narrow with grep rather than dumping the file. |
+| `f2b_forcedcat` | `f2-biglog` | Same log, but the prompt forbids narrowing and demands `cat` through bash, forcing the single oversized tool output the natural phrasing avoids. |
+| `f3_goalloop` | `f3-goalloop` | The goal/eval loop: stub `slugify()`, iterate with the eval tool until the checker scores 100. |
+| `f3b_reverify` | `f3-goalloop` | Same task, but the prompt tells the model to re-confirm a failing verdict without editing, which manufactures the precondition for the re-verify fast path. |
+| `f4_embedder` | `f4-embedder` | `--no-local-tools`, single call, pure reasoning puzzle. Isolates system-prompt size with near-zero variance, which makes it the cleanest prompt-size probe in the set. |
+
+Success is checked per task, not eyeballed: pytest exit status for `f1`, the
+required request id and failure reason appearing in the answer for `f2`, a
+final checker score of 100 for `f3`, and the correct one-word answer for `f4`.
+
+## Static prompt capture, at zero model cost
+
+`mock_capture.py` serves an OpenAI-compatible endpoint on `127.0.0.1:1234`,
+records the first request body, and answers with a trivial completion. Point a
+binary at it to diff the exact system prompt two builds send without spending
+anything:
+
+```sh
+python3 mock_capture.py before.json &
+GRAFF_EVAL_BEFORE_BINARY --model lmstudio -p "hi"
+```
+
+This is how the per-call prompt-size deltas in the reports were obtained. It is
+far more precise than inferring prompt size from live token counts, and it is
+the right first instrument for any prompt change.
+
+## Reading the reports honestly
+
+`make_results.py` reports medians with full min-max spread and marks whether
+the two ranges separate. When they overlap, the comparison is inconclusive and
+no claim should be drawn from it. At n=3 against a live model, single-digit
+percentage deltas are usually sampling noise. Wall time is the noisiest column
+and should never carry a conclusion on its own.

scripts/eval/live-ab/RESULTS-2026-08-05-prime-batch.md

@@ -0,0 +1,233 @@
+# Live A/B: pending prime batch vs its base
+
+**Model** `gpt-5.6-sol` (codex, subscription flat-rate) - **36 live runs**,
+6 tasks x 2 binaries x 3 runs, fresh fixture copy per run, `--no-telemetry`
+throughout, companion MCP disabled in every fixture so tool schemas could not drift
+between arms.
+
+- **BEFORE** `wt-expbase` @ `693039c` (v0.0.239-14) - the batch's merge base
+- **AFTER** `wt-compare` @ `49084d0` (v0.0.240-25), branch `integration/prime-batch`,
+  brought current with `origin/feat/prime-409-spill` + `origin/feat/prime-412-goalfp`;
+  `zig build test` **1041/1041 green, exit 0** before measuring
+- Both built `-Doptimize=ReleaseSafe` (the pre-existing BEFORE binary was a release
+  build; a Debug AFTER would have made wall-time meaningless)
+
+Regenerate everything below with `python3 make_results.py > RESULTS.md`.
+
+## Topline
+
+| task | mechanism under test | api calls | input tokens | output tokens | wall s | success |
+|---|---|---|---|---|---|---|
+| F1 bugfix | whole-batch regression check (no single PR targeted) | 12->11 (-8.3%) | 104645->97055 (-7.3%) | 1012->883 (-12.7%) | 51.5->44.2 (-14.2%) | 3/3 -> 3/3 |
+| F2a big-output (natural prompt) | #409 spill vs truncation | 3->3 (+0.0%) | 22929->23552 (+2.7%) | 137->147 (+7.3%) | 10.8->10.7 (-1.0%) | 3/3 -> 3/3 |
+| F2b big-output (forced `cat`) | #409 spill vs truncation, oversized output forced | 3->3 (+0.0%) | 23145->23855 (+3.1%) | 227->324 (+42.7%) | 13.3->14.5 (+8.6%) | 3/3 -> 3/3 |
+| F3 goal loop | #412 skip-a-re-verify | 6->6 (+0.0%) | 40113->41202 (+2.7%) | 542->498 (-8.1%) | 24.8->24.3 (-2.1%) | 3/3 -> 3/3 |
+| F3b goal loop (RED re-confirm) | #412 skip-a-re-verify, precondition forced | 7->7 (+0.0%) | 50605->50142 (-0.9%) | 689->553 (-19.7%) | 25.7->30.3 (+17.8%) | 3/3 -> 3/3 |
+| F4 embedder prompt | #421/#410 capability-gated system prompt | 1->1 (+0.0%) | 5900->5575 (-5.5%) | 7->7 (+0.0%) | 1.8->2.0 (+10.7%) | 3/3 -> 3/3 |
+
+Success was 3/3 on every task for both binaries: **36/36 runs passed their task
+check, no timeouts, no wedged runs.** The batch is behaviourally safe. What it is
+not is a broad efficiency win -- three of the six comparisons are noise, one is a
+measured win, one is a measured (small) cost, and one shows a mechanism that never
+fires at all.
+
+## The single number that explains most of the table
+
+The batch changes the system prompt in two opposite directions, and which one you
+get depends on whether local tools are on. Captured statically against a local mock
+endpoint (`promptcap/`, zero model cost), first request of a session:
+
+| session shape | BEFORE system-prompt chars | AFTER | delta | |
+|---|---|---|---|---|
+| full capability (local tools on) | 9113 | 10073 | +960 | +10.5% |
+| embedder mode (`--no-local-tools`) | 9113 | 7703 | -1410 | -15.5% |
+
+#421 *gates* segments away when a capability is absent -- worth -1,410 chars in
+embedder mode. But the same PR also *adds* two always-on doctrine paragraphs
+("never invent a tool, a parameter, or a wrapper API..." and "Run the target
+project through its OWN environment...") and #410 adds the durable-transcript line.
+At full capability nothing is gated, so only the additions land: **+960 chars**.
+
+That per-call delta is multiplied by the number of api calls in a run, which
+predicts the observed input-token deltas closely: ~+240 tok/call x 3 calls = +720
+predicted vs +623 observed (F2a); x 6 = +1,440 vs +1,089 (F3); and -350 x 1 call =
+-325 observed exactly (F4). The one task that breaks the pattern is F3b, for the
+reason given in its section.
+
+
+## F1 bugfix
+
+*Mechanism under test: whole-batch regression check (no single PR targeted)*
+
+| metric | BEFORE median (min-max) | AFTER median (min-max) | delta | separated? |
+|---|---|---|---|---|
+| api calls | 12 (9-13) | 11 (10-12) | -8.3% | no (overlap) |
+| input tokens | 104645 (76787-112184) | 97055 (88143-105849) | -7.3% | no (overlap) |
+| cached tokens | 83456 (63488-93184) | 80896 (77824-83456) | -3.1% | no (overlap) |
+| output tokens | 1012 (866-1098) | 883 (878-1011) | -12.7% | no (overlap) |
+| tool calls | 15 (15-20) | 15 (15-17) | +0.0% | no (overlap) |
+| wall seconds | 51.5 (39.1-83.0) | 44.2 (40.7-52.9) | -14.2% | no (overlap) |
+| success rate | 3/3 | 3/3 | - | - |
+
+The batch is not a regression: both binaries fixed the planted median bug and got
+all 11 tests green in 3/3 runs. Every efficiency metric moved AFTER's way at the
+median (api calls -8.3%, input -7.3%, wall -14.2%), **but every one of those
+ranges overlaps** -- BEFORE spent 9-13 calls, AFTER 10-12. At n=3 with a live
+model this is model-sampling noise, not a measured improvement. The honest read
+is: no detectable difference in agentic bugfix efficiency, and no regression.
+
+
+## F2a big-output (natural prompt)
+
+*Mechanism under test: #409 spill vs truncation*
+
+| metric | BEFORE median (min-max) | AFTER median (min-max) | delta | separated? |
+|---|---|---|---|---|
+| api calls | 3 | 3 | +0.0% | no (overlap) |
+| input tokens | 22929 (22926-22937) | 23552 (23548-23591) | +2.7% | **yes** |
+| cached tokens | 20992 (14336-22016) | 14336 (14336-20992) | -31.7% | no (overlap) |
+| output tokens | 137 (137-142) | 147 (140-185) | +7.3% | no (overlap) |
+| tool calls | 2 | 2 | +0.0% | no (overlap) |
+| wall seconds | 10.8 (8.7-12.6) | 10.7 (7.8-11.5) | -1.0% | no (overlap) |
+| success rate | 3/3 | 3/3 | - | - |
+
+This is the null result the batch's own design predicts, and it is worth stating
+plainly: **#409's spill never fired.** With a naturally phrased prompt the model
+did not dump the file at all -- it called `read_file`, then `grep 'status=FAILED'
+build.log`, and answered from two small outputs. Only ~23k input tokens moved, on
+a 168 KB log. Both binaries did the same thing and both got the right answer 3/3.
+
+
+## F2b big-output (forced `cat`)
+
+*Mechanism under test: #409 spill vs truncation, oversized output forced*
+
+| metric | BEFORE median (min-max) | AFTER median (min-max) | delta | separated? |
+|---|---|---|---|---|
+| api calls | 3 | 3 | +0.0% | no (overlap) |
+| input tokens | 23145 (23127-23177) | 23855 (23817-23876) | +3.1% | **yes** |
+| cached tokens | 20992 (18944-22016) | 18944 (7680-20992) | -9.8% | no (overlap) |
+| output tokens | 227 (205-267) | 324 (282-338) | +42.7% | **yes** |
+| tool calls | 2 (2-3) | 2 | +0.0% | no (overlap) |
+| wall seconds | 13.3 (10.7-14.2) | 14.5 (12.4-14.9) | +8.6% | no (overlap) |
+| success rate | 3/3 | 3/3 | - | - |
+
+Forcing `cat build.log` through bash still did not reach #409. The reason is
+structural, and it is the most useful finding in this report. Three caps sit in
+series, and the first two are **identical in both binaries**:
+
+1. `tools.zig:358 bash_stdout_cap = 128 KB` -- exec truncates the command's stdout
+   and appends `[stdout truncated at 128 KB]`;
+2. `agent_tools.zig:56 tool_preview_chars = 2_000` -- `persistToolResult` writes the
+   (already 128 KB-capped) text to `.graff/tool-results/<run>-<n>.txt` and the model
+   receives a 2,000-char head plus a pointer to that file;
+3. `provider.perOutputCap()` = min(context/2, 256 KB) = **136,000 B** for
+   codex/gpt-5.6-sol -- this is the send-time cap that #409 hooks.
+
+Because layer 2 reduces every `execTool` result to 2,000 chars, layer 3's 136 KB
+threshold is unreachable for bash/read_file/webfetch/codedb. Both binaries wrote a
+`.graff/tool-results/` artifact (pre-existing behaviour, present in BEFORE too);
+neither wrote a #409 spill artifact, in any of the 12 F2 runs. **#409's spill path
+is effectively dead code for the local toolset** -- it can only fire for a tool
+result that bypasses `toolPreviewText`. That is a design observation, not a bug in
+the PR, but it means #409 buys nothing measurable on ordinary coding work.
+
+
+## F3 goal loop
+
+*Mechanism under test: #412 skip-a-re-verify*
+
+| metric | BEFORE median (min-max) | AFTER median (min-max) | delta | separated? |
+|---|---|---|---|---|
+| api calls | 6 | 6 | +0.0% | no (overlap) |
+| input tokens | 40113 (40111-40128) | 41202 (41067-41340) | +2.7% | **yes** |
+| cached tokens | 33664 (32640-34688) | 34688 (31616-34688) | +3.0% | no (overlap) |
+| output tokens | 542 (492-613) | 498 (474-673) | -8.1% | no (overlap) |
+| tool calls | 5 | 5 | +0.0% | no (overlap) |
+| wall seconds | 24.8 (23.5-28.4) | 24.3 (21.5-29.4) | -2.1% | no (overlap) |
+| success rate | 3/3 | 3/3 | - | - |
+
+Both binaries solved the two-property checker identically: baseline `eval` (which
+reveals both required properties in its RED output), one `edit_file`, second `eval`
+-> 100. 6 api calls, 5 tool calls, 3/3 success, on both. #412's fast path never
+had an opportunity, because `skipReverify()` only fires when the **last eval
+FAILED and nothing changed** -- here every eval followed an edit. AFTER's ~+1,089
+input tokens is the system-prompt delta below, multiplied by 6 calls.
+
+
+## F3b goal loop (RED re-confirm)
+
+*Mechanism under test: #412 skip-a-re-verify, precondition forced*
+
+| metric | BEFORE median (min-max) | AFTER median (min-max) | delta | separated? |
+|---|---|---|---|---|
+| api calls | 7 | 7 | +0.0% | no (overlap) |
+| input tokens | 50605 (50511-50645) | 50142 (49983-50182) | -0.9% | **yes** |
+| cached tokens | 42240 (40192-42240) | 43264 (42240-43264) | +2.4% | no (overlap) |
+| output tokens | 689 (607-750) | 553 (530-696) | -19.7% | no (overlap) |
+| tool calls | 6 | 6 | +0.0% | no (overlap) |
+| wall seconds | 25.7 (25.1-39.3) | 30.3 (29.9-34.4) | +17.8% | no (overlap) |
+| success rate | 3/3 | 3/3 | - | - |
+| #412 re-verify skips fired | 0 | 3 | - | **yes** |
+
+This variant manufactures #412's actual precondition by telling the model to
+re-confirm a RED verdict before editing, and it is the one place the batch shows a
+clean, deterministic win. **The skip fired in 3/3 AFTER runs and 0/3 BEFORE runs.**
+On BEFORE the checker really ran a second time over an identical tree and its full
+RED report -- the whole failing-cases block -- entered the history. On AFTER the
+`eval` tool returned the one-line no-progress steer instead.
+
+The saving compounds, because an elided tool output is not paid once but re-sent on
+every subsequent request: retained history at end of run is 16,070 chars on BEFORE
+vs 14,413 on AFTER (-10.3%), and BEFORE's history carries two `failing cases`
+blocks where AFTER carries one. That is why F3b is the only tool-enabled task where
+AFTER's input tokens come out *lower* (-0.9%) despite paying the larger system
+prompt on all 7 calls -- the elision more than covers it. Wall time is noisier and
+went the other way (+17.8%, ranges overlap); with a ~50 ms checker there is no real
+time to win. The payoff scales with verifier cost, so on a repo whose eval is
+`zig build test` this is seconds per redundant re-confirm, not milliseconds.
+
+
+## F4 embedder prompt
+
+*Mechanism under test: #421/#410 capability-gated system prompt*
+
+| metric | BEFORE median (min-max) | AFTER median (min-max) | delta | separated? |
+|---|---|---|---|---|
+| api calls | 1 | 1 | +0.0% | no (overlap) |
+| input tokens | 5900 | 5575 | -5.5% | **yes** |
+| cached tokens | 3584 (0-5632) | 0 (0-4608) | -100.0% | no (overlap) |
+| output tokens | 7 | 7 | +0.0% | no (overlap) |
+| tool calls | 0 | 0 | - | no (overlap) |
+| wall seconds | 1.8 (1.6-2.4) | 2.0 (1.7-3.1) | +10.7% | no (overlap) |
+| success rate | 3/3 | 3/3 | - | - |
+
+The cleanest measurement in the set, and a real win: **5,900 -> 5,575 input tokens,
+identical in all three runs on each binary, zero variance, -5.5%.** This isolates
+#421 exactly. The static capture confirms the cause: with `--no-local-tools` the
+system prompt drops from 9,113 to 7,703 chars (-1,410, -15.5%) because the
+capability gate removes the read_file/edit_file/write_file/codedb/bash paragraph,
+the `.graff/traces` paragraph, the `gh issue create` paragraph and the batching
+note's tool examples -- instructions an embedder could never act on. 1,410 chars is
+~350 tokens, which matches the -325 measured live. Both binaries answered the
+puzzle correctly 3/3.
+
+
+## Caveats, honestly
+
+- **n=3 per cell.** Medians are reported with full min-max spread and every table
+  says whether the two ranges separate. Where they overlap (all of F1, most wall
+  times) the comparison is **inconclusive**, and no claim is made from it.
+- **AFTER is not purely the five PRs.** `integration/prime-batch` also carries the
+  0.0.240 release merge and engine slice-1c (#422/#330) -- typed-event refactors
+  that should be behaviour-neutral but are inside the measured diff.
+- **F2 could not exercise #409 at all**, by construction of the harness rather than
+  by fixture error; see F2b. The 168 KB fixture and forced `cat` were both correct
+  and still could not reach the send-time cap.
+- **F3b's prompt is deliberately artificial** ("the checker has been flaky, re-run
+  it before editing"). It was written to satisfy `skipReverify`'s precondition
+  after F3 showed the natural phrasing never does. It proves the mechanism works;
+  it does not prove models often hit it unprompted.
+- Wall time includes provider-side latency and is the noisiest column here; it
+  should not carry any conclusion on its own.
+

scripts/eval/live-ab/empty-mcp.json

@@ -0,0 +1 @@
+{"mcpServers":{}}

scripts/eval/live-ab/fixtures/f1-bugfix/.gitignore

@@ -0,0 +1,2 @@
+.graff
+__pycache__/

scripts/eval/live-ab/fixtures/f1-bugfix/inventory.py

@@ -0,0 +1,34 @@
+"""In-memory inventory of stocked items."""
+
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class Item:
+    sku: str
+    name: str
+    quantity: int
+    unit_cost: float
+
+
+class Inventory:
+    def __init__(self, items=None):
+        self._items = {i.sku: i for i in (items or [])}
+
+    def add(self, item):
+        self._items[item.sku] = item
+
+    def get(self, sku):
+        return self._items.get(sku)
+
+    def skus(self):
+        return sorted(self._items)
+
+    def quantities(self):
+        return [self._items[s].quantity for s in self.skus()]
+
+    def unit_costs(self):
+        return [self._items[s].unit_cost for s in self.skus()]
+
+    def total_value(self):
+        return sum(i.quantity * i.unit_cost for i in self._items.values())

scripts/eval/live-ab/fixtures/f1-bugfix/pricing.py

@@ -0,0 +1,25 @@
+"""Pricing rules layered on top of an Inventory."""
+
+from stats import mean, median, spread
+
+
+def markup_price(unit_cost, markup_pct):
+    return round(unit_cost * (1.0 + markup_pct / 100.0), 2)
+
+
+def typical_quantity(inventory):
+    """The representative stock level, resistant to one huge outlier."""
+    return median(inventory.quantities())
+
+
+def average_unit_cost(inventory):
+    return mean(inventory.unit_costs())
+
+
+def cost_spread(inventory):
+    return spread(inventory.unit_costs())
+
+
+def restock_target(inventory, factor=2):
+    """Restock every SKU up to `factor` x the typical quantity."""
+    return int(typical_quantity(inventory) * factor)