[glennneuber]

ADR 0023 asks for n=3 per think-on cell because on-policy sampling isn't deterministic — but nothing turned three scores files into one row. Computing those means by hand into a table is exactly what ADR 0012 rule 8 forbids, so this does it.

python3 summarize_reps.py rocm-n3-qwen38-thinkoff rocm-n3-qwen38-thinkon

Files match as scores_<tag>.json and scores_<tag>-rep<N>.json, so an arm measured once and an arm measured three times both work, and the reported n says which.

It reports spread, and that is the point

Max-min rather than a standard deviation: at n=3 the range is what a reader can reason about, and it's the number that decides whether a difference between arms survives the noise within one. It's printed separately as "the bar any cross-arm claim must clear".

That framing isn't decorative — it comes from getting this wrong twice this week on qwen3.8:

n	what I concluded about name_bbox
1	think-on is a gain (0.570 → 0.790)
2	retracted — spread 0.159 ≈ the gain, so it's noise
5	real after all — think-on's lowest (0.631) clears think-off's highest (0.571); the arms never overlap

Two samples showed the spread was large, which was true, but not that the arms overlapped — and I conflated those. A tool printing both numbers side by side would have stopped the flip-flop.

Pooled at n=5 vs n=2 it reads:

| doc name_bbox IoU | 0.571 ±0.001 | 0.742 ±0.103 |
| multi q4-bbox     | 2/2 ✅        | 0/5 ❌        |
  think-off spread: scene bbox IoU 0.003, doc name_bbox IoU 0.001

Incidentally establishing that think-off is near-deterministic but not exactly so — 0.003 on scene IoU across two greedy runs, consistent with GPU float-reduction ordering. The ADR's convention of a single think-off run rests on an assumption that is very nearly, but not quite, true.

Derives finetext correct/fabricated

Per-tier recall alone misleads. Every model emits all 20 codes regardless of what it can resolve (finetext_probe.py says so), so a zeroed small tier means fabricated codes, not omitted ones — and tiers can swap between runs while the total holds: two think-on runs scored 4/4/4/1/0 and 4/4/4/0/1, identical totals, opposite tiers. Reading that as "lost the 9px tier" is wrong; the total is the signal.

Verified

Against the seven qwen3.8 runs on 10.8.0.4, including pooling this morning's arms with tonight's to n=5, plus the no-args and missing-tag paths.

🤖 Generated with Claude Code