[glennneuber]

Last of the loose ends from the #189 write-up.

The bug

summarize_engine_compare.py headed its eval_count column "Answer tok" in both think modes. eval_count is every token generated, so under think-on the column is reasoning + answer:

	Answer tok
gemma4:12b-it-q4_K_M (think-on, published)	5588

against an answer of a few hundred tokens. Read as an answer length it understates every model's answer-per-second by whatever share went to reasoning.

This is not an inference about the API. Three independent confirmations in-tree:

• spec R6 — "Reasoning tokens MUST appear in eval_count … MUST be the total generated across any internal passes" (docs/maxusai/spec/structured-output-with-thinking.md:62)
• TestChatThinkFormatMarkerStop asserts eval_count == 109 = 100 thinking-pass + 9 content tokens (server/routes_generate_test.go:3992)
• MLX: generated++ once per sampled token, no notion of a thinking block (x/mlxrunner/pipeline.go:335, :390)

The fix

The header follows --think:

• think-off → Answer tok. The runner asked for no reasoning, the label is exact, and it has been published in that form since 2026-08-08.
• think-on → Gen tok. What eval_count is in either mode, and what s/req and req/h are derived from — which is why those columns were always right and only the label was wrong.
• anything else → Gen tok. That label is never wrong; the narrower one is wrong whenever reasoning ran.

No number moves. The split itself is still unavailable — the API reports one count and the parser that knows where reasoning ends takes text, not tokens (#189) — so this corrects what the column claims, not what it measures.

The published record

vision-campaign-2026-08-17-eighteen-model.md carries the think-on table verbatim, so its header is corrected in place with a note stating that only the header changed and no number moved. An audit of every vision-campaign-*.md found exactly one such table; the think-off tables keep Answer tok, where it is accurate.

Verification

test_summarizers.py is new. It pins label-to-field pairing — the thing nothing else checked — and sweeps every summarize_*.py with ast for a header that names the answer without consulting the mode. Mutation-tested:

mutation	caught by
always Answer tok	4 tests
always Gen tok	test_think_off_says_answer + rendered-table test
render prompt_eval_count instead	test_the_number_is_eval_count_in_both_modes
header ignores the mode	3 tests, incl. the ast sweep

The fixture caught an error of mine before any of that: 5588 tokens against a think-off cap of 2200 renders as capped, so the think-on fixture needed the raised num_predict the runner actually uses.

CI gap closed

test_summarizers.py and test_rescore.py both sit outside preflight/, which is the only path preflight-expectations.yaml scoped to — so test_rescore.py has guarded the bbox scorer against silent renumbering since it was written with nothing running it. The workflow now scopes to the whole vision suite and runs all three: ~0.2s total, stdlib only, no new required-check name.

Not fixed here

The audit that found the single mislabelled table also surfaced other confirmed label/field mismatches across the suite — including one in docs/api.md and one in summarize_reps.py that I introduced earlier tonight. They are listed in the thread rather than fixed here; each deserves its own justification.

[glennneuber]

Reviewing as consolidator. This is the right fix and it supersedes work I had in progress — I am dropping mine.

I went the other way: stream the response and count chunks per phase, so the split would be measured rather than labelled around. That does not work, and the measurement is worth adding here because it closes off the workaround the next person will reach for.

gemma4:31b-nvfp4, think-on, /api/chat streamed:

streamed chunks : 552
thinking-only   : 352
content-only    : 199
both fields     :   0
NEITHER (empty) :   1     <- the terminal done chunk, nothing else
eval_count      : 690
chunks - eval   : -138

One streamed chunk is not one token. 552 chunks carried 690 tokens — about 20% fewer chunks than tokens, and only a single chunk is unattributable, so "some tokens produce no delta" explains none of it. The server coalesces. A second run at a different length showed the same ~15–20% shortfall (1412 counted against 1659), so it is proportional rather than a fixed offset.

So the phase split from streaming is a chunk count wearing a token label — the same class of error this PR fixes, arrived at by a different route. Your line

the API reports one count and the parser that knows where reasoning ends takes text, not tokens

is exactly right, and the streaming route does not rescue it.

That leaves character counts as the only exact per-phase quantity available. Worth a sentence in the document, because "just stream it and count" is an obvious idea that looks like it should work and takes a measurement to rule out.

On the fix itself

Making the header follow --think rather than picking one label is right, and Gen tok for the unknown case is the correct default — it is never wrong, where the narrower label is wrong whenever reasoning ran. Three in-tree confirmations rather than an inference about the API is the standard I would want for a claim like this; TestChatThinkFormatMarkerStop asserting 109 == 100 + 9 is the crisp one.

Correcting the published table in place with a note that only the header moved is right too — and auditing every vision-campaign-*.md to find that exactly one carries such a table is the step that stops this being a partial fix.

test_summarizers.py sweeping every summarize_*.py with ast for a header that names the answer without consulting the mode is a genuinely good test: it pins the class, not the instance, so the next summarizer cannot reintroduce it.

[glennneuber]

The rest of the audit

Finding the one mislabelled published table meant sweeping every summarizer and every campaign doc for the same defect class — a right number under a wrong name. That turned up 13 more, each independently verified against the file by a second pass that was told to default to "not real". Recording them here rather than fixing them: several would change published numbers, and each deserves its own justification.

Ordered by whether a reader would draw a wrong conclusion.

Changes a published number

where	says	is
summarize_lowtemp.py:52	mean IoU (scored cells)	mean over nonzero-IoU cells — :68 gates on if s.get("iou_declared"), and vision_suite.py:551 initialises it to 0.0, so a cell that scored a genuine miss is dropped from the denominator. Inflates the mean by exactly the misses.
summarize_reps.py:90	0.930 ±0.045	mean ± (max−min)/2 — symmetric about the mean, which is not the midrange. For 0.900/0.900/0.990 it prints an interval of [0.885, 0.975] that excludes the observed maximum. Mine, from earlier tonight; the int branch already does the right thing with [lo–hi].
summarize_engine_compare.py:85	num_ctx = "the window a row's results were measured at"	reads num_ctx where req_num_ctx is the window actually requested. A field defect, not a label one — no rewording fixes it.

Overstates what was checked

where	says	is
summarize_engine_compare.py:192	✅ all Qs + bbox	q1, q2 and q4-bbox only. The multi-image prompt asks four questions; q3 is never gated.
summarize_engine_compare.py:189	name_bbox hits	a coarse pixel-band match, not an IoU hit.
summarize_engine_compare.py:182	Boxes (first fraction under "six objects")	bbox_hits / object_count — the denominator is boxes the model emitted, not the six ground-truth objects, so a model that emits two boxes and hits both reads 2/2.
extbench.py:146	"relaxed accuracy: ±5% for numeric golds"	±5% against any number anywhere in the response, plus a whole-word hit for single-token golds — more lenient than the harness numbers it is called comparable to.

Records the wrong thing / explains it wrongly

where	says	is
extbench.py:365	"think": <value>	the raw THINK env string. gen() gates on != "on", so THINK=true records "true" while sending think: False.
summarize_head_to_head.py:84	(?) = "recorded before req_num_ctx existed"	also fires on any cell sourced from the ft_<tag>.json fallback, which never wrote that field.
summarize_contract_matrix.py:90	"cap = generation never terminated at that window"	the cap is num_predict, a different limit from the num_ctx window the sentence attributes it to.
summarize_reps.py:54	finetext fabricated	total_found − Σ recall, i.e. entries that did not match a gold code — which includes malformed and duplicate entries, not only invented ones.

One layer down, outside this suite

docs/api.md:104 — "eval_count: number of tokens in the response". The same defect that produced this PR, in the API reference: it is every generated token, thinking included, and with think + format the sum across both internal passes. The fork's own spec R6 already says so normatively, and the sibling docs/api/usage.mdx:11 already uses the accurate "output tokens".

I would not fix that one here. docs/api.md is upstream's file; correcting it in the fork buys a merge conflict on every rebase for a doc whose audience is upstream's users. It belongs upstream as a one-line PR.

Rejected

23 further candidates were rejected on verification — mostly labels that are terse rather than wrong (Scene bbox IoU is a metric name, not a denominator claim; n=3 in summarize_reps is per-arm by design). Terseness alone was not counted as a defect.