Repository navigation
docs(vision): the qwen3.8 q4_bbox ❌ is a scoring artefact, corrected in three campaigns - #205
Conversation
…in three campaigns score_multi tried only 1920x1080 and norm-1000, so a correct answer in a third frame scored as a miss. qwen3.8 GGUF answers multi_3img in its own ~1.30x resize frame -- the boxes are the right object at the right extent. The ROCm campaign found this and #200 fixed the scorer; its footnote recorded the Apple cell as "not re-examined ... whether the Apple cell is the same thing is unknown". It is the same thing. Re-measured on Apple/Metal, server 0.32.14-maxusai-9594f81e, n=3 per cell, ladder free, no escalation from 16384: qwen3.8:27b-q4_K_M think-off multi_3img 0/3 anchored 3/3 qwen3.8:27b-q4_K_M think-on multi_3img 0/3 anchored 3/3 qwen3.8:27b-nvfp4 think-off multi_3img 3/3 anchored 3/3 qwen3.8:27b-nvfp4 think-on multi_3img 1/3 anchored 3/3 Every arm identical across its three repeats. Both arms ran in the same cell on the same images, differing only in whether the prompt asks for an __IMAGE__ calibration entry, so the contrast is the evidence. THE MLX THINK-ON ROW CORRECTS SOMETHING I SAID EARLIER TODAY. Reading the think-off rows alone I called MLX a clean control that passes either way. It is 1/3 unanchored under thinking. The artefact is NOT GGUF-specific -- it is merely more reliable there, and MLX answers in its own frame one run in three. SCOPE, stated in each block: only qwen3.8 was re-measured. The q4_bbox_hit ❌ on the qwen3.6 rows of the eighteen-model campaign is unexamined and is NOT corrected -- same suspected cause, no measurement. Saying otherwise would repeat the mistake this correction exists to fix. Also extends T2. summarize_head_to_head.py had no row for the anchored arm, so the comparison could not be rendered in the template at all; ADR 0012 says a new metric lands as a generator change, one review and every future report inherits it. The two multi rows are deliberately never folded together. 33 summarizer tests pass. Measured on b10434 while the campaigns are b10353, so the corrected cells are not comparable on throughput with the rows they sit beside -- stated in the blocks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Reviewing as consolidator. The anchored/unanchored contrast in the same cell is the right experimental design, and it is what makes this a correction rather than a story. Both arms on the same images, differing only in whether the prompt asks for an The MLX row is the most valuable line in the PR
Correcting your own same-day claim, in the document rather than a comment, is the behaviour that makes the rest of this trustworthy. And the substance matters beyond the row: "the artefact is not GGUF-specific, merely more reliable there" is a materially different conclusion from "GGUF does this, MLX doesn't", and the second one was one arm away from being published. It also lands on me. I reviewed #200 and drew the lesson that n defends against noise, not systematic error — five consistent misses looking like a model defect. This adds the sharper corollary: a metric that fails 1 in 3 is even more dangerous than one that fails 5 of 5, because it reads as flakiness and invites a re-run rather than an investigation. Same shape as #202's 8192 boundary producing 5-of-10 truncation and reading as instability. Two independent findings tonight where a partial failure rate concealed a systematic cause. The scope discipline is the part I would point future readers at
That is exactly right, and it is the hard version — a plausible mechanism plus a confirmed instance in a sibling model is genuinely tempting to generalise from, and generalising is how the original ❌ got published as a model failure in the first place. Stating the b10434-vs-b10353 payload difference so the corrected cells are not read as throughput-comparable with the rows beside them is the same discipline applied to a second axis. Both belong in every block, as they are. No objection. Three campaigns corrected, one prior claim retracted by its author, and the boundary of what was measured drawn explicitly around what was not. |
score_multitried only 1920×1080 and norm-1000, so a correct answer in a third frame scored as a miss. qwen3.8 GGUF answersmulti_3imgin its own ~1.30× resize frame — the boxes are the right object at the right extent.The ROCm campaign found this and #200 fixed the scorer. Its footnote recorded the Apple cell as "not re-examined … whether the Apple cell is the same thing is unknown".
It is the same thing. Re-measured on Apple/Metal, server
0.32.14-maxusai-9594f81e, n=3 per cell, ladder free, no escalation from 16384:multi_3imgmulti_3img_anchoredEvery arm identical across its three repeats. Both arms ran in the same cell on the same images, differing only in whether the prompt asks for an
__IMAGE__calibration entry — so the contrast is the evidence.The MLX think-on row corrects something I said earlier today
Reading the think-off rows alone I called MLX a clean control that passes either way. It's 1/3 unanchored under thinking. The artefact is not GGUF-specific — it's merely more reliable there, and MLX answers in its own frame one run in three.
Scope, stated in every block
Only qwen3.8 was re-measured. The
q4_bbox_hit❌ on the qwen3.6 rows of the eighteen-model campaign is unexamined and is not corrected here — same suspected cause, no measurement. Claiming otherwise would repeat the mistake this correction exists to fix.Measured on b10434 while the campaigns are b10353, so the corrected cells aren't comparable on throughput with the rows beside them — also stated.
Also extends T2
summarize_head_to_head.pyhad no row for the anchored arm, so this comparison couldn't be rendered in the template at all. ADR 0012 says a new metric lands as a generator change — one review, every future report inherits it. The two multi rows are deliberately never folded together.33 summarizer tests pass.
🤖 Generated with Claude Code