Repository navigation
feat(vision-suite): make a comparable Apple arm runnable - #199
Conversation
|
Reviewing as consolidator. Both of these are real, and the second one describes something I did today. finetext's windowThe table is the argument:
Same test, same model, different windows, for the one metric that is entirely about how much small text survives — and nothing surfaced it until #198's rung row started printing Keeping the existing coupling as the default and pinning it with its own test is right. Worth noting for whoever runs the Apple side: my MLX/CUDA arms were driven through The tag mismatch — I hit this and worked around it badly
I rendered today's MLX/CUDA tables by copying score files into a scratch directory under names the summarizer would accept — So: no objection, and it removes a step I would otherwise have proposed separately. One thing to confirm before mergeThis is stacked on #198 and both edit |
Two things block anyone from producing an Apple arm that can be compared with the qwen3.8 ROCm campaign. Both were hit head-on rendering that campaign. FINETEXT'S WINDOW COULD NOT BE PINNED. finetext carries its own 32768/4000 and falls back to NUM_CTX/NUM_PREDICT when set. run_engine_compare.sh always sets them, so a runner-driven arm measures finetext at the suite's window, while the ROCm arms -- driven straight from vision_suite.py with NUM_CTX unset -- measured it at 32768. Same test, same model, different windows for the one metric that is entirely about how much small text survives, with no way to reconcile them and nothing reporting it until the rung row began printing "(finetext 32768)". FINETEXT_NUM_CTX / FINETEXT_NUM_PREDICT now pin that test alone. The existing coupling stays the default, and a test pins it so this is an escape hatch rather than a behaviour change. A REPEATED ARM COULD NOT BE POOLED UNDER A NAME. run_engine_compare.sh writes the rep number as a tag PREFIX; summarize_reps globs its own -rep<N> SUFFIX. They never meet, so REPEATS=3 produces three files that pool only via a shell glob -- which then becomes the column header of a published table. Worse, an arm pooling two tag families cannot be written as one glob at all: the ROCm n=5 arm was rendered by copying score files to matching names by hand, exactly the manipulation ADR 0012 rule 8 keeps out of a record. An arm may now name itself and list what it pools, and the published table reproduces from the real tags with nothing copied. An "=" in the label is fatal rather than a warning. It does not yield an empty pattern; it yields a plausible one that matches nothing, so the arm is dropped and the table renders one column short -- and a record missing an arm reads as a record that only had one. Mutation-tested: dropping multi-pattern support, rendering the spec instead of the label, removing the guard, and ignoring the finetext override each turn the tests red. The label mutation survived the first pass -- parse_arm was covered but the rendered header was not, which is the entire point of the syntax -- so there is now a test that renders a table and asserts the header. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
c95badc to
eae4bba
Compare
The ROCm side of the qwen3.8 campaign is finished. The Apple side can't be run from here, but two things would block whoever does run it — I hit both rendering the ROCm arms, so this lands the fixes rather than leaving them to be rediscovered.
1. finetext's window could not be pinned
finetextcarries its own 32768/4000 and falls back toNUM_CTX/NUM_PREDICTwhen set.run_engine_compare.shalways sets them (run_engine_compare.sh:199), so:vision_suite.py,NUM_CTXunset — the ROCm armsrun_engine_compare.shatNUM_CTX=16384— any normal Apple armSame test, same model, different windows for the one metric that is entirely about how much small text survives — with no way to reconcile them. Nothing reported it until #198's rung row started printing
(finetext 32768).FINETEXT_NUM_CTX/FINETEXT_NUM_PREDICTnow pin that test alone:The existing coupling stays the default and is pinned by its own test, so this is an escape hatch, not a behaviour change.
2. A repeated arm could not be pooled under a name
run_engine_compare.shwrites the rep number as a tag prefix (lt1_…);summarize_reps.pyglobs its own-rep<N>suffix. They never meet.REPEATS=3therefore produces three files that pool only via a shell glob — which then becomes the column header of a published table. And an arm pooling two families can't be written as one glob at all: the ROCm n=5 arm was rendered by copying score files to matching names by hand, the manipulation ADR 0012 rule 8 exists to keep out of a record.An arm can now name itself and list what it pools, and the published table reproduces from the real tags with nothing copied:
An
=in the label is fatal, not a warning: it doesn't produce an empty pattern, it produces a plausible one matching nothing, so the arm is dropped and the table renders one column short. A record missing an arm reads as a record that only had one.Verification
* survived the first pass —
parse_armwas covered but the rendered header was not, which is the entire point of the syntax. There is now a test that renders a table and asserts the header carries no*.29 summarizer + 15 rescore + 43 preflight, all green. README gains the pooling syntax, the label rule, and a worked example of matching the ROCm window from another host.
Not included
The Apple measurement itself. No host reachable from here runs an MLX build —
10.8.0.6holds the-nvfp4models but answersMLX not available … /dist/linux-amd64/lib/ollama, and there is no:11436anywhere. Separately, the published Apple arms are0.32.5-maxusai-*, which is not running either, so a run today would be a new baseline rather than the n=3 repeat the campaign's gap asks for.