Repository navigation
fix(vision-suite): the rung ⚠ marks escalation, not finetext - #198
Conversation
#197's rung row scans every section of every run, so the finetext block sets it off on its own. vision_suite.py:1136 gives finetext `NUM_CTX or 32768` where every other section takes `NUM_CTX or 16384`, so with NUM_CTX unset the two differ in every run ever measured — and the qwen3.8 ROCm arms, whose vision tests all sat at 16384, rendered "16384/32768 ⚠". A warning that fires on every arm carries no information. The docstring's own justification is escalation *between repeats*, which is a different axis from sections *within* a run, and the implementation collapsed both into one list. Sections that own their window are now excluded from the ⚠ and annotated instead: "16384 (finetext 32768)". The alarm is left for what the docstring describes. Requested-over-served is kept and now has a test. It looks wrong — two pre-#153 finetext blocks record req_num_ctx 32768 against num_ctx 16384 — but those are exactly the files where `num_ctx` carries the pre-fold defect of reporting the suite default, so served-first would report a block at a window it did not run at. The qwen3.8 ROCm campaign is re-rendered with both rows, and two things it never disclosed are now on the page. Its finetext arm ran at 32768/4000, not the "16384 throughout" the prose claimed. And the pooled n=5 think-on arm is not homogeneous in num_predict — four runs at 2200 and the NUM_PREDICT=4400 ceiling diagnostic at 4400 — which is the one ⚠ that survives scoping. Pooling those is defensible only because no think-on cell was capped, and the doc now says so rather than leaving it implicit. Mutation-tested: unscoping the warning, dropping the annotation, preferring the served value, and removing the ⚠ entirely each turn the new tests red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Reviewing as consolidator. The "different axes" observation is the whole PR and it is correct.
That is precise, and the cross-reference to The evidence is the right kind, too: dumping all seven score files section by section, and finding scene/document/multi and all eight Annotating rather than dropping — On
|
Follow-up to #197, and it finishes the qwen3.8 ROCm campaign record.
The ⚠ fired on every arm
_windowscans every section of every run.finetexttakes its own window —vision_suite.py:1136gives itNUM_CTX or 32768where every other section takesNUM_CTX or 16384— so withNUM_CTXunset the two differ in every run ever measured. The qwen3.8 ROCm arms, whose vision tests all sat at 16384, rendered:I dumped all seven score files section by section: scene, document, multi and all eight
bbox_contractarms are at 16384/2200 in every run. Onlyfinetextdiffers, and it differs by design.The docstring justifies the ⚠ as "the arm escalated between repeats" — runs within an arm. The implementation also folded in sections within a run. Those are different axes, and collapsing them means the alarm can no longer isolate the one it names. Same shape as
token_accounting_okon #189: a check that always fires stops being read.Sections that own their window are now excluded from the ⚠ and annotated instead, so the information survives without the false alarm:
Requested-over-served is right, and now has a test
sec.get(f"req_{key}") or sec.get(key)looks backwards — two pre-#153 finetext blocks recordreq_num_ctx 32768againstnum_ctx 16384. But those are precisely the files wherenum_ctxcarries the pre-fold defect of reporting the suite default, so reading served-first would report a block at a window it never ran at. Kept, with a test naming the reason.What this surfaced in the campaign record
The one ⚠ that survives scoping is real, and the doc had never disclosed it:
finetextran at 32768/4000, not the "num_ctx16384 throughout" the prose claimed.num_predict— four runs at 2200 plus theNUM_PREDICT=4400ceiling diagnostic. Pooling them is defensible only because no think-on cell was capped (that run used 46% of its ceiling), so no score in the table is budget-limited. The doc now states that rather than leaving it implicit, and flags that it would not hold for a throughput comparison.Both window rows in the doc are byte-identical to a fresh render.
Verification
OWN_WINDOW = ())21 tests in
test_summarizers.py, plus 15 rescore and 43 preflight, all green.