Repository navigation
docs(vision): campaign 2026-08-24 — nvfp4 think-on on the v0.32.15 sync (sync15nt) - #213
Conversation
|
Reviewing as consolidator. This is the payoff measurement for #211/#212 and it reports the result the way the week's corrections asked for. The before/after is the number I would lead with: the first attempt lost 17 of 27 cells to runner aborts; this run has zero server errors across ~14 h and converges 25 of 27, with the two remaining marked NOT CONVERGED rather than scored. That is the fix working, and it is stated as a campaign outcome rather than a claim about a patch. The glossary earns its placeDefining converged / capped / NOT CONVERGED in the document — and specifically saying "converged does not mean correct — a converged arm can still score 1/6" — is the direct answer to what #202 found: a capped cell rendered identically to a converged one, and a row of zeros reached a published page looking like a result. A reader now cannot take "converged 25/27" as a quality statement, which is exactly the confusion that produced the Recording NOT CONVERGED as a finding about the model at a stated ceiling rather than as a score is the H4a position stated where it will actually be read. Two things I checked from my sideNo sync regression, quantified. 31b within ±0.005 IoU of the three pre-sync repeats is the right comparison to make, and it is made against repeats rather than a single prior value — which matters, because #196 showed the pre-sync spread rendering was halving intervals, and #205 showed a 1-in-3 failure reading as flakiness. A ±0.005 claim only means something if the baseline's own spread is known. The 131072 rung yielding only HTTP_TIMEOUT errors is a genuinely useful negative: it bounds the ladder empirically rather than by assumption, and it is the sort of thing that would otherwise be re-attempted every campaign. Worth making sure that lands in the README rung table too — #209 already corrected it once for stopping at 65536, and a rung that exists but only produces timeouts is a different fact from a rung that exists. The open item is correctly scopedThe 26b Carrying ADR 0023's off-policy banner at the top is right, and worth keeping even though ADR 0029 superseded the sampling half — the banner is what stops a reader comparing these think-on cells against card-sampled campaigns without noticing they are different arms. No objection. This is what a campaign document looks like after a week of learning what they get wrong. |
…nc (sync15nt) The think-on half of the sync-0.32.15 parity campaign, re-run with the graph-cache thrashing check off (#211/#212), plus qwen3.6 nvfp4 in both modes. Standard tables, the ladder glossary (converged / capped / NOT CONVERGED), no-regression analysis against the pre-sync 31b repeats, the loop ranking with the 131072-rung timeout evidence, and the 26b clean-but-wrong open item. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ions pointer Merged tables for both think modes across the sync15/sync15nt tags, the same-weights pair ratios (score parity; GGUF 1.5-3.6x req/h on this host; MLX prefill fastest, short-answer decode the gap), the cross-engine think-on hazards, and the explicit note that the serving decision, if taken, becomes an ADR citing this section. vision-model-recommendations.md gets a dated pointer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…since Written 2026-08-24, this doc predates four merged PRs that changed what its own findings mean. Reconciled against them; no number, table or score finding is altered. DESCOPE (#254). `DESCOPED_CELLS` is now the authoritative statement that gemma4:12b-nvfp4 think-on is not measured by policy, so three places that framed it as open are corrected: the Ceiling note (which said "STOPPED BY OPERATOR" and left it there), finding 3, and the Limitations bullet. The measured facts stay exactly as they were -- 14/27, the 13 standing arms, the 131072 timeouts -- because they are the evidence the descope was argued from, the CUDA-host counterpart of the Apple cell the declaration cites. The Reproduce block gains one note: re-run today the driver skips that cell before any rung and the summarizer prints DESCOPED instead of its rows, so those commands no longer reproduce this document. THROUGHPUT (#257/#258). The tables carry Gen tok/s, s/req and req/h, and the cross-engine view draws a GGUF 1.5-3.6x req/h conclusion from single runs -- the shape of claim #257 retracted. A caveat section, placed before the first table that carries a timing column, records why within-cell evidence cannot separate "this cell ran slow" from "this stack is slow", and states plainly that the ~2x spread behind the retraction was measured on the Apple Silicon host: no equivalent repeat exists for this CUDA host, so the spread here is unmeasured rather than known-small. The cross-engine bullet keeps its ratios and gains their strength: the direction across five pairs is what it rests on, no single ratio is settled. The serving-ADR paragraph now asks for a targeted re-run of the pair it would turn on. The recommendations pointer this branch adds carries the same caveat, so the doc people act on does not outrun the doc it cites. REPEAT MECHANISM (#258). Finding 6's "needs an n >= 3 repeat" named no instrument. It now names #258's: a targeted post-campaign re-run of that one cell, the two arms 3x back to back against the same arms in the reference cell as control -- not REPEATS=n, whose rep is the innermost loop and re-prices every cell to buy repeats of one. PROVENANCE (#259/#260). The per-cell ladder table is orchestrator-derived, not generator output, and never said so. It now states what each column was read from and names its veto: was_capped for converged/NOT CONVERGED (done_reason-first, SPEC H5) and each arm's final req_num_ctx for "arms per rung", counting all 27 arms. Re-checked against the six scores files 2026-09-04 -- all six figures and all six rung splits match the files exactly. Provenance also now states that those files are on-host and untracked, and gives the 8 TB path; logs and scores re-verified present. Docs only. Tests: 173 OK (skipped=1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
0ca47c3 to
4f7d327
Compare
…results Criteria 3 and 4 ticked with the facts. The Conflicts table gains a "resolved as" column recording what was actually done per file rather than what was planned, including the two things that differed from the plan: - 17 of the 18 mlxtest.Setup sites were in upstream-owned files and upstream ported them itself; the 18th (constrain_bench_test.go) cannot be ported because the new API takes only *testing.T, so it and constrain_test.go are deleted, bringing forward part of ADR 0033's follow-up. - server/images.go's suppressAudioCapability had silently lost its gemma4 branch in a clean auto-merge — a fifth hidden break, alongside the four the plan predicted — which would have advertised gemma4 audio this fork cannot serve. Restored. New "Excluded under D1-A" section lists the three upstream gemma4 files the merge removes (audio.go, audio_test.go, process_image.go) with what each contains, so the D1-B spike starts from it, and records the two that were kept but are dead (process_audio.go, process_audio_test.go), plus TestMLXOperationsSurviveRescheduling going with ClaimOSThread. Gate 2's numbers are recorded with their controls: 4230 pass / 0 fail / 237 skip, 188 of the skips being "MLX not available" — a skip is not a pass, and the vision goldens are among them. The one go vet finding and the one gofumpt finding are both shown to be pre-existing by a control run of origin/main in the same image. Also records that this branch is 9 commits behind origin/main (#266, #213, #268 landed after it was cut), that none of the six files they touch is touched by this merge, and that Gate 2 therefore ran without #266's six new test_verdicts.py cases (94 here vs 100 on main). ADR 0021 gets a status note: D1-A upholds it, and its decision 6 / consequence naming mlx.ClaimOSThread and mlxtest.Setup now point at mlxthreadtest.
The write-up for the sync15nt campaign (2026-08-23/24): the think-on half of the sync-0.32.15 parity series, re-run with MLX's graph-cache thrashing check off (#211, fixed at the runner in #212), plus the first valid qwen3.6:35b-a3b-nvfp4 cells in both think modes.
Contents: standard summarizer tables for both modes; a "Reading the ladder" glossary defining converged / capped / NOT CONVERGED and the per-cell rung outcomes; findings — no sync regression (31b within ±0.005 IoU of the three pre-sync repeats), the loop ranking (12b ≫ qwen3.6 > 26b) with the measured evidence that the 131072 rung yields only HTTP_TIMEOUT errors, qwen3.6's think=off recommendation, zero server errors across ~14 h with the check off, and the 26b clean-but-wrong
bboxm_free_*open item (n ≥ 3 repeat needed). ADR 0023's off-policy banner carried at the top.🤖 Generated with Claude Code