perf(decision): compute only requested output rows behind the scoring API #586

Parent: #583
Depends on: #585
Planned branch: codex/decision-fast-path.

Problem / motivation

The first scoring API still computes the full vocabulary projection. When callers need a handful of alternatives, the output-head work and readback can be reduced while preserving the established score contract.

Proposal / scope

• Add an optional architecture readout hook at the final hidden-state/output-head boundary; reuse the backbone and resolved-weight kernels.
• Resolve capability and select the implementation at handle/plan creation. Keep a reference implementation behind the same API for parity comparisons.
• Compute only the requested output rows, with bounded preallocated workspace and small result transfers.
• Preserve final normalization, head scaling/softcaps, tied-weight handling and the output-head Hadamard transform for Bonsai. The output path must respect the checkpoint's stored basis.
• Keep selected-candidate normalization explicit. A subset of rows cannot supply the full-vocabulary softmax denominator; requests for full-vocabulary probabilities need the dense path or an explicit unsupported result.
• Normal generation continues through its existing output path. Do not copy or quantize the whole backbone just to enable decisions.

Acceptance

• On supported model/backend pairs, selected scores agree with that backend's dense reference under the existing arithmetic/quantization contract; candidate ranking/normalization is covered, and tolerances are not widened to hide regressions.
• Cover ordinary Qwen/Llama-family weights and Bonsai PQ2_0 with prism.hadamard where supported; unsupported combinations report their capability explicitly.
• Instrumentation demonstrates that the optimized call skips the full vocabulary projection/readback, rather than computing and then slicing it.
• Measure the head microbenchmark and end-to-end decisions over multiple prompt lengths and candidate counts; record latency, allocations, memory and whether the fast path actually ran.
• Rerun the standard prefill/decode sweep and relevant real-model quality checks. Existing generation cannot regress unnoticed.
• Feature-off behavior, instance isolation, checked sizes and all applicable AGENT.md checks remain covered.

Existing work to reuse

Bonsai format/rotation support is already on main. #432 and #561 are completed regression history, not new implementation tasks.

Implementation and validation

Draft PR: #599 on codex/decision-fast-path, stacked on #592. Apple Silicon CPU/Metal first; x86 and other backends explicitly unsupported for selected rows. All acceptance items have implementation/validation evidence in the PR and benchmark/results/DECISION_SELECTED_ROWS_2026-10-04.json.

400 recorded benchmark samples include 80 paired DENSE/selected trials with exact candidate logits. Release and ASan/UBSan suites each pass 70 tests with 40 expected skips; feature toggles, optimized GCC and native row/tail tests pass. Qwen, Bonsai/Hadamard and Gemma/softcap are covered; full ordinary logits and eight token IDs are byte-identical before/after on CPU and Metal. The ordinary prefill/decode sweep and confirmation runs retain their raw samples and timing noise.

Final standalone Bonsai-shaped head means on M1 Max / 64 GiB are 3359.3 → 19.4 µs CPU and 1939.4 → 412.7 µs Metal. Whole-query Bonsai shows no reliable gain; Qwen gains about 1–6%. This is performance-only evidence, not MMLU accuracy or Jev parity. Representative quality/calibration follows in #587, with MMLU first on Apple Silicon.

The issue remains open until its implementation PR lands.