[q8atnight]

Summary

In a captured decode window qsa_block_scores launched a (max_blocks / 8) x nq grid (~24,600 mostly idle blocks per
layer) and re-read every pooled key block once per query. New kernel: a fixed grid strides over the key blocks, reads
each once and scores it for all of the window's queries (up to 8); per (block, query) the same arithmetic in the same
order as block_scores_kernel. Used only when no active-block count is given (the captured decode window).

One commit on top of 0.1.27 (a790805); all four of my current branches were compile-tested together on 0.1.27.

Measured

Measured on 2x RTX 3090, IQ3_S, with the engine this was developed in (0.1.24 + the dual-GPU work): at 32K context the
"scores+topk" stage 1.02 -> 0.90 ms per window; verify window 20.28 -> 20.04 ms.

Correctness

Bit-identical scores.

Switch

STRATA_SCORES_MULTI=0 = the old grid.

[Adamyno]

Independent sm_75 data point (single RTX 2080 Ti 22GB, CUDA 13.0, arch 75, IQ3_XXS, MTP spec=4;
built together with #186/#188 on 0.1.27, A/B-ed via the env switches).

One observation you may want to check: with your scores kernel active alone (the other two off),
greedy outputs are character-identical to the state with #188 active alone — but BOTH differ from
the same binary with all three switches off, at one early token ("hydraulic" → "low-frequency").
If the per-(block, query) arithmetic is truly in the same order, that points at a shared
selection/ordering or capture-window effect rather than numeric drift — flagging for your look.

Speed on this card at short context is neutral, consistent with your stage numbers:
1,495 prompt + 1024 gen: 48.1-50.0 tok/s vs main 49.3-50.8; 30,174 + 512: 46.5-47.1 vs 46.1
(warm medians). No regression.

[Niko1221]

The arithmetic matches block_scores_kernel element for element, so it goes to the byte gate for 0.1.29. Please fix the comment: prefill's pooled16 call and the uncaptured decode also take this path.

[q8atnight]

Thanks — good catch to flag it, and the pattern is interesting. We re-checked the diff on our side:
the switch gates the whole change (STRATA_SCORES_MULTI=0 runs the old kernel and nothing else
differs), and the per-(block, query) arithmetic is textually identical to block_scores_kernel —
same float4 loads, same per-head dot with the same shfl reduction order, same ReLU accumulation,
same +1e9f sentinel. On our 2×3090 rig the bit-identical gate (dual == single, every prompt) passes
with the kernel active.

What you measured — scores-alone and recurrence-alone giving character-identical output, both
differing from all-off at one early token — is hard to explain from either kernel's arithmetic
(they live in different subsystems). Our best lead: the flip token is a near-tie that any small
perturbation flips the same way, and the perturbation may be upstream's own #152 — CPU native-expert
results depend on the expert-group width (singleton vec_dot vs AVX multi-token kernels round
differently), so window composition alone can flip a near-tie even with bit-identical GPU kernels.

Three things that would pin it down, if you have the rig handy:

1. Is the flip stable across repeated runs of the SAME config, or does it move? (deterministic vs run-to-run)
2. Does a plain 0.1.27 binary (without the three branches) produce the same output as your merged
   binary with all switches off? (isolates a build/merge side effect from the switches)
3. With STRATA_NO_IQ512=1 STRATA_NO_IQ4NL=1 STRATA_NO_IQ256=1 (the width-invariant CPU-expert trio
   from Native CPU expert results depend on verifier width (plain/MTP divergence) #152), does the all-off state still differ from the scores-alone state?

Speed neutral on your card is expected — the win is in the 32K-context window structure, as your
numbers show. Thanks again for the independent data point.

[q8atnight]

Fixed and pushed (rebased on 0.1.28): the comment now names every caller that takes the multi kernel (the
captured decode window, the uncaptured decode, and prefill's pooled16 call — any call without an active-block
count and at most 8 queries), and the box-specific note is gone.

On the one-token question (@Adamyno): we re-ran it on 0.1.28 + #186/#187/#188, single RTX 3090, IQ3_S, with
the #152 trio (STRATA_NO_IQ512/256/IQ4NL=1) + no-stream-k: all switches off twice → identical (the reference is
repeatable), then scores-only, recurrence-only and both-on → all three prompts byte-identical to all-off.
So on our side the kernels don't move a token; our best guess for your flip is the #152 width-dependent CPU-expert
rounding (a near-tie token), which the trio removes. If you can repeat your A/B with the trio set, that would
settle it.

[Niko1221]

Released in 0.1.29: with your comment follow-up; byte-identical on all four quants. Merged under your name (cherry-picked, so GitHub doesn't show it as merged). Thank you!