prefill: software-pipelined loads in the column-split GDN recurrence (bit-identical)

[q8atnight]

Summary

The port Niko mentioned when closing #108 ("your software-pipelined loads in the recurrence go further than ours
(-29 % vs our -17 %)"): gdn_rec_cols_kernel with the next token's q/k rows, v, gate and beta loaded into registers
while the current token computes. Templated on columns per block (32 = upstream's split; 16 available via
STRATA_GDN_CB=16, measured neutral).

One commit on top of 0.1.27 (a790805); all four of my current branches were compile-tested together on 0.1.27.

Measured

Measured on 2x RTX 3090, IQ3_S, with the engine this was developed in (0.1.24 + the dual-GPU work): recurrence time
-29 % vs -17 % for the column split alone (as quoted in #108); prompt speed at 32K within a percent or two overall.

Correctness

The same arithmetic in the same order: bit-identical.

Switch

STRATA_GDN_PIPELINE=0 = upstream's kernel.

[Adamyno]

Independent sm_75 data point (same rig as my #187 note: RTX 2080 Ti 22GB, CUDA 13.0, arch 75,
IQ3_XXS, MTP spec=4, the three branches built together on 0.1.27).

Your pipelined recurrence is output-identical to #187's state end-to-end (character-identical
across three greedy runs), but both differ from the same binary with all three switches off at
one early token — shared with the #187 note; if both kernels are bit-identical at kernel level,
something shared (selection order, capture-window shape) is the candidate.

Speed on this card: neutral at 1,495 + 1024 (47.8-49.8 vs main 49.3-50.8 tok/s) and at
30,174 + 512 (46.5-47.1 vs 46.1, warm medians). The recurrence win is presumably inside the
32K-context window structure — no regression here either.

[Niko1221]

Same order as upstream, line for line. Queued for the exactness gate in 0.1.29. Could you drop the CB=16 variant unless it wins somewhere?

[q8atnight]

Thanks — same observation as on #187, and the same conclusion applies from our side: the switch gates
the whole change (STRATA_GDN_PIPELINE=0 runs upstream's kernel, nothing else differs), and the
pipelined kernel keeps the arithmetic textually identical — same fmaf sequence per state row, same
__expf / rsqrtf, same four-way red[] summation order. On our 2×3090 rig the in-engine
self-check (STRATA_PF_REC_CHECK=1, which compares the pipelined kernel against upstream's kernel
state and output per layer) prints IDENTICAL on every layer, and the bit-identical gate passes.

Since the recurrence is prefill-side and the scores kernel on #187 is decode-side, "both single-PR
states identical to each other but one early token off from all-off" points away from either
kernel's math. Two checks that would help on your rig:

1. Run with STRATA_PF_REC_CHECK=1 — if it prints IDENTICAL on sm_75 too, the flip comes from
   somewhere else entirely.
2. Is the flip stable across repeated runs of the same config? And does a plain 0.1.27 binary match
   your merged all-off binary on the same prompt? (isolates a build side effect)
   Our current best suspect is the known Native CPU expert results depend on verifier width (plain/MTP divergence) #152 effect (CPU native-expert rounding depends on expert-group
   width, so window composition alone can flip a near-tie token) — the STRATA_NO_IQ512=1 STRATA_NO_IQ256=1 STRATA_NO_IQ4NL=1 trio makes those results width-invariant if you want to test it.

Neutral speed at short context is expected — the recurrence win is inside the 32K-context window
structure. No regression on your card is exactly what we'd want to see. Thanks for taking the time.

[q8atnight]

Dropped and pushed (rebased on 0.1.28): CB=16 measured neutral here, so STRATA_GDN_CB and the template are
gone — one pipelined kernel on upstream's 32-column split, STRATA_GDN_PIPELINE=0 = the existing kernel.

Byte check on 0.1.28 (single RTX 3090, #152 trio + no-stream-k): pipeline on vs off → identical on all three gate
prompts (same result as on #187, where the details are).

[Niko1221]

Released in 0.1.29: with your follow-up (one kernel, the CB=16 variant dropped); the prompt path's residual dumps are byte-identical. Merged under your name (cherry-picked, so GitHub doesn't show it as merged). Thank you!