[imanu86]

Title

Issue: none (an opt-in speed change, bit-exact).

Summary

STRATA_ATTN_DAILY=1 (opt-in, off by default) runs the QSA layers' decode attention - the verify window's and the MTP
drafter's, single query and batch - through PR #540's chunk kernel (attn_chunk_kernel_pre75: two cells per step
with a 12-shuffle reduction, cell-major probabilities, V loaded four at a time) on every CUDA card, and merges the
chunks with attn_merge_v2_kernel, a merge that skips the chunks the chunk kernel marked empty. Upstream compiles the
pre75 kernel only with STRATA_EXPERIMENTAL_SM60 and picks it below sm_75; here it is compiled in every CUDA build
and selected by the variable. The output is bit for bit the default kernels' (see below). HIP is untouched.

Why

The default attn_chunk_kernel reduces each cell with 5 warp_sum shuffles per head (60 per cell pair); the pre75
body does two cells per step with reduce12 (16 shuffles) and reads the V rows four at a time. nsys on an RTX 3060
(stage 0 of a layer split, 131K context): attn_chunk_kernel 99.8 us a call against 46.4 us with the pre75
body; the same ratio on an RTX 2080 Ti. The decode attention is ~8% of the per-row cost of a window at 131K on these
cards, so the end-to-end gain is small but free.

What changed

• src/kernels/cuda/qsa_decode_attn.cu: attn_chunk_kernel_pre75<M, NC, ...> compiled in every CUDA build;
  daily_attn() reads STRATA_ATTN_DAILY; the kernel choice STRATA_ATTN_CHUNK_SEL(M, PRE) takes it on any card
  when set (below sm_75 in an experimental build as before). attn_merge_v2_kernel with the chunk-empty marks;
  STRATA_ATTN_MERGE_V2=0|1 forces the merge alone, STRATA_ATTN_PRE75=0 turns the chunk kernel off (A/B).
  spt shares sq's shared memory, so the kernel uses 15,872 B like the default one (68-78 registers against 37-41,
  256 threads a block).
• include/strata/kernels/qsa_decode_attn.hpp: the selection helpers.
• src/kernels/qsa_merge_daily_parity.cpp (tests): memcmp of the chunk kernel and the merge against upstream's on
  four KV formats, widths at the chunk borders and non-resident pages; --bench times the two chunk kernels back to
  back. Registered as ctest qsa_merge_daily_parity_default and _daily.
• CMakeLists.txt: the harness.

Bit for bit

reduce12 adds at every level the same two pairs as warp_sum's butterfly (xor 16, 8, 4, 2, 1; float addition
commutes), the V cells fold in the same cell order with the same fmaf, the masked cells stay excluded; the merge v2
does the same fmaf in the same order and skips exactly the chunks the chunk kernel marks empty. qsa_merge_daily_parity on an RTX 2080 Ti (v0.1.43 build, tests on): 1,168 cases, 0 differ (fp16 290, int8 291, q4_0 294, k8v4 293; 21,190,308 nonzero output values), with STRATA_ATTN_DAILY unset and =1. In the engine the greedy text and the drafter's acceptance are unchanged (deterministic
131K bench, forced tokens and windows).

Measurements

RTX 2080 Ti 22 GB (sm_75) + RTX 3060 12 GB (sm_86), Windows 11, CUDA 12.6, layer split, Qwen3.8-Flash-Next IQ3_XXS,
131K context, --kv int8.

qsa_merge_daily_parity --bench on the RTX 2080 Ti (CUDA events, 200 calls per arm, B-A-B: Daily kernel, upstream
kernel, Daily kernel again; int8 KV, 32,768-cell capacity, window of n_q queries):

n_q	width (cells)	chunk kernel: upstream	PR #540 body	ratio	merge: upstream	merge v2	ratio
1	32,768	448.9 us	264.3 us	0.59	182.6 us	45.6 us	0.25
2	32,768	849.8	483.9	0.57	186.6	60.9	0.33
3	32,768	1,259.8	727.3	0.58	202.3	85.1	0.42
4	32,768	1,692.3	969.0	0.57	208.4	106.5	0.51
4	2,051	132.7	86.6	0.65	41.3	9.9	0.24

fp16, q4_0 and k8v4 KV show the same ratios (0.57-0.63 for the chunk kernel, 0.24-0.51 for the merge). In the engine
(nsys, RTX 3060, 131K context) attn_chunk_kernel went from 99.8 us to 46.4 us a call.

End to end it is part of a stack that matches its reference within noise; the attention alone is ~8% of the per-row
cost, so no separate end-to-end claim is made here.

Built and started on v0.1.43 with STRATA_ATTN_DAILY=1 (RTX 2080 Ti + RTX 3060 layer split, Windows): the engine serves as before, same answer.

Risks / limits

Opt-in: nothing changes with the variable unset. The kernel body is PR #540's (sskver) and the merge is Hardin22's
attn_merge_v2_kernel, both carried in this fork since 0.1.40 and credited in the source. #1888 (split attention on
the int8 mma path, sm_80+) is a different kernel and not bit-exact; the two do not overlap (this one serves Turing).

Part of a series from a public patch registry (stack/ in the fork imanu86/Strata-2080Ti).

[q8atnight]

Hardware: 2x RTX 3090 24 GB (NVLink bridge), Ryzen 9 3950X (AVX2, no AVX-512), 121 GB RAM, PCIe 4.0 x8 per card. Qwen3.8-Flash-Next unsloth UD-Q4_K_XL, --layer-split 24, experts in locked RAM, 262K context, int8 KV, spec 4, one stream, v0.1.43.1 base.

What we compared: the same stack build with STRATA_ATTN_DAILY=1 off vs on, one restart per arm, everything else identical.

Answers: word-for-word identical with the flag on, against a stock v0.1.43.1 reference — our exactness gate is 10 greedy prompts, a 20K long gate, an image prompt and a 10-question verifiable probe; all clean.

Decode (flashbench medians, greedy; the flag-on arm also had #1937's --conversation-disk-dir active, which does not touch the decode path):

decode t/s	1K	8K	32K	128K
stack, flag off	123.8	127.5	120.9	111.5
stack, STRATA_ATTN_DAILY=1	129.3	133.6	129.6	118.6

That is +4–7 % greedy at every context. Sampled rows: 118.4 / 124.1 / 110.9 / 109.2 → 124.5 / 123.2 / 119.3 / 110.6 (+5 % at 1K and 32K, flat at 8K).

Second box, different class (IQ3_XXS, --resident-experts, so a different attention mix): 2x RTX 5060 Ti 16 GB layer split + RTX 2080 Super for vision, Ryzen 7 3700X, ~32 GB RAM, 196K context. With only this flag added, decode 1K/32K went 117.8 / 107.7 → 121.6 / 117.5 t/s, prefill unchanged. On that box our gate was already 1/3 identical from earlier stack items, and adding the flag moved the math prompt (differs at char 138) — so on IQ3_XXS here it was not bit-exact, while the Q4 box above was clean.

[Niko1221]

Thanks. Ported as an opt-in (STRATA_ATTN_DAILY=1). Text is identical to the default in our gate on sm_86 and on the Pascal build. Decode at a 30K context: +3.2% on an RTX 3060 (5 of 5 pairs), +0.7% on an RTX A4000. We keep it off by default for now because one tester saw a changed answer on a 5060 Ti; if you or they can send that prompt we will look.

[q8atnight]

The 5060 Ti tester is us — that row is ours from 10-14. Happy to hand over the details, and the switch is already in our
daily driver here, so the row below is live traffic rather than a bench.

The box. 2× RTX 5060 Ti 16 GB (sm_120, layer split, second card at PCIe x8), Ryzen 7 3700X (Zen 2, no AVX-512),
~32 GB RAM, Qwen3.8-Flash-Next IQ3_XXS with the native expert pack, --resident-experts + --mmap-experts,
196,144-token context, int8 KV, MTP spec 4, one stream, Linux, container.

What we ran. The same stack built with STRATA_ATTN_DAILY=1 off and on, one restart per arm, nothing else changed.
Our exactness gate on this box is 3 greedy prompts (chat / code / math), a 20K long-prompt gate, an image prompt and a
10-question verifiable probe, all against a stock reference from the same base.

The result. With the flag on, chat and code came back word-for-word; math differed from the first character
138. The 10-question probe was 10/10 both ways, and decode went 117.8 → 121.6 t/s at 1K and 107.7 → 117.5 t/s at 32K,
prefill unchanged — so we kept it on and it has been on in daily use since 10-14 (subagent traffic, thinking mode,
nothing has gone wrong with it). Two caveats so the row is read correctly:

• On this box the gate was already 1/3 identical before we added your flag, from other stack items we carry — so
  "adding the flag moved the math prompt" is a within-stack delta, not a clean stock-vs-flag delta.
• This is the IQ3_XXS pack with --resident-experts, i.e. a different expert-residency and attention mix from the IQ3_S
  and UD-Q4 boxes where the same flag was bit-exact (our 2× 3090 UD-Q4_K_XL box: gate clean, decode +4-7 %).

What we can send you. The exact math prompt and both outputs, and if useful a token-level diff of the two
generations (first divergence at char 138, then where they rejoin). Say which shape you want.

And an offer, because 0.1.45 makes it clean. Now that the switch is upstream, we can run the confounder out of it:
stock 0.1.45 with STRATA_ATTN_DAILY=0 vs =1, same binary, one restart per arm, on this IQ3_XXS box — that isolates
the switch from everything else we carry, on the first NVIDIA Blackwell-class rig anyone has gated it on. It needs a
restart window on this machine (it is our secondary engine, so the window is short notice rather than free); if you want
it, say so and we will schedule it.

One question while we are at it: the bounded merge skips empty chunks — is the threshold that decides that
format-dependent in any way? On a native-pack box with most experts resident the attention path sees a different mix
than on the cards you measured, and if the merge boundary can move with the pack, that would explain why the same switch
is bit-exact on sm_86 with IQ3_S / UD-Q4 and not here.

— Andy (q8atnight), 2× RTX 3090 (NVLink, peer tier) + 2× RTX 5060 Ti (layer split)

[Niko1221]

Thanks for the pull request.

Your change is already in main for 0.2.0-beta, with your credit, so there is nothing left for you to do here. We're closing it because we're resetting the tracker for 0.2.0-beta: there are more than 1,300 open issues and pull requests, and we're closing everything that isn't being worked on right now.

If part of your work did not make it in and you want it considered, open a new pull request on top of main that follows CONTRIBUTING.md, or comment /reopen after fixing the title area prefix and the template.